Method, device, electronic device and storage medium for audio and video conferencing

Abstract

The embodiment of the present application discloses a method, device, electronic device and storage medium for audio and video conferencing, which belongs to the field of communication core network technology and can improve the convenience of participating in audio and video conferences. It includes: receiving an incoming signaling from a participant in an audio and video conference by calling an intermediate number through a signaling server and sending it to a communication media server; establishing a communication connection with a conference signaling server through the communication media server, and using the conference signaling server to send communication information to the conference media server; according to the communication information, reserving a listening address for audio and video through the conference media server and synchronizing it to the communication media server; when the intermediate number is connected, mixing the audio and video through the conference media server to obtain a single-channel audio and video stream, pulling the single-channel audio and video stream through the communication media server according to the listening address and sending it to the participant, so as to conduct an audio and video conference based on the single-channel audio and video stream.

Description

Technical Field

[0001] The present application relates to the field of communication core network technology, and in particular to a method, device, electronic device and storage medium for audio and video conferencing. Background Art

[0002] Currently, video conferencing based on an Internet Protocol (IP) Multimedia Subsystem (IMS) network is very common.

[0003] In the related art, most video conferencing methods using an IMS network require conference participants to download corresponding conference software or access a corresponding webpage before joining the conference, resulting in a low convenience in participating in the conference. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for audio and video conferencing, so as to solve the problem of low convenience in participating in audio and video conferencing.

[0005] To solve the above technical problems, the embodiments of the present application are implemented as follows:

[0006] In the first aspect, an embodiment of the present application provides a method for audio and video conferencing, including: receiving incoming signaling from a participant in an audio and video conference calling an intermediate number through a signaling server, and sending it to a communication media server; creating a communication connection with a conference signaling server through the communication media server, and using the conference signaling server to send communication information to the conference media server, the communication information including: a message of the participant joining the conference, and a media address reserved by the communication media server for the audio and video of the incoming signaling; according to the communication information, a listening address is reserved for the audio and video through the conference media server, and synchronized to the communication media server, the listening address is used to push and pull the audio and video streams of the audio and video; when the intermediate number is connected, the audio and video are mixed through the conference media server to obtain a single-channel audio and video stream, and according to the listening address, the single-channel audio and video stream is pulled through the communication media server and sent to the participant, so as to conduct the audio and video conference based on the single-channel audio and video stream.

[0007] In the second aspect, an embodiment of the present application provides a device for audio and video conferencing, including: a receiving device for receiving incoming signaling from a participant in the audio and video conference calling an intermediate number through a signaling server, and sending it to a communication media server; a creating device for creating a communication connection with a conference signaling server through the communication media server, and using the conference signaling server to send communication information to the conference media server, wherein the communication information includes: a message that the participant joins the conference, and a media address reserved by the communication media server for the audio and video of the incoming signaling; a monitoring device for reserving a monitoring address for the audio and video through the conference media server according to the communication information, and synchronizing it to the communication media server, wherein the monitoring address is used to push and pull the audio and video streams of the audio and video; a processing device for mixing the audio and video through the conference media server when the intermediate number is connected to obtain a single-channel audio and video stream, and pulling the single-channel audio and video stream through the communication media server according to the monitoring address and sending it to the participant, so as to conduct the audio and video conference based on the single-channel audio and video stream.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory electrically connected to the processor, wherein the memory stores a computer program, and the processor is used to call and execute the computer program from the memory to implement the above-mentioned audio and video conferencing method.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program can be executed by a processor to implement the above-mentioned audio and video conferencing method.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the above-mentioned audio and video conferencing method.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the above-mentioned audio and video conferencing method.

[0012] The technical solution of the embodiment of the present application is adopted, and the incoming signaling of the participants of the audio and video conference calling the middle number is received by the signaling server and sent to the communication media server. The communication connection with the conference signaling server is established by the communication media server, and the communication information is sent to the conference media server using the conference signaling server. The communication information includes: the message of the participant joining the conference, and the media address reserved by the communication media server for the audio and video of the incoming signaling; according to the communication information, the conference media server reserves a listening address for the audio and video, and synchronizes it to the communication media server. The listening address is used to pull and push the audio and video streams of the audio and video. When the middle number is connected, the audio and video are mixed by the conference media server to obtain a single-channel audio and video stream. According to the listening address, the single-channel audio and video stream is pulled by the communication media server and sent to the participant to conduct an audio and video conference based on the single-channel audio and video stream. It can be seen that the participants of the meeting can connect to the audio and video conference by dialing the middle number. By reserving the media address and monitoring address for the audio and video corresponding to the participants, the audio and video streams corresponding to the audio and video can be pushed and pulled, allowing the participants to join the audio and video conference. Participants can enter the audio and video conference without downloading the audio and video conference software, solving the problem of low convenience in participating in audio and video conferences. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate one or more embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0014] Figure 1 This is a schematic structural diagram of an intermediate number platform according to an embodiment of the present application;

[0015] Figure 2 is a schematic flow chart of a method for audio and video conferencing according to an embodiment of the present application;

[0016] Figure 3 2 is a schematic diagram of media format conversion during an incoming voice call according to an embodiment of the present application;

[0017] Figure 4 2 is a schematic diagram of media format conversion during an incoming video call according to an embodiment of the present application;

[0018] Figure 5 Schematic diagram of a conference media server processing audio and video streams according to an embodiment of the present application;

[0019] Figure 6is a schematic swim lane diagram of a method for audio and video conferencing according to an embodiment of the present application;

[0020] Figure 7 is a schematic swim lane diagram of ending an audio or video conference according to an embodiment of the present application;

[0021] Figure 8 is a schematic flow chart of a method for audio and video conferencing according to another embodiment of the present application;

[0022] Figure 9 is a schematic block diagram of an audio and video conferencing device according to an embodiment of the present application;

[0023] Figure 10 This is a hardware structure diagram of an audio and video conferencing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] The embodiments of the present application provide a method, apparatus, electronic device, and storage medium for audio and video conferencing, which are used to solve the problem of low convenience in participating in audio and video conferencing.

[0025] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0026] The audio and video conferencing method provided in the embodiments of the present application can be executed by an electronic device or by software installed in the electronic device. Specifically, the electronic device can be a terminal device or a server device. The terminal device can include a smartphone, a laptop computer, a smart wearable device, an in-vehicle terminal, etc. The server device can include an independent physical server, a server cluster consisting of multiple servers, or a cloud server capable of cloud computing.

[0027] Figure 1 This is a schematic diagram of the structure of an intermediate number platform according to an embodiment of the present application. Figure 1As shown, the architecture of the intermediary number platform encompasses both the Information Technology (IT) and Communication Technology (CT) domains. The IT domain includes the conference service frontend, conference signaling services, and conference media services, while the CT domain includes signaling services and communication media services. The IT and CT domains are connected through communications, leveraging the interoperable and low-coupling architecture of Information and Communications Technology (ICT). This allows for independent deployment and scalability of each service, maintaining the stability of existing communications services while also facilitating the integration of new features.

[0028] The CT domain is deployed in the IMS domain. It should be understood that the IMS domain also includes: the Open Service Architecture (OSA) Service Capability Server (SCS), the OSA Application Server (OSA AS), the Home Subscriber Server (HSS), the Service Call Session Control Function (S-CSCF), the Multimedia Resource Function Controller (MRFC), the Multimedia Resource Function Processor (MRFP), and the Cloud Service Engine (CSE). Specifically, some media functions, such as stream mixing processing, are extracted from the communication media server. Since the communication media server is within the IMS communication network and the conference media server is external to the IMS communication network, extracting the stream mixing processing function from the IMS communication network can achieve modularization and independence of IMS communication network functions, helping to improve the flexibility and scalability of the corresponding entire system and facilitate the independent optimization, upgrade, and maintenance of specific media functions without affecting the stable operation of the entire communication media server. At the same time, extracting the media mixing function from the communication network can reduce the load and complexity of each server within the communication network, improving its overall performance and stability. In addition, because the conference media server is external to the communication network, it is easier to integrate and interact with related external systems or services.

[0029] The following describes in detail a method for audio and video conferencing provided by an embodiment of the present application through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0030] Figure 2 A schematic flow chart of an audio and video conferencing method provided by an embodiment of the present invention is shown, and the method includes the following steps:

[0031] S202: Receive incoming signaling from a participant of an audio or video conference calling an intermediate number through a signaling server, and send the signaling to a communication media server.

[0032] The signaling server (Application Server, AS) is similar to the role of the back-to-back user agent (B2BUA). In the communication network, it uses the Session Initiation Protocol (SIP) to implement sessions. Specifically, the SIP protocol can be used to negotiate signaling with the participants' terminals. The signaling server is also used to transmit the data corresponding to the participants' audio and video to the communication media server.

[0033] A middle number is a virtual number provided for privacy protection or temporary communication. An audio or video conference room generates a virtual number for entering the conference. The signaling generated by a user dialing the middle number is an incoming call.

[0034] The communication media server (MS) receives data packets corresponding to the participants' real-time audio and video from the signaling server and forwards, records, and saves the data. Furthermore, the communication media server pulls the mixed single-channel audio and video stream from the conference media server and pushes it to the participants' terminals via the core network, enabling them to have real-time online video conversations.

[0035] When a participant applies to join an audio or video conference through a voice or video call intermediate number, call signaling is generated. The call signaling is routed to the signaling server through the core network. The signaling server receives the call signaling and synchronizes it to the communication media server based on the call signaling, so that the media information of the audio and video corresponding to the incoming signaling passes through the communication media server, where the media information includes: data corresponding to the participant's audio and video.

[0036] S204: Create a communication connection with the conference signaling server through the communication media server, and use the conference signaling server to send communication information to the conference media server.

[0037] The communication information includes: the message that the participant joins the conference, and the media address reserved by the communication media server for the audio and video of the incoming signaling.

[0038] The Central Application Server (CAS) coordinates and manages audio and video conferences, such as allocating conference rooms, corresponding conference numbers, and synchronizing media, room numbers, and media addresses for pulling and pushing streams.

[0039] The communication connection includes a full-duplex communication protocol websocket connection mode, that is, a websocket connection.

[0040] The Conference Media Server (CMS) is used to aggregate, control, transcode, mix and synthesize collected media resources, and supports various streaming protocols.

[0041] The reserved media address specifically includes: the media resource applied for by the communication media server, which generates a media address for receiving the audio and video corresponding to the incoming signaling, including video data and audio and video stream information.

[0042] The communication media server establishes a communication connection with the conference signaling server, directly connecting the IT and CT domains. The conference signaling server, utilizing this communication connection, transmits communication information to the conference media server. This information includes information about the participant joining the conference and the media addresses reserved by the communication media server for incoming signaling audio and video. The conference signaling server synchronizes this information with the conference media server, allowing the conference media server to determine the participant's joining the video conference and the media addresses of the participant's audio and video on the communication media server.

[0043] S206: Reserve a monitoring address for audio and video through the conference media server according to the communication information, and synchronize it to the communication media server.

[0044] The monitoring address is used to push and pull the audio and video streams.

[0045] Streaming refers to the process of transmitting packaged content to a server. Specifically, the audio and video data is encapsulated into streaming data using a streaming protocol. Common streaming protocols include the Real-Time Messaging Protocol (RTMP).

[0046] Stream pulling refers to the process of establishing a connection between the communication media server and the conference media server and pulling data based on the RTMP protocol. Specifically, the communication media server pulls a single audio and video stream from the conference media server.

[0047] Optionally, the listening address includes a streaming address for streaming and a streaming address for pulling streaming.

[0048] Based on the communication information, the audio and video corresponding to the participant can be determined. The conference media server reserves a listening address for the audio and video and synchronizes it to the communication media server, allowing the conference media server and the communication media server to push and pull the audio and video streams through the listening address. The listening address can be a network resource used to push and pull the audio and video streams.

[0049] S208, when the middle number is connected, the audio and video are mixed through the conference media server to obtain a single-channel audio and video stream. According to the monitoring address, the single-channel audio and video stream is pulled through the communication media server and sent to the participants to conduct an audio and video conference based on the single-channel audio and video stream.

[0050] Audio and video streaming refers to the audio and video data corresponding to audio and video transmitted in the form of a stream over the network. This transmission method allows audio and video data to be played while being downloaded, thus achieving real-time or near real-time audio and video playback.

[0051] After S204 reserves the media address for the participant's audio and video, and S206 reserves the monitoring address for the participant's audio and video, the participant can dial the middle number to connect to the audio and video conference and enter the conference. When the middle number is connected, the audio and video are mixed, that is, the audio and video streams corresponding to the audio and video generated by the participant are mixed to obtain a mixed single audio and video stream.

[0052] It should be noted that there are one or more participants, and there can be one or more audio and video generated by the participants, that is, there are one or more audio and video streams corresponding to the audio and video generated by the participants, and the conference media server mixes the audio and video streams corresponding to the audio and video generated by the participants to obtain one mixed audio and video stream. It can be seen that the conference media server can mix multiple audio and video streams into a single audio and video stream. Among them, the first participant enters the audio and video conference through the intermediate number, and some initial resource data of the intermediate number platform will be initialized. Participants who enter the audio and video conference later do not need to initialize the data, and the remaining process is the same as the first participant, and no specific limitation is made here.

[0053] The conference media server can determine the monitoring address of the mixed single-channel audio and video stream, and the communication media server pulls the mixed single-channel audio and video stream. The communication media server sends the pulled single-channel audio and video stream to the participants through the core network, and the participants conduct an audio and video conference based on the single-channel audio and video stream.

[0054] The technical solution of the embodiment of the present application is adopted, and the incoming signaling of the participants of the audio and video conference calling the middle number is received by the signaling server and sent to the communication media server. The communication connection with the conference signaling server is established by the communication media server, and the communication information is sent to the conference media server using the conference signaling server. The communication information includes: the message of the participant joining the conference, and the media address reserved by the communication media server for the audio and video of the incoming signaling; according to the communication information, the conference media server reserves a listening address for the audio and video, and synchronizes it to the communication media server. The listening address is used to pull and push the audio and video streams of the audio and video. When the middle number is connected, the audio and video are mixed by the conference media server to obtain a single-channel audio and video stream. According to the listening address, the single-channel audio and video stream is pulled by the communication media server and sent to the participant to conduct an audio and video conference based on the single-channel audio and video stream. It can be seen that the participants of the meeting can connect to the audio and video conference by dialing the middle number. By reserving the media address and monitoring address for the audio and video corresponding to the participants, the audio and video streams corresponding to the audio and video can be pushed and pulled, allowing the participants to join the audio and video conference. Participants can enter the audio and video conference without downloading the audio and video conference software, solving the problem of low convenience in participating in audio and video conferences.

[0055] In one embodiment, after receiving an incoming signaling ...

[0056] Step A1: Notify the communication media server to reserve a media address for the audio and video of the incoming signaling.

[0057] When a participant generates call signaling via a voice or video call, the signaling server receives the signaling and sends it to the communication media server. The server then requests media resources for the incoming signaling. Upon receiving the request, the communication media server reserves media addresses for the incoming signaling's audio and video. Specifically, the communication media server pre-assigns media addresses for receiving the corresponding media information for the call. This media information includes the corresponding media codec, payload type, IP address, media format, port number, and supported audio.

[0058] Step A2: The media address is returned to the signaling server, and the media format of the participant's audio and video is determined through the signaling server.

[0059] According to the media address reserved by the communication media server in step A1, the media address is returned to the signaling server. The signaling server unifies the media format of the participants' audio and video to obtain the media format of the participants' audio and video, wherein the address used by the audio and video in the unified media format is the media address.

[0060] In this embodiment, the communication media server reserves a media address for audio and video and returns it to the signaling server. The media address is received through the signaling server, and the media format of the participants' audio and video is uniformly processed to determine the media format of the participants' audio and video. The audio and video in the unified media format can flow into the communication media server through the signaling server, and the intermediate number platform and the participants' terminals can communicate using a unified media format to the greatest extent, thereby reducing the transcoding overhead and the problem of media information synchronization.

[0061] In one embodiment, the signaling server determines the media format of the participant's audio and video (i.e., step A2), and the following step A21 may be performed:

[0062] Step A21: When the incoming signaling is a voice call, the signaling server and the participant negotiate the media format of audio and video and the video, and change the media format of audio and video to the negotiated media format.

[0063] Media formats include audio, video, images and other types, and each type has multiple formats to choose from.

[0064] When the incoming signaling is a voice call, the audio media format is formed. The signaling server and the participants negotiate the media format of the generated audio and video to generate a unified media format. The unified media format is the media format consistent with the participants and the signaling server. All participants use this media format for audio and video conferences. If there are multiple participants, each participant negotiates with the signaling server and processes the media format of the audio and video generated by the participants into the unified media format of the signaling server.

[0065] Video negotiation refers to the need to pull video negotiation to negotiate with the participants' video calls to confirm whether to enable the camera function on the participants' terminals in order to conduct an audio and video conference.

[0066] Specifically, by determining that the incoming signaling is a voice incoming signaling, the signaling server negotiates with the participant to convert the media format of the audio and video into a unified media format in the signaling server, wherein the address of the converted media format is the media address reserved by the communication media server, and video negotiation is conducted with the participant to enable the participant to turn on the video function.

[0067] Optionally, when the incoming signaling is a video call, the signaling server and participants negotiate the media format of the generated audio and video to generate a unified media format. A unified media format is a media format that is consistent between the participants and the signaling server, and all participants use this media format for audio and video conferencing. If there are multiple participants, each participant negotiates with the signaling server and processes the media format of the audio and video generated by the participants into the unified media format of the signaling server.

[0068] Specifically, by determining that the incoming signaling is a video incoming signaling, the signaling server converts the media format of the audio and video into a unified media format in the signaling server. Since the camera in the terminal of the participant in the video incoming signaling is turned on, the Service Delivery Platform (SDP) in the video incoming signaling already carries the video offer, so there is no need to add an additional process to initiate the video negotiation process. The address for converting the media format is a reserved media address.

[0069] In this embodiment, based on the incoming signaling, the signaling server and the participants conduct half-end negotiation. When it is a voice call, video negotiation is required with the participant's terminal. If it is a video call, video negotiation is not required. The signaling server and the participants negotiate the media format, and the media format of the audio and video is converted into a unified media format of the signaling server, which can reduce the transcoding overhead. The address used for negotiation adopts the media address reserved by the communication media server, which facilitates the flow of audio and video streams corresponding to the audio and video into the communication media server, thereby improving the processing efficiency of audio and video.

[0070] In one embodiment, before the communication media server establishes a communication connection with the conference signaling server (i.e., S106), the following steps B1-B2 may be performed:

[0071] Step B1: The audio and video information after the media format of the audio and video is determined is synchronized to the communication media server and stored in the media address of the communication media server.

[0072] The audio and video information after determining the media format of the audio and video, wherein the audio and video information can be media information. Since the resource address corresponding to the audio and video information is the media address reserved by the communication media server, when the signaling server synchronizes the audio and video information to the communication media server, the audio and video information can be stored in the media address of the communication media server, so that the communication media server can perform operations such as transcoding, media forwarding, and recording on the audio and video data of both parties to the call.

[0073] Step B2: Send a start message for connecting the audio and video conference to the participants through the signaling server, and send the start message to the communication media server.

[0074] The start message may include a ringing message, etc. As an example, Figure 3The figure shows a schematic diagram of media format conversion during an incoming voice call. Specifically, a participant sends a first request to a signaling server and receives a first received signaling. The first request is used to request the signaling server to receive the incoming signaling sent by the participant, and the first received signaling is used to notify the participant that the signaling server has received the incoming signaling. Interactions can be performed using codes such as: request (offer: audio data) INVITE (offer: audio sendrecv) and request (receive: audio data) 183 (answer: audio sendrecv). The signaling server receives the first request and notifies the communication media server to create an audio and video conference call for the audio and video corresponding to the incoming signaling, as well as to reserve the media addresses for the audio and video. The communication media server returns the reserved media addresses for the audio and video to the signaling server. The participant and the signaling server conduct a signaling negotiation. A first provisional response acknowledgement indicates receipt of information sent by the signaling server. The response to the first provisional response indicates confirmation of the signaling server's final response to the received message. Specific codes include: provisional response acknowledgment (PRACK) and response to the first provisional response 200-PRACK. The incoming signaling carries the media information of the participants. The signaling server receives the media information of the participants' audio and video, and synchronizes the media information and media address and other information to the communication media server. Among them, the first resource update includes converting the media format of the audio and video into media information formed by a unified media format in the signaling server, and updating it in the reserved media address. The response to the first resource update includes the participant receiving the updated media information and media address and other information. The specific codes for interaction between the participant and the signaling server include: update UPDATE, 200-UPDATE. By determining that the incoming signaling is a voice incoming signaling, there is no camera function in the voice incoming call. Therefore, the signaling server and the participant initiate video negotiation, and turn on the camera function of the participant terminal by pulling the video negotiation to conduct an audio and video conference. For example, the second resource update includes video negotiation with the participant through the signaling media server, with the purpose of turning on the camera function of the participant terminal. The response to the second resource update includes turning on the camera function of the participant terminal. The specific code may include: UPDATE (offer: audio sendrecv, video sendrecv), 200-UPDATE (offer: audio sendrecv, video sendrecv).The signaling server synchronizes the negotiated audio and video, as well as the media information and media address converted into a unified media format, to the communication media server. The call can begin, and a start message of the call is sent to the communication media server and the participant. The first ringing message is used to indicate that a ringing notification is sent to the participant to indicate that the call can begin. The response to the first request is used to indicate that the participant's initial request has been received. The first final confirmation is used to indicate that the data has been received correctly. Among them, some interaction codes between the participant and the signaling server can be expressed as: 180, 200-INVITE, and confirmation character (Acknowledge character, ACK).

[0075] As another example, Figure 4The figure shows a schematic diagram of media format conversion during an incoming video call. Specifically, a participant sends a second request to the signaling server and receives a second received signaling. The second request requests the signaling server to receive the incoming signaling sent by the participant, and the second received signaling notifies the participant that the signaling server has received the incoming signaling. This interaction can be accomplished through codes such as INVITE (offer: audio sendrecv) and 183 (answer: audio sendrecv). The signaling server receives the second request and notifies the communication media server to create an audio and video conference call for the audio and video corresponding to the incoming signaling, and to reserve a media address for receiving the audio and video. The communication media server returns the reserved audio and video media address to the signaling server. The participant and the signaling server conduct a signaling negotiation. The second provisional response acknowledges receipt of the information sent by the signaling server. The response to the second provisional response indicates confirmation of the signaling server's final response to the received message. Specific codes include PRACK and 200-PRACK. The incoming signaling carries the participant's media information. The signaling server receives the participant's audio and video media information and synchronizes the media information to the communication media server. Among them, the third resource update includes converting the media format of audio and video into media information formed by a unified media format in the signaling server, and updating it in the reserved media address. The response to the third resource update includes the participant receiving the updated media information and media address and other information. The specific codes for interaction between the participant and the signaling server include: UPDATE, 200-UPDATE. By determining that the incoming signaling is a video incoming signaling, there is no need to add additional processes to start the video negotiation process. Finally, the signaling server synchronizes the media information converted into a unified media format to the communication media server, and the call can start. The call start message is sent to the communication media server and the participant. The second ringing message is used to indicate that a ringing notification is sent to the participant to indicate that the call can start. The response to the second request is used to indicate that the participant's initial request has been received. The second final confirmation is used to indicate that the data has been received correctly. Among them, some of the interaction codes between the participant and the signaling server can be represented as: 180, 200-INVITE, ACK.

[0076] In this embodiment, the audio and video information after the media format of the audio and video is determined is synchronized to the communication media server and stored in the media address of the communication media server. The stored audio and video can be processed subsequently by the communication media server; at the same time, the signaling server sends a start message to the participants and the communication media server, which can inform the participants that they are accessing the video conference, and can notify the communication media server that network resources need to be reserved for the participants' audio and video through the conference media server.

[0077] In one embodiment, based on the communication information, a monitoring address is reserved for audio and video by the conference media server and synchronized to the communication media server (i.e., S108). Specifically, the following steps C1-C3 may be performed:

[0078] Step C1: according to the communication information sent by the conference signaling server, the conference media server starts the RTMP streaming media server including the RTMP protocol.

[0079] In RTMP streaming media servers, streaming media refers to a media form that transmits audio, video and multimedia files in a streaming manner over the network. The main function of a streaming media server is to transmit audio and video files to the client using a streaming protocol for users to watch online. It can also receive real-time audio and video streams through video capture and compression software, and then send them to the client using a streaming protocol. Streaming protocols include the RTMP protocol.

[0080] Because the communication information includes: the message of the participant joining the conference and the media address reserved by the communication media server for the participant to push the stream. After the conference signaling server sends the communication information to the conference media server, the conference media server receives the message of the participant joining the conference and determines the media address reserved by the communication media server. After that, the conference media server can start the RTMP streaming media server including the RTMP protocol.

[0081] Step C2: Based on the RTMP streaming media server, a monitoring address is reserved for audio and video.

[0082] Based on the RTMP streaming server, the conference media server can reserve listening addresses for participants to pull and push streams using the RTMP protocol within the RTMP streaming server. Specifically, the listening address includes the IP address and port number required for the pull and push processes. The listening address acts like a connection channel for pushing media data. The listening address includes both the push and pull addresses. The push address can be used by multiple participants, but the pull address remains the same.

[0083] Step C3: The conference media server returns the monitoring address to the conference signaling server, and the conference signaling server synchronizes the monitoring address to the communication media server.

[0084] The conference media server synchronizes the listening address to the conference signaling server and sends it to the communication media server so that the communication media server and the conference media server can push and pull the audio and video streams through the listening address.

[0085] In this embodiment, based on the communication information sent by the conference signaling server, the conference media server can start the RTMP streaming media server, reserve a listening address for the audio and video streams generated by the participants, and synchronize it to the communication media server, which can facilitate the subsequent communication media server and conference media server to push and pull the audio and video streams.

[0086] In one embodiment, after the conference media server reserves a monitoring address for audio and video and synchronizes it to the communication media server (i.e., S108), the following steps D1-D2 may be performed:

[0087] Step D1: Use the communication media server to send a reservation message to the signaling server.

[0088] The reservation message includes a message indicating that the media address and monitoring address have been successfully reserved for audio and video.

[0089] The communication media server is used to notify the signaling server of a successful reservation of the media resources of the reserved network for the audio and video of the participant, wherein the media resources include a media address and a monitoring address.

[0090] Step D2: According to the reservation message, it is sent to the participants through the signaling server.

[0091] The signaling server synchronizes the reservation message to the participants. For example, the signaling server returns a 200 OK status code to participant A, allowing the participant to successfully call the middle number.

[0092] In this embodiment, the communication media server sends a reservation message to the signaling server, communicates with the participants through the signaling server, and informs the participants that the call to the middle number is connected, so that the participants can confirm to join the audio and video conference and make an audio and video call.

[0093] In one embodiment, the conference media server performs mixing processing on the audio and video to obtain a single-channel audio and video stream (i.e., S108), and the following step E may be performed:

[0094] Step E: Receive audio and video through the conference media server, and mix the audio and video streams corresponding to the audio and video.

[0095] The mixing process includes processing the audio and video streams corresponding to multiple channels of audio and video into a single channel of audio and video stream.

[0096] The mixing process involves the conference media server processing audio and video streams using a multipoint control unit (MCU). Specifically, the conference media server decodes one or more received audio and video streams, re-arranges and mixes the decoded audio and video streams for video synthesis, and then performs audio mixing and resampling to generate a single audio and video stream. Finally, encoding, media format encapsulation, and synchronization are performed to generate a single audio and video stream. Because the audio and video media formats are the same, mixing and resampling are possible.

[0097] It should be noted that the communication media server pulls the mixed single-channel audio and video stream and forwards it to the participants via the core network. Furthermore, the multiple audio and video streams received by the conference media server are pushed by the communication media server via the push stream address in the listening address. The mixed single-channel audio and video stream pulled by the communication media server is pulled via the pull stream address. Since it is a single-channel audio and video stream, there is only one pull stream address.

[0098] As an example, Figure 5 Figure 2 shows a schematic diagram of the conference media server's audio and video stream processing process. Participants include participant A, participant B, and participant C; the communication media server cluster includes communication media servers 1, 2, and 3. Participant A corresponds to communication media server 1, participant B corresponds to communication media server 2, and participant C corresponds to communication media server 3. Each participant pushes an audio and video stream to the communication media server. The communication media server can also push audio and video streams to other participants. The single-channel audio and video stream pushed by the communication media server to a participant is a single-channel audio and video stream that is pulled and synthesized. After receiving the audio and video streams pushed by the participant, the communication media server pushes the participant's audio and video stream to the conference media server via the push address of the listening address in the RTMP protocol. The conference media server decodes the received audio and video stream to obtain the decoded audio and video stream. The audio and video streams are then re-arranged and mixed to form a video synthesis. The synthesized video stream is then mixed through audio mixing and resampling. The mixed multiple audio and video streams are synchronized and encoded and packaged to generate a single audio and video stream. The communication media server pulls the single-channel audio and video stream from the conference media server by listening to the pull stream address in the address.

[0099] In this embodiment, by performing mixing processing on multiple audio and video streams, such as sound mixing and video synthesis, and processing them into a single audio and video stream compatible with the core network, the transmission efficiency of audio and video can be improved and the user experience can be enhanced.

[0100] In one embodiment, before receiving an incoming signaling signal from a participant of an audio or video conference calling an intermediate number through a signaling server (i.e., S102), the following steps F1-F3 may be performed:

[0101] Step F1: Acquire the audio and video conference established by the user through the conference service front end, and send a request to create an audio and video conference to the conference signaling server.

[0102] The conference service front end refers to the form of software or web page that provides functions such as booking meetings and querying meeting information.

[0103] Step F2: Send the request to the conference media server via the conference signaling server, and use the conference media server to create a conference room.

[0104] The conference signaling server sends a request to create an audio and video conference to the conference media server. The conference media server creates a conference room and sets a scheduled time. Participants can join the audio and video conference only within the scheduled time.

[0105] Step F3: Use the conference signaling server to assign an intermediate number to the conference room, and send the intermediate number to the conference service front end for display.

[0106] The conference signaling server receives the conference room created for the participant in step F2, assigns a corresponding intermediate number to the conference room, and sends the intermediate number to the conference service front end for display. Participants of the video conference can join the audio and video conference by calling the intermediate number within the scheduled time.

[0107] It should be noted that the applied intermediate number can be reused, and there is no need to reallocate a new number or identifier every time an audio or video conference is organized. For example, if the user who organizes the audio or video conference organizes the audio or video conference within the scheduled time, when he wants to organize the audio or video conference again at another time, he can directly modify the scheduled time and join the audio or video conference by dialing the intermediate number, which has better cost-effectiveness and operational convenience. Of course, if you want to reapply for the audio or video conference to obtain the corresponding intermediate number, you can also reapply, and there is no specific limitation here.

[0108] In this embodiment, the conference service front end requests the conference signaling server to create a conference room, the conference room is created through the conference media server, and the conference signaling server allocates an intermediate number and displays it on the conference service front end. Participants can directly enter the audio and video conference by dialing the intermediate number without downloading the corresponding software for the audio and video conference, etc., saving the time and energy of users applying for audio and video conferences, improving the efficiency of convening audio and video conferences, and also improving the convenience and flexibility of participants joining audio and video conferences.

[0109] In one embodiment, the method may further perform the following steps G1-G3:

[0110] Step G1: When a participant exits the audio and video conference, a signaling server receives an exit message from the participant and sends the exit message to the communication media server.

[0111] Participants can hang up the middle number when they exit the audio or video conference.

[0112] The signaling server can receive an exit message from a participant to exit the audio and video conference, wherein the exit message from the participant to exit the audio and video conference, for example, the signaling server receives a BYE message and sends the exit message to the communication media server so that the communication media server ends the participant's call.

[0113] Step G2: Stop pushing and pulling the audio and video of the participant through the communication media server, and send a notification of the participant exiting the conference to the conference signaling server.

[0114] The communication media server receives the exit message from the signaling server indicating that the participant has exited the audio and video conference, stops pushing the participant's audio and video, and stops pulling the video from the conference media server. Simultaneously, it sends a notification to the conference signaling server indicating that the participant has exited the conference.

[0115] Step G3: The conference signaling server sends a notification of exiting the conference to the conference media server, and the conference media server removes the participant from the audio and video conference.

[0116] The conference media server removes the participant from the audio and video conference and closes the RTMP connection allocated to the participant.

[0117] In this embodiment, when a participant actively exits the audio and video conference, the signaling server receives the participant's exit message from the audio and video conference and sends it to the communication media server. The communication media server stops pushing and pulling the audio and video streams corresponding to the participant's audio and video, and notifies the participant of the synchronous exit from the conference. When the participant actively exits the conference, the communication media server can immediately stop processing the participant's audio and video streams.

[0118] In one embodiment, the method may further perform the following steps H1-H3:

[0119] Step H1: When the audio and video conference ends, a conference end notification is sent to the communication media server through the conference signaling server, and the communication media server is notified to close the audio and video conference.

[0120] When the audio and video conference ends, that is, the time for creating the conference expires or the user who applied to create the audio and video conference ends the conference, the conference signaling server sends a conference end notification to the communication media server, and the communication media server closes the audio and video conference room and stops the related RTMP connection.

[0121] Step H2: Use the communication media server to stop the push and pull processing of audio and video, and synchronize the notification of the end of the conference to the signaling server.

[0122] In step H1, the communication media server receives the notification of the end of the meeting and closes the audio and video conference, stops pushing and pulling the audio and video streams corresponding to the participants, plays the prompt tone of the end of the meeting to all online participants of the video conference, and synchronizes the notification of the end of the meeting to the signaling server.

[0123] Step H3: Send a notification of the end of the conference to the participants via the signaling server.

[0124] After receiving the notification of the end of the conference, the signaling server sends BYE to all online participants to end the call.

[0125] In this embodiment, when an audio or video conference expires, the conference signaling server sends a conference end notification to the communication media server. The communication media server then closes the audio or video conference, stops pushing and pulling audio and video streams, and synchronizes the conference end notification to the signaling server. The signaling server then sends the conference end notification to the participants. When the audio or video conference ends, the processing of the participants' audio and video streams can be immediately stopped, and the participants are notified of the conference end.

[0126] In summary, the following is a specific scenario to illustrate a method for audio and video conferencing provided by this embodiment. In this embodiment, the audio and video conferencing is applied to the intermediate number platform. Among them, the participants of the audio and video conference are A and B. Through the interaction between participant A, the conference service front end, the conference signaling server, the conference media server, the signaling server, the communication media server and participant B, participant A and participant B can join the audio and video conference. Figure 6 The embodiment shown is described as follows. Figure 6 As shown, the following steps S6.1-S6.42 are included:

[0127] S6.1: The conference service front end sends a request to the conference signaling server to create an audio and video conference.

[0128] S6.2: The conference signaling server sends a request to the conference media server to create a room corresponding to the audio and video conference.

[0129] S6.3: The conference media server creates an audio and video conference room.

[0130] S6.4: The conference media server returns the room number of the audio and video conference to the conference signaling server.

[0131] S6.5: The conference signaling server allocates an intermediate number for the audio and video conference and associates it with the room number.

[0132] S6.6: The conference signaling server returns the audio and video conference middle number to the conference service front end and displays it.

[0133] S6.7: Participant A calls the middle number of the audio and video conference.

[0134] S6.8: The signaling server sends a message to the communication media server to create an audio and video conference call.

[0135] S6.9: The communication media server allocates media resources and IP ports to the signaling server.

[0136] S6.10: Participant A and the signaling server perform half-end negotiation.

[0137] S6.11: The signaling server and the communication media server synchronize the audio and video media data and the media address of participant A, etc.

[0138] S6.12: Participant A and the signaling server send a ringing message, and the middle number can be connected.

[0139] S6.13: The signaling server sends a message to the communication media server to start the call.

[0140] S6.14: The communication media server establishes a websocket connection with the conference signaling server.

[0141] S6.15: Send a message to the conference signaling server that participant A has joined the audio and video conference with the room number in the middle, along with the media address.

[0142] S6.16: The conference signaling server sends a message to the conference media server indicating that participant A has joined the corresponding audio and video conference room.

[0143] S6.17: The conference media server starts the RTMP streaming server and returns the RTMP pull stream address and participant A's push stream address.

[0144] S6.18: The conference media server returns the monitoring address to the conference signaling server.

[0145] S6.19: The conference signaling server returns the RTMP pull stream address and push stream address to the communication media server.

[0146] S6.20: The communication media server sends a notification to the signaling server that participant A has joined the audio and video conference.

[0147] S6.21: Participant A connects to the audio and video conference, wherein media data can be transmitted between participant A and the communication conference server via the core network.

[0148] S6.22: The communication conference server pushes the audio and video stream of participant A to the conference media server via the RTMP protocol.

[0149] S6.23: The conference media server receives the audio and video streams of participant A and performs audio mixing and video synthesis using the MCU.

[0150] S6.24: The communication media server pulls the synthesized single-channel audio and video stream from the conference media server via the RTMP protocol.

[0151] S6.25: Participant B sends a call signaling to the signaling server to call the middle number.

[0152] S6.26: The signaling server sends a message to the communication media server to create an audio and video conference call.

[0153] S6.27: The communication media server allocates media resources and IP ports to the signaling server.

[0154] S6.28: Participant B and the signaling server perform half-end negotiation.

[0155] S6.29: The signaling server and the communication media server synchronize the audio and video media information and the media address of participant B.

[0156] S6.30: Participant B and the signaling server send a ringing message, and the middle number can be connected.

[0157] S6.31: The signaling server sends a message to the communication media server to start the call.

[0158] S6.32: The communication media server establishes a websocket connection with the conference signaling server.

[0159] S6.33: Send a message to the conference signaling server that participant A has joined the audio and video conference with the room number in the middle, along with the media address.

[0160] S6.34: The conference signaling server sends a message to the conference media server indicating that participant A has joined the corresponding audio and video conference room.

[0161] S6.35: The conference media server starts the RTMP streaming server and returns the RTMP pull stream address and participant B's push stream address.

[0162] S6.36: The conference media server returns the monitoring address to the conference signaling server.

[0163] S6.37: The conference signaling server returns the RTMP pull stream address and push stream address to the communication media server.

[0164] S6.38: The communication media server sends a notification to the signaling server that participant A has joined the audio and video conference.

[0165] S6.39: Participant B connects to the audio and video conference, wherein media data can be transmitted between participant B and the communication conference server through the core network.

[0166] S6.40: The communication conference server pushes the audio and video stream of participant B to the conference media server via the RTMP protocol.

[0167] S6.41: The conference media server receives the audio and video streams of participant B and performs audio mixing and video synthesis using the MCU.

[0168] S6.42: The communication conference server pulls the synthesized single-channel audio and video stream from the conference media server through the RTMP protocol.

[0169] The specific process of S6.1-S6.42 has been described in detail in the above embodiment and will not be repeated here.

[0170] In this embodiment, through interaction between participant A, the conference service front end, the conference signaling server, the conference media server, the signaling server, the communication media server, and participant B, a participant can join the audio and video conference by dialing the intermediate number. Furthermore, without changing other content of the audio and video conference, the participant can join the audio and video conference by adding the intermediate number, thereby improving the convenience of joining the audio and video conference.

[0171] like Figure 7 As shown, a swim lane diagram for ending an audio or video conference is shown, which may include the following steps S7.1-S7.14:

[0172] S7.1: When participant A exits the audio / video conference, participant A sends a hang-up notification to the signaling server.

[0173] S7.2: The signaling server sends a notification to the communication media server to stop the call of participant A.

[0174] S7.3: The communication media server sends a notification of participant A exiting the audio and video conference to the conference signaling server through the communication connection with the conference signaling server.

[0175] S7.4: When participant A exits the audio and video conference, the communication media server stops pushing the stream to participant A and stops pulling the stream from the conference media server.

[0176] S7.5: The conference signaling server sends a message to the conference media server indicating that participant A has exited the audio and video conference room.

[0177] S7.6: The conference media server removes participant A from the audio and video conference room and closes the RTMP connection allocated to participant A.

[0178] S7.7: The conference media server notifies the conference signaling server to confirm deletion of participant A.

[0179] S7.8: When an audio or video conference ends due to timeout or the initiator actively closes the conference, the audio or video conference will end.

[0180] S7.9: The communication media server stops pushing and pulling streams for participant A and plays the audio signal indicating the end of the conference to participants A and B.

[0181] It should be noted that there can be one or more participants in an audio or video conference, and there is no specific limitation.

[0182] S7.10: Notify the signaling server that the audio and video conference has ended.

[0183] S7.11: Send a notification of hang-up to the participant.

[0184] As shown in the figure, a hang-up notification is sent to participant B, and a hang-up notification is also sent to participant A at the same time.

[0185] S7.12: At the same time, the conference signaling server sends a message to the conference media server to close the room corresponding to the audio and video conference.

[0186] S7.13: The conference media server closes the room and releases the room space and related RTMP connections.

[0187] S7.14: The conference media server sends a notification confirming the close to the conference signaling server.

[0188] The specific process of S7.1-S7.14 has been described in detail in the above embodiment and will not be repeated here.

[0189] In this embodiment, by communicating with the server in the IT domain and the server in the CT domain, participants can actively exit the audio and video conference without changing the original audio and video conference information. The audio and video conference can be ended due to timeout, or the initiator of the audio and video conference can also end the audio and video conference, thereby improving the flexibility of exiting the audio and video conference.

[0190] Figure 8 is a schematic flow chart of a method for audio and video conferencing according to another embodiment of the present application, such as Figure 8 As shown, the method includes the following steps:

[0191] S801: The conference service front end sends a request for establishing an audio or video conference to the conference signaling server, and the conference signaling server sends the request to the conference media server.

[0192] S802, the conference media server creates a room number for the audio and video conference and sends the room number to the conference signaling server. The conference signaling server assigns an intermediate number to the room number and displays the intermediate number on the conference service front end.

[0193] S803, the conference participant generates incoming signaling by dialing the middle number, and sends it to the communication media server through the signaling server. The communication media server reserves a media address for the audio and video of the incoming signaling and returns the media address to the signaling server.

[0194] S804: The signaling server and the participant conduct a half-end negotiation to change the media format of the audio and video to a media format supported by the communication media server.

[0195] In the case where the incoming signaling is a voice call, a video negotiation is performed between the signaling server and the participant, and the camera of the participant's terminal is turned on for audio and video conferencing.

[0196] S805: Synchronize the media information of the audio and video in the unified media format to the communication media server, and the signaling server sends a call start message to the participant and the communication media server.

[0197] S806: The communication media server establishes a communication connection with the conference signaling server and sends communication information to the conference signaling server.

[0198] The communication information includes: the message that the participant joins the conference, and the media address reserved by the communication media server for the audio and video of the incoming signaling.

[0199] S807, the conference signaling server sends the communication information to the conference media server, and the conference media server starts the RTMP streaming server and generates a listening address using the RTMP protocol for pushing and pulling the audio and video streams.

[0200] S808, the conference signaling server synchronizes the monitoring address to the communication media server, and the communication media server notifies the signaling server that the media address and monitoring address reserved for the participant have been completed, so that the call is connected.

[0201] S809, after the middle number is connected, the audio and video generated by the participants flow into the communication media server through the core network according to the media address, and the communication media server pushes the audio and video streams generated by the participants to the conference media server through the RTMP protocol.

[0202] S810: The conference media server mixes the audio and video streams corresponding to the audio and video to obtain a single-channel audio and video stream.

[0203] S811: According to the monitoring address, the communication media server pulls a single-channel audio and video stream from the conference media server and sends it to the participant, so as to conduct an audio and video conference based on the single-channel audio and video stream.

[0204] S812, when the participant exits the audio and video conference, the communication media server stops pushing and pulling the participant's audio and video, and sends a notification of the participant exiting the conference to the conference signaling server, and removes the participant from the audio and video conference through the conference media server.

[0205] S813, when the audio and video conference ends, a conference end notification is sent to the communication media server through the conference signaling server and synchronized to the signaling server, and the conference end notification is sent to the participants through the signaling server.

[0206] Among them, the communication media server will close the audio and video conference and stop pushing and pulling audio and video streams.

[0207] The specific process from S801 to S813 has been described in detail in the above embodiment and will not be repeated here.

[0208] The technical solution of the embodiment of the present application is adopted, and the incoming signaling of the participants of the audio and video conference calling the middle number is received by the signaling server and sent to the communication media server. The communication connection with the conference signaling server is established by the communication media server, and the communication information is sent to the conference media server using the conference signaling server. The communication information includes: the message of the participant joining the conference, and the media address reserved by the communication media server for the audio and video of the incoming signaling; according to the communication information, the conference media server reserves a listening address for the audio and video, and synchronizes it to the communication media server. The listening address is used to pull and push the audio and video streams of the audio and video. When the middle number is connected, the audio and video are mixed by the conference media server to obtain a single-channel audio and video stream. According to the listening address, the single-channel audio and video stream is pulled by the communication media server and sent to the participant to conduct an audio and video conference based on the single-channel audio and video stream. It can be seen that the participants of the meeting can connect to the audio and video conference by dialing the middle number. By reserving the media address and monitoring address for the audio and video corresponding to the participants, the audio and video streams corresponding to the audio and video can be pushed and pulled, allowing the participants to join the audio and video conference. Participants can enter the audio and video conference without downloading the audio and video conference software, solving the problem of low convenience in participating in audio and video conferences.

[0209] In summary, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

[0210] The above is a method for audio and video conferencing provided in an embodiment of the present application. Based on the same idea, an embodiment of the present application also provides a device for audio and video conferencing.

[0211] Figure 9 FIG is a structural diagram of an audio and video conferencing device according to an embodiment of the present invention. Figure 9 As shown, the apparatus for audio and video conferencing includes: a receiving module 91, a creating module 92, a monitoring module 93, and a processing module 94:

[0212] Receiving module 91, used to receive incoming signaling from a participant of an audio or video conference calling an intermediate number through a signaling server, and send it to a communication media server;

[0213] A creation module 92 is configured to establish a communication connection with the conference signaling server through the communication media server, and to send communication information to the conference media server using the conference signaling server, the communication information including: a message indicating that the participant has joined the conference, and a media address reserved by the communication media server for audio and video of the incoming signaling;

[0214] The monitoring module 93 is used to reserve a monitoring address for audio and video through the conference media server according to the communication information and synchronize it to the communication media server. The monitoring address is used to push and pull the audio and video streams of the audio and video;

[0215] The processing module 94 is used to mix the audio and video through the conference media server when the intermediate number is connected to obtain a single-channel audio and video stream. According to the monitoring address, the single-channel audio and video stream is pulled through the communication media server and sent to the participants to conduct an audio and video conference based on the single-channel audio and video stream.

[0216] In one embodiment, the apparatus further comprises:

[0217] Notification module 95, used to notify the communication media server to reserve media addresses for audio and video of incoming signaling;

[0218] The determination module 96 is configured to return the media address to the signaling server, and determine the media format of the participant's audio and video through the signaling server.

[0219] In one embodiment, the determination module 96 is specifically configured to negotiate the audio and video media formats and the video media formats through the signaling server and the participant when the incoming signaling is a voice call, and change the audio and video media formats to the negotiated media formats.

[0220] In one embodiment, the apparatus further comprises:

[0221] Synchronization module 97, for synchronizing the audio and video information after determining the media format of the audio and video to the communication media server and storing it in the media address of the communication media server;

[0222] The first sending module 98 is configured to send a start message for connecting the audio and video conference to the participants via the signaling server, and send the start message to the communication media server.

[0223] In one embodiment, the monitoring module 93 is specifically configured to:

[0224] According to the communication information sent by the conference signaling server, the conference media server starts the RTMP streaming media server including the RTMP protocol;

[0225] Based on RTMP streaming media server, a monitoring address is reserved for audio and video;

[0226] The monitoring address is returned to the conference signaling server through the conference media server, and the monitoring address is synchronized to the communication media server through the conference signaling server.

[0227] In one embodiment, the apparatus further comprises:

[0228] The second sending module 99 is used to use the communication media server to send a reservation message to the signaling server, where the reservation message includes a message indicating that the media address and the monitoring address for the audio and video are successfully reserved;

[0229] The third sending module 100 is configured to send the reservation message to the participant via the signaling server.

[0230] In one embodiment, the processing module 94 is specifically configured to receive audio and video through a conference media server and perform mixing processing on the audio and video streams corresponding to the audio and video. The mixing processing includes processing the audio and video streams corresponding to multiple channels of audio and video into a single channel of audio and video stream.

[0231] In one embodiment, the apparatus further comprises:

[0232] The acquisition module 101 is used to obtain the audio and video conference established by the user through the conference service front end and send a request to create an audio and video conference to the conference signaling server;

[0233] A creation module 102 is configured to send a request to a conference media server via a conference signaling server, and create a conference room using the conference media server;

[0234] The allocation module 103 is used to allocate an intermediate number to the conference room using the conference signaling server, and send the intermediate number to the conference service front end for display.

[0235] In one embodiment, the apparatus further comprises:

[0236] The exit module 104 is used to receive the exit message of the participant exiting the audio and video conference through the signaling server when the participant exits the audio and video conference, and send the exit message to the communication media server;

[0237] The first notification module 105 is used to stop the push and pull streaming of the participant's audio and video through the communication media server, and send a notification of the participant's exit from the conference to the conference signaling server;

[0238] The removal module 106 is configured to use the conference signaling server to send a notification of exiting the conference to the conference media server, and remove the participant from the audio and video conference via the conference media server.

[0239] In one embodiment, the apparatus further comprises:

[0240] End module 107, used for sending a conference end notification to the communication media server through the conference signaling server when the audio and video conference ends, and notifying the communication media server to close the audio and video conference;

[0241] The second notification module 108 is used to stop the push and pull of audio and video streams using the communication media server and synchronize the notification of the end of the conference to the signaling server;

[0242] The sending module 109 is configured to send a notification of the end of the conference to the participants via the signaling server.

[0243] The technical solution of the embodiment of the present application is adopted, and the incoming signaling of the participants of the audio and video conference calling the middle number is received by the signaling server and sent to the communication media server. The communication connection with the conference signaling server is established by the communication media server, and the communication information is sent to the conference media server using the conference signaling server. The communication information includes: the message of the participant joining the conference, and the media address reserved by the communication media server for the audio and video of the incoming signaling; according to the communication information, the conference media server reserves a listening address for the audio and video, and synchronizes it to the communication media server. The listening address is used to pull and push the audio and video streams of the audio and video. When the middle number is connected, the audio and video are mixed by the conference media server to obtain a single-channel audio and video stream. According to the listening address, the single-channel audio and video stream is pulled by the communication media server and sent to the participant to conduct an audio and video conference based on the single-channel audio and video stream. It can be seen that the participants of the meeting can connect to the audio and video conference by dialing the middle number. By reserving the media address and monitoring address for the audio and video corresponding to the participants, the audio and video streams corresponding to the audio and video can be pushed and pulled, allowing the participants to join the audio and video conference. Participants can enter the audio and video conference without downloading the audio and video conference software, solving the problem of low convenience in participating in audio and video conferences.

[0244] Those skilled in the art should understand that Figure 9 The audio and video conferencing device can be used to implement the audio and video conferencing method described above. The detailed description should be similar to the description of the method part above. To avoid tediousness, it will not be repeated here.

[0245] Based on the same technical concept, an embodiment of the present application further provides an electronic device for executing the above-mentioned audio and video conferencing method. Figure 10The following is a schematic diagram of the structure of an electronic device for implementing various embodiments of the present application. Electronic devices may vary significantly due to different configurations or performances, and may include a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040. The processor 1010, the communications interface 1020, and the memory 1030 communicate with each other via the communication bus 1040. The processor 1010 may call a computer program stored in the memory 1030 and executable on the processor 1010 to perform the following steps:

[0246] Receive incoming signaling from a participant in an audio or video conference calling a middle number through the signaling server and send it to the communication media server;

[0247] Establishing a communication connection with the conference signaling server through the communication media server, and using the conference signaling server to send communication information to the conference media server, the communication information includes: a message that the participant has joined the conference, and a media address reserved by the communication media server for the audio and video of the incoming signaling;

[0248] According to the communication information, the conference media server reserves a monitoring address for audio and video and synchronizes it to the communication media server. The monitoring address is used to push and pull the audio and video streams of the audio and video;

[0249] When the middle number is connected, the audio and video are mixed through the conference media server to obtain a single-channel audio and video stream. According to the monitoring address, the single-channel audio and video stream is pulled through the communication media server and sent to the participants to conduct an audio and video conference based on the single-channel audio and video stream.

[0250] The technical solution of the embodiment of the present application is adopted, and the incoming signaling of the participants of the audio and video conference calling the middle number is received by the signaling server and sent to the communication media server. The communication connection with the conference signaling server is established by the communication media server, and the communication information is sent to the conference media server using the conference signaling server. The communication information includes: the message of the participant joining the conference, and the media address reserved by the communication media server for the audio and video of the incoming signaling; according to the communication information, the conference media server reserves a listening address for the audio and video, and synchronizes it to the communication media server. The listening address is used to pull and push the audio and video streams of the audio and video. When the middle number is connected, the audio and video are mixed by the conference media server to obtain a single-channel audio and video stream. According to the listening address, the single-channel audio and video stream is pulled by the communication media server and sent to the participant to conduct an audio and video conference based on the single-channel audio and video stream. It can be seen that the participants of the meeting can connect to the audio and video conference by dialing the middle number. By reserving the media address and monitoring address for the audio and video corresponding to the participants, the audio and video streams corresponding to the audio and video can be pushed and pulled, allowing the participants to join the audio and video conference. Participants can enter the audio and video conference without downloading the audio and video conference software, solving the problem of low convenience in participating in audio and video conferences.

[0251] The specific execution steps can refer to the various steps of the above-mentioned audio and video conferencing method embodiment, and can achieve the same technical effect. To avoid repetition, they will not be described here.

[0252] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals, or other devices other than terminals.

[0253] The above electronic device structure does not constitute a limitation of the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. to configure the display panel. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include but are not limited to a physical keyboard, function keys (such as volume control buttons, switch buttons, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0254] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus random access memory (DRRAM).

[0255] The processor may include one or more processing units; optionally, the processor may integrate an application processor and a modem processor, wherein the application processor primarily handles operations related to the operating system, user interface, and application programs, and the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into the processor.

[0256] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned audio and video conferencing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0257] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0258] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned audio and video conferencing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0259] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0260] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the processor is used to run the program or instructions to implement the various processes of the above-mentioned product recommendation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0261] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0262] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0263] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for audio and video conferencing, characterized in that: The method comprises: Receive incoming signaling from a participant in an audio or video conference calling a middle number through the signaling server and send it to the communication media server; Establishing a communication connection with a conference signaling server through the communication media server, and using the conference signaling server to send communication information to the conference media server, the communication information including: a message of the participant joining the conference, and a media address reserved by the communication media server for the audio and video of the incoming signaling; According to the communication information, a monitoring address is reserved for the audio and video through the conference media server and synchronized to the communication media server, and the monitoring address is used to push and pull the audio and video streams of the audio and video; When the intermediate number is connected, the audio and video are mixed by the conference media server to obtain a single-channel audio and video stream. According to the monitoring address, the single-channel audio and video stream is pulled by the communication media server and sent to the participant to conduct the audio and video conference based on the single-channel audio and video stream.

2. The method according to claim 1, characterized in that After receiving the incoming signaling of the participant of the audio and video conference calling the intermediate number through the signaling server and sending it to the communication media server, it also includes: Notifying the communication media server to reserve the media address for the audio and video of the incoming signaling; The media address is returned to the signaling server, and the media format of the audio and video of the participant is determined by the signaling server.

3. The method according to claim 2, characterized in that The determining, by the signaling server, the media format of the audio and video of the participant includes: In the case that the incoming signaling is a voice call, the signaling server and the participant negotiate the media format of the audio and video and the video, and change the media format of the audio and video to the negotiated media format.

4. The method according to claim 2, characterized in that Before establishing a communication connection with a conference signaling server according to the communication media server, the method includes: Synchronizing the audio and video information after determining the media format of the audio and video to the communication media server and storing it in the media address of the communication media server; A start message for connecting the audio and video conference is sent to the participant via the signaling server, and the start message is sent to the communication media server.

5. The method according to claim 1, wherein The method of reserving a monitoring address for the audio and video through the conference media server according to the communication information and synchronizing the address to the communication media server includes: According to the communication information sent by the conference signaling server, starting the RTMP streaming media server including the Real Time Messaging Protocol RTMP protocol through the conference media server; Based on the RTMP streaming media server, reserve the monitoring address for the audio and video; The conference media server returns the monitoring address to the conference signaling server, and the conference signaling server synchronizes the monitoring address to the communication media server.

6. The method according to claim 1, characterized in that After reserving a monitoring address for the audio and video by the conference media server, the method includes: Using the communication media server to send a reservation message to the signaling server, the reservation message including a message that the media address and the monitoring address are successfully reserved for the audio and video; According to the reservation message, it is sent to the participant through the signaling server.

7. The method according to claim 1, characterized in that The mixing of the audio and video by the conference media server to obtain a single-channel audio and video stream includes: The audio and video are received by the conference media server, and audio and video streams corresponding to the audio and video are mixed, wherein the mixing includes processing multiple audio and video streams corresponding to the audio and video into the single audio and video stream.

8. The method according to claim 1, characterized in that Before the signaling server receives an incoming signaling signal from a participant of an audio or video conference calling an intermediate number, the method includes: Acquire the audio and video conference established by the user through the conference service front end, and send a request to the conference signaling server to create the audio and video conference; Sending the request to the conference media server through the conference signaling server, and creating a conference room using the conference media server; The conference signaling server is used to allocate the intermediate number to the conference room, and the intermediate number is sent to the conference service front end for display.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: When the participant exits the audio and video conference, receiving an exit message of the participant exiting the audio and video conference through the signaling server, and sending the exit message to the communication media server; Stopping push and pull streaming of the audio and video of the participant through the communication media server, and sending a notification of the participant exiting the conference to the conference signaling server; The conference signaling server is used to send the notification of exiting the conference to the conference media server, and the participant is removed from the audio and video conference through the conference media server.

10. The method according to any one of claims 1 to 8, characterized in that The method further comprises: When the audio and video conference ends, sending a conference end notification to the communication media server through the conference signaling server, and notifying the communication media server to close the audio and video conference; Using the communication media server to stop the push and pull processing of the audio and video, and synchronizing the notification of the end of the conference to the signaling server; A notification of the end of the conference is sent to the participants via the signaling server.

11. An audio and video conferencing device, characterized in that: The device comprises: A receiving module is used to receive incoming signaling from a participant of an audio or video conference calling an intermediate number through a signaling server and send the signaling to a communication media server; A creation module, configured to establish a communication connection with a conference signaling server through the communication media server, and to send communication information to the conference media server using the conference signaling server, wherein the communication information includes: a message indicating that the participant has joined the conference, and a media address reserved by the communication media server for the audio and video of the incoming signaling; A monitoring module is used to reserve a monitoring address for the audio and video through the conference media server according to the communication information, and synchronize it to the communication media server, wherein the monitoring address is used to push and pull the audio and video streams of the audio and video; A processing module is used to mix the audio and video through the conference media server to obtain a single-channel audio and video stream when the intermediate number is connected. According to the monitoring address, the single-channel audio and video stream is pulled through the communication media server and sent to the participant to conduct the audio and video conference based on the single-channel audio and video stream.

12. An electronic device, characterized in that: It includes a processor and a memory electrically connected to the processor, the memory stores a computer program, and the processor is used to call and execute the computer program from the memory to implement the audio and video conferencing method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The storage medium is used to store a computer program, and the computer program can be executed by a processor to implement the audio and video conferencing method according to any one of claims 1 to 10.

Images