Method, device and equipment for recording audio in video conference

Abstract

The embodiment of the invention provides a method, device and equipment for recording audio in a video conference, and belongs to the field of video conferences, the method is applied to a client, and the method comprises the following steps: sending an audio recording request message to a media forwarding server, the audio recording request message is used for indicating the media forwarding server to be converted from an audio routing forwarding mode to an audio recording mode; receiving audio data through a routing audio receiving channel corresponding to an audio routing forwarding mode, switching to subscription audio receiving channels in one-to-one correspondence with consumers subscribed by the client, and receiving the audio data of the consumers subscribed by the client and sent by the media forwarding server; and recording the received audio data to obtain a recorded audio. According to the embodiment of the invention, the problem that local audio recording is missing can be solved.

Description

Technical Field

[0001] The present application relates to the field of video conferencing, and in particular, to a method, apparatus and device for recording audio in a video conferencing. Background Art

[0002] In video conferencing scenarios, in order to meet users' needs for offline playback of conference content, local recording of audio and video streams is a basic requirement.

[0003] Many existing video conferencing products are developed based on the Selective Forwarding Unit (SFU) architecture. The biggest feature of the SFU architecture is that the media server only forwards audio and video streams and does not decode or encode media data. Therefore, the SFU architecture can not only meet the publishing, subscription, and forwarding functions of audio and video streams in multi-person video conferencing, but also greatly reduce transmission delays and improve user experience.

[0004] However, when the existing video conferences for SFU architecture are recorded locally, the audio of some participants subscribed by the customer is usually missing, which reduces the effect of conference recording and affects the user experience.

[0005] Therefore, how to provide a video recording method that can solve the situation where local audio recording is missing becomes a technical problem that needs to be solved. Summary of the invention

[0006] An embodiment of the present application aims to provide a method, apparatus and device for recording audio in a video conference. The technical solution of the embodiment of the present application can solve the problem of missing local audio recording and improve the user experience.

[0007] In a first aspect, an embodiment of the present application provides a method for recording audio in a video conference, which is applied to a client, and the method includes: sending an audio recording request message to a media forwarding server, wherein the audio recording request message is used to instruct the media forwarding server to switch from an audio routing forwarding mode to an audio recording mode; receiving audio data through a routing audio receiving channel corresponding to the audio routing forwarding mode, switching to a subscription audio receiving channel corresponding one by one to consumers subscribed by the client, and receiving audio data of the consumers subscribed by the client sent by the media forwarding server; and recording the received audio data to obtain recorded audio.

[0008] The embodiment of the present application receives the audio data of the consumers subscribed by the client sent by the media forwarding server through the subscription audio receiving channel corresponding one by one to the consumers subscribed by the client, and can receive all the subscribed audio, thereby solving the problem in the prior art that only part of the audio can be received, resulting in the loss of local recorded audio.

[0009] In one implementation, the method further includes: performing routing and mixing on the received audio data to obtain and play the mixed audio.

[0010] In one embodiment, the audio data is received by the routing audio receiving channel corresponding to the audio routing forwarding mode, switched to the audio receiving channel corresponding one-to-one to the consumers subscribed by the client, and the audio data of the consumers subscribed by the client sent by the media forwarding server includes: creating a subscription audio receiving channel corresponding one-to-one to the consumers subscribed by the client; pausing the reception of audio data through the routing audio receiving channel, and starting the reception of audio data by the subscription audio receiving channel.

[0011] In one embodiment, the method further includes: copying the audio data packets and the decoded data after decoding in the cache of the selected audio receiving channel to the cache of the subscribed audio receiving channel; and / or copying the negative confirmation request message in the cache of the selected audio receiving channel to the cache of the subscribed audio receiving channel.

[0012] In one embodiment, the method also includes: copying the correspondence between the real-time transport protocol RTP timestamp and the NTP timestamp in the cache of the selected audio receiving channel to the cache of the subscribed audio receiving channel; and synchronizing the audio data and video data received by the selected audio receiving channel using the correspondence between the RTP timestamp and the NTP timestamp.

[0013] In one embodiment, the method also includes: sending a request message to close audio recording to a media forwarding server, wherein the request message to close audio recording is used to instruct the media forwarding server to switch from an audio recording mode to an audio routing forwarding mode; receiving audio data of consumers subscribed by the client sent by the media forwarding server through a subscription audio receiving channel corresponding to each consumer subscribed by the client, and switching to receiving audio data through a routing audio receiving channel corresponding to the audio routing forwarding mode.

[0014] In a second aspect, an embodiment of the present application provides a method for recording audio in a video conference, which is applied to a media forwarding server, and the method includes: receiving an audio recording request message sent by a client, the audio recording request message is used to instruct the media forwarding server to switch from an audio routing forwarding mode to an audio recording mode; transmitting audio data by an active speaking list consumer corresponding to the audio routing forwarding mode, switching to an ordinary consumer corresponding one-to-one to the consumers subscribed by the client, and transmitting the audio data of the consumers subscribed by the client.

[0015] In the third aspect, an embodiment of the present application provides a device for recording audio in a video conference, which is applied to a client, and the device includes: a sending unit, which is used to send an audio recording request message to a media forwarding server, and the audio recording request message is used to instruct the media forwarding server to switch from an audio routing forwarding mode to an audio recording mode; a processing unit, which is used to receive audio data from a routing audio receiving channel corresponding to the audio routing forwarding mode, switch to a subscription audio receiving channel corresponding one by one to consumers subscribed by the client, and receive the audio data of the consumers subscribed by the client sent by the media forwarding server; record the received audio data to obtain recorded audio.

[0016] In one implementation, the processing unit is further configured to perform routing and mixing on the received audio data to obtain and play the mixed audio.

[0017] In one embodiment, the processing unit is specifically used to create a subscription audio receiving channel corresponding to the consumers subscribed by the client; suspend receiving audio data through the selected audio receiving channel, and start receiving audio data through the subscription audio receiving channel.

[0018] In one embodiment, the processing unit is also used to copy the audio data packets and the decoded data after decoding in the cache of the selected audio receiving channel to the cache of the subscribed audio receiving channel; and / or, copy the negative confirmation request message in the cache of the selected audio receiving channel to the cache of the subscribed audio receiving channel.

[0019] In one embodiment, the processing unit is also used to copy the correspondence between the real-time transport protocol RTP timestamp and the NTP timestamp in the cache of the selected audio receiving channel to the cache of the subscribed audio receiving channel; and use the correspondence between the RTP timestamp and the NTP timestamp to synchronize the audio data and video data received by the selected audio receiving channel.

[0020] In one embodiment, the sending unit is also used to send a request message to close audio recording to the media forwarding server, and the request message to close audio recording is used to instruct the media forwarding server to switch from an audio recording mode to an audio routing forwarding mode; the processing unit is also used to receive the audio data of the consumers subscribed by the client sent by the media forwarding server through a subscription audio receiving channel corresponding to the consumers subscribed by the client, and switch to receiving audio data through a routing audio receiving channel corresponding to the audio routing forwarding mode.

[0021] In a fourth aspect, an embodiment of the present application provides a device for recording audio in a video conference, which is applied to a media forwarding server, and the device includes: a receiving unit, which is used to receive an audio recording request message sent by a client, and the audio recording request message is used to instruct the media forwarding server to switch from an audio routing forwarding mode to an audio recording mode; a processing unit, which is used to transmit audio data from an active speaking list consumer corresponding to the audio routing forwarding mode, switch to an ordinary consumer corresponding one-to-one to the consumers subscribed by the client, and transmit the audio data of the consumers subscribed by the client.

[0022] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in the first aspect and any embodiment of the first aspect or the method described in the second aspect, that is, any embodiment of the second aspect.

[0023] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, it can implement the method described in the first aspect and any embodiment of the first aspect or the method described in the second aspect, i.e. any embodiment of the second aspect.

[0024] In the seventh aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, wherein when the computer program is executed by a processor, it can implement the method described in the first aspect and any embodiment of the first aspect or the method described in the second aspect, that is, any embodiment of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solution of an embodiment of the present application, the drawings required for use in an embodiment of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A schematic diagram of a video conferencing system scenario provided for an embodiment of the present application;

[0027] Figure 2 A flow chart of a method for recording audio in a video conference provided for one embodiment of the present application;

[0028] Figure 3 A schematic diagram of an audio routing forwarding mode provided for an embodiment of the present application;

[0029] Figure 4 A schematic diagram of an audio recording mode provided for an embodiment of the present application;

[0030] Figure 5 A schematic diagram of a device for recording audio in a video conference provided in accordance with an embodiment of the present application;

[0031] Figure 6 A schematic diagram of a device for recording audio in a video conference provided in accordance with another embodiment of the present application;

[0032] Figure 7 A schematic diagram of an electronic device is provided for one embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0034] When local audio recording is performed on existing SFU-based video conferences, the audio of some participants subscribed by the customer is usually missing, which reduces the effect of conference recording and affects the user experience.

[0035] Specifically, in terms of forwarding audio streams, according to actual experience, if more than a certain number of participants (usually 3) speak at the same time in a meeting, the voices heard by other participants will be very noisy, and the participants with quieter voices will often not be heard. Therefore, existing video conferencing media servers (such as SFU) often perform audio routing, selecting the channels (usually 3 channels) with the largest volume (energy) from all received audio streams for forwarding, and not forwarding other audio data. Based on this, the client audio and video engine will create a fixed number (usually 3 channels) of audio receiving channels to receive the audio streams with the largest volume.

[0036] Since the local recording function requires a specified subscription to receive and record the audio of other participants, in the scenario where the route is selected based on the volume, it is impossible to ensure that the audio data of the specified participant will be received. The reason is that the audio data of the specified participant may not be one of the channels with the highest volume, and therefore may not be forwarded.

[0037] In view of the above problems, the present application provides a method for local recording of a video conference, which receives the audio data of the consumers subscribed by the client sent by the media forwarding server through the one-to-one subscription audio receiving channel of the consumers subscribed by the client, and records it, which can solve the problem of missing local recorded audio in the prior art. Specifically, the embodiment of the present application designs a method for turning on and off local recording in a video conference, and is compatible with the existing video conference audio routing architecture.

[0038] In order to facilitate understanding and description, the following describes the execution process and actions of the real-time communication method of the present application in a real-time communication system as an example rather than a limitation.

[0039] The following is combined with Figure 1 The overall structure of a video conferencing system provided by an embodiment of the present application is exemplified.

[0040] like Figure 1 As shown, an embodiment of the present application provides a video conferencing system including: multiple clients, a control terminal and a media forwarding server.

[0041] In the embodiment of the present application, the control end may also be referred to as the management end or the signaling server. The control end is used to perform signaling interaction with the client to transmit the necessary information required for establishing a connection between the clients. At the same time, the control end is used to obtain the subscription relationship of the client, and send instructions to the client and the SFU server to determine the producer and consumer based on the subscription relationship. The role of the control end in the embodiment of the present application is to act as an intermediary to help both parties establish a connection with as little privacy exposure as possible. The communication method between the control end and the client and the SFU server can use any method, for example, WebSocket or XMLHttpRequest, etc. to exchange each other's token information. The embodiment of the present application is not limited to this.

[0042] In an embodiment of the present application, the media forwarding server may be an SFU server, such as Mediasoup, Janus, Jitsi, Kurento or Medooze, etc. The embodiment of the present application is not limited thereto, and the following description will only take the SFU server as Mediasoup as an example. The SFU server may accept instructions from the control terminal to manage and control producer and consumer objects, and may then decide whether to receive and / or send audio and / or video streams of real-time communications of the client. It should be understood that the SFU server may also be called an SFU or a forwarding server, and if not otherwise specified below, the several names of the SFU server may be replaced, and the embodiment of the present application does not limit this. It should be understood that in the present application, the control terminal and the SFU may be two independent devices, or they may be integrated into one device.

[0043] It should be understood that the client in the embodiment of the present application can also be called a user end or a user device or a terminal device. In the real-time example of the present application, the terminal device can be installed with a browser, and real-time communication can be performed through the browser, or an APP or a small program can be installed, and real-time communication can be performed through the APP or the small program. The terminal device in the present application can include a smart phone, a tablet computer, (personal digital assistant, PDA personal digital assistant), a computer, a game console, a wearable device, a tablet computer (portable android device, PAD), etc., and the embodiment of the present application is not limited thereto.

[0044] It should be understood that Figure 1 In the video conferencing scenario shown in , only two clients are illustrated, but the embodiments of the present application are not limited to this. In an actual real-time communication scenario, the number of clients may include any number, depending on the actual situation.

[0045] The following is combined with Figure 2 A method for recording audio in a video conference provided by an embodiment of the present application is exemplified.

[0046] like Figure 2 The method shown is applied to Figure 1 In the video conferencing system shown, Figure 2 The methods shown include:

[0047] 210. The client sends an audio recording request message to the media forwarding server.

[0048] Correspondingly, the media forwarding server receives the audio recording request message sent by the client, wherein the audio recording request message is used to instruct the media forwarding server to switch from the audio routing forwarding mode to the audio recording mode.

[0049] For example, when the local recording function is started by command in the meeting, such as by clicking the local recording button or by voice command, the client will notify the media forwarding server to switch the audio subscription mode through signaling. Optionally, the audio recording mode in the embodiment of the present application can also be called the audio full forwarding mode or the audio subscription mode, and the embodiment of the present application is not limited thereto.

[0050] 220, the media forwarding server switches the audio data transmitted by the active speaking list consumers corresponding to the audio routing forwarding mode to the common consumers corresponding one-to-one to the consumers subscribed by the client, and transmits the audio data of the consumers subscribed by the client.

[0051] Correspondingly, the client receives audio data through the routing audio receiving channel corresponding to the audio routing forwarding mode, switches to the subscription audio receiving channel corresponding to the consumers subscribed by the client, and receives the audio data of the consumers subscribed by the client sent by the media forwarding server. Among them, the above-mentioned active speaking list consumers correspond to the routing audio receiving channel, and the ordinary consumers correspond to the subscription audio receiving channel.

[0052] Optionally, as another embodiment, the audio data is received by the routing audio receiving channel corresponding to the audio routing forwarding mode, and the audio receiving channel corresponding to the consumers subscribed by the client is switched to receive the audio data of the consumers subscribed by the client sent by the media forwarding server, including:

[0053] A subscription audio receiving channel corresponding to each consumer subscribed by the client is created; receiving audio data through the routing audio receiving channel is suspended, and receiving audio data through the subscription audio receiving channel is started.

[0054] For example, the SFU server can first create a full set of ordinary consumers Simple Consumer, and through the interaction with the client through SDP, the client creates a full set of Audio tracks (corresponding to the routing audio receiving channel mentioned above); after the forwarding audio module of the SFU server and the receiving audio module of the client are created, the SFU server forwards the audio stream produced by the Producer to the corresponding Simple Consumer, and no longer forwards it to the audio routing module, and at the same time sets the ASL Consumer (corresponding to the active speaking list consumer mentioned above) status to pause, and the client also sets the local three ASL Audio tracks (corresponding to the routing audio receiving channel mentioned above) status to pause to reduce the client performance overhead.

[0055] For example, Figure 3As shown, clients A, B, C, D, and E join the same conference. When the local client is E and the local client subscribes to AD, clients AD correspond to the producers of the audio of consumer E. In the audio routing architecture, AD publishes audio data to the SFU server. The audio routing module of the SFU server selects the channels with the largest volume from the 4 channels of AD, for example, 3 channels, and forwards them to the 3 audio receiving channels (Audio track) of E through the active speaking list consumers (ASL Consumer1-ASL Consumer13). After receiving the audio, the local client E performs audio mixing and audio playback.

[0056] When client E in the conference turns on the local recording function, it can notify the SFU server through signaling to switch to audio recording mode (also called audio subscription mode). The data transmission process between the server and the client after the switch is as follows: Figure 4 shown.

[0057] After local recording is enabled, the SFU server creates a full set of simple consumers, and client E creates a full set of audio receiving channels (Audio track) to receive and decode each audio stream. At the same time, the producer (Producer) of the SFU server no longer forwards audio data packets to the audio routing module, the active speaker list consumer (ASLConsumer) stops working, and client E stops receiving audio data packets through the active speaker audio channel (ASL Audio track) corresponding to the active speaker list consumer, and no longer obtains pulse code modulation data (Pulse Code Modulation, PCM) data from the ASL Audio track. Correspondingly, the data stream is composed of Figure 4 The corresponding dotted part switches to Figure 4 The solid line part in the corresponding position.

[0058] 230. The client records the received audio data to obtain recorded audio.

[0059] Specifically, after receiving the audio data, the client performs corresponding local recording and storage to obtain the recorded audio.

[0060] The embodiment of the present application receives the audio data of the consumers subscribed by the client sent by the media forwarding server through the subscription audio receiving channel corresponding one by one to the consumers subscribed by the client, and can receive all the subscribed audio, thereby solving the problem in the prior art that only part of the audio can be received, resulting in the loss of local recorded audio.

[0061] It should be understood that the recording files in the embodiments of the present application may include the following three forms: (1) separate audio files and video files of other participants in the meeting, (2) audio and video files after video screen mixing and audio mixing of other participants in the meeting (3) audio and video files after the local audio and video stream is mixed with the audio and video streams of other participants in the meeting. It should be understood that the recording scheme mentioned in the embodiments of the present application above is only the recording scheme of the audio described. For the recording of the video, the embodiments of the present application can be forwarded by the media server according to the specific subscription situation of the client, and the client can record. For example, the client subscribes to 9 users, and the media server actually forwards all the video images of the 9 users. Only for audio, the existing media server only forwards the audio of the users with the loudest volume among the 9 users. Therefore, for video recording, since the video images forwarded by the media forwarding server are consistent with the subscription situation of the client, there will be no problems in audio recording. For the above three forms of recording files, the above scheme can be adopted for audio files, and the existing video recording method can be adopted for video files. The embodiments of the present application are not limited to this.

[0062] Optionally, as another embodiment, the method may further include: the client performs routing and mixing on the received audio data, and obtains and plays the mixed audio.

[0063] Specifically, since the media forwarding server forwards the audio of all users subscribed by the user, in order to allow the local user to hear the conference audio clearly, the client (i.e., the receiving end) in the embodiment of the present application will select a route when playing the audio and only play the audio with the highest volume. For example, after receiving the audio packet, the client will decode the audio into pulse code modulation (PCM) data, calculate the audio volume, and the client will only play the audio with the highest volume.

[0064] For example, Figure 4 As shown, after receiving 4 channels of audio, client E first performs client audio routing, for example, selects the three channels of audio with the loudest volume, then mixes the selected audio, and then plays the audio.

[0065] Optionally, as another embodiment, in the embodiment of the present application, although the client subscribes to multiple users in the video conference, only the audio of the users with the loudest volume is played during playback. Therefore, in the embodiment of the present application, when the client plays audio, the volume playback status or icon can also be displayed in the corresponding user display window on the video conference display interface of the client. For example, the volume can be indicated by the color depth of the microphone icon, and the microphone of the user whose audio is not played can be displayed in gray or other icons used to indicate that it is not played.

[0066] Specifically, the client can receive voice packets through the UDP socket, parse the RTP packets and put them into the jitter buffer. In the embodiment of the present application, only the RTP packets with the largest audio can be put into the cache. When playing, the packets are taken out from the jitter buffer at regular intervals and decoded to obtain PCM data, which are then sent to the player for playback after post-processing.

[0067] It should be noted that, since the human ear is extremely sensitive to the continuity of sound, when the client switches from receiving the same audio data from the ASL Audiotrack to the Audio track, the embodiment of the present application can also add a solution to achieve continuous and smooth switching of sound quality. For example, the information of the original track can be copied to the new track, and corresponding audio processing can be performed. Specifically, the client can copy the audio Real-time Transport Protocol (RTP) packets cached in the audio NetEQ (Jitterbuffer) module and the decoded PCM data from the ASL Audio track to the Audio track, and copy the NACK request information in the audio NetEQ module from the ASL Audio track to the Audio track.

[0068] Correspondingly, as another embodiment, the method may further include: copying the audio data packets and the decoded data after decoding in the cache of the selected audio receiving channel to the cache of the subscribed audio receiving channel; and / or, copying the negative acknowledgment (NACK) request message in the cache of the selected audio receiving channel to the cache of the subscribed audio receiving channel.

[0069] Optionally, as another embodiment, the method may further include: copying the correspondence between the real-time transport protocol RTP timestamp and the network time protocol (NTP) timestamp in the cache of the selected audio receiving channel to the cache of the subscribed audio receiving channel; and synchronizing the audio data and video data received by the selected audio receiving channel using the correspondence between the RTP timestamp and the NTP timestamp.

[0070] It should be understood that the RTP timestamp can be used to represent the timestamp of media data in the RTP message. The role of the RTP timestamp is to restore the timing relationship of the media data based on the timestamp at the receiving end so that the audio and video can be played correctly. The NTP timestamp in the RTCP message is used to help calculate statistical information such as network delay, synchronization, and jitter. In other words, the NTP timestamp is used to report the reference time and calculate statistical information, while the RTP timestamp is used for timing control and synchronization of media data.

[0071] Specifically, the embodiment of the present application can also add a stream binding switching solution for audio and video synchronization. For example, the audio and video synchronization module of the client switches each video stream from being originally bound to the ASL Audio track to being bound to the corresponding Audio track. Specifically, the client switches the audio and video synchronization module from being originally bound to the ASL Audio track to being bound to the corresponding Audio track, and obtains the corresponding relationship of the RTP / NTP timestamp from the RTCP SR message of the new Audio track.

[0072] The embodiment of the present application multiplexes the correspondence between the RTP timestamp and the NTP timestamp in the cache of the selected audio receiving channel without recalculation, thereby improving the accuracy of audio and video synchronization, ensuring the efficiency and smoothness of switching, and improving the user experience.

[0073] It should be noted that in the above-mentioned embodiment of the present application, after the previous data is copied, the previous channel can be suspended or destroyed, and the cache data corresponding to the previous channel can be deleted.

[0074] Optionally, as another embodiment, the method may further include: sending a request message to close audio recording to a media forwarding server, wherein the request message to close audio recording is used to instruct the media forwarding server to switch from an audio recording mode to an audio routing forwarding mode; receiving audio data of consumers subscribed by the client sent by the media forwarding server through a subscription audio receiving channel corresponding to each consumer subscribed by the client, and switching to receiving audio data through a routing audio receiving channel corresponding to the audio routing forwarding mode.

[0075] It should be understood that the process of turning off local audio recording here is the opposite of the process of turning on local audio recording. Figure 2 For description of the embodiments, please refer to Figure 4 As shown, the data flow consists of Figure 4 middle Figure 4 The corresponding solid line part is switched to Figure 4 The dotted part in the corresponding position is switched to Figure 3 The situation shown will not be repeated here.

[0076] Please refer to Figure 5 , Figure 5 A block diagram showing the composition of an apparatus for recording audio in a video conference provided by an embodiment of the present application. Figure 5 The device 500 shown can be a client or a module in the client. It should be understood that the device 500 corresponds to the client in the above method embodiment and can execute each step performed by the client involved in the above method embodiment. The specific functions of the device 500 can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here.

[0077] Figure 5 The device 500 shown includes at least one software function module that can be stored in a memory or fixed in the device in the form of software or firmware. Figure 5 The device 500 shown includes: a sending unit 510, used to send an audio recording request message to a media forwarding server, wherein the audio recording request message is used to instruct the media forwarding server to switch from an audio routing forwarding mode to an audio recording mode; a processing unit 520, used to receive audio data through a routing audio receiving channel corresponding to the audio routing forwarding mode, switch to a subscription audio receiving channel corresponding one by one to consumers subscribed by the client, and receive audio data of consumers subscribed by the client sent by the media forwarding server; and record the received audio data to obtain recorded audio.

[0078] In one implementation, the processing unit is further configured to perform routing and mixing on the received audio data to obtain and play the mixed audio.

[0079] In one embodiment, the processing unit is specifically used to create a subscription audio receiving channel corresponding to the consumers subscribed by the client; suspend receiving audio data through the selected audio receiving channel, and start receiving audio data through the subscription audio receiving channel.

[0080] In one embodiment, the processing unit is also used to copy the audio data packets and the decoded data after decoding in the cache of the selected audio receiving channel to the cache of the subscribed audio receiving channel; and / or, copy the negative confirmation request message in the cache of the selected audio receiving channel to the cache of the subscribed audio receiving channel.

[0081] In one embodiment, the processing unit is also used to copy the correspondence between the real-time transport protocol RTP timestamp and the NTP timestamp in the cache of the selected audio receiving channel to the cache of the subscribed audio receiving channel; and use the correspondence between the RTP timestamp and the NTP timestamp to synchronize the audio data and video data received by the selected audio receiving channel.

[0082] In one embodiment, the sending unit is also used to send a request message to close audio recording to the media forwarding server, and the request message to close audio recording is used to instruct the media forwarding server to switch from an audio recording mode to an audio routing forwarding mode; the processing unit is also used to receive the audio data of the consumers subscribed by the client sent by the media forwarding server through a subscription audio receiving channel corresponding to the consumers subscribed by the client, and switch to receiving audio data through a routing audio receiving channel corresponding to the audio routing forwarding mode.

[0083] Please refer to Figure 6 , Figure 6 A block diagram showing the composition of an apparatus for recording audio in a video conference provided by an embodiment of the present application. Figure 6 The device 600 shown can be a media forwarding server or a module in a media forwarding server. It should be understood that the device 600 corresponds to the media forwarding server in the above method embodiment, and can execute each step performed by the media forwarding server involved in the above method embodiment. The specific functions of the device 600 can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here.

[0084] Figure 6 The device 600 shown includes at least one software function module that can be stored in a memory or fixed in the device in the form of software or firmware. Figure 6 The device 600 shown includes: a receiving unit 610, which is used to receive an audio recording request message sent by a client, and the audio recording request message is used to instruct the media forwarding server to switch from an audio routing forwarding mode to an audio recording mode; a processing unit 620, which is used to transmit audio data from an active speaking list consumer corresponding to the audio routing forwarding mode, switch to an ordinary consumer corresponding one-to-one to the consumers subscribed by the client, and transmit the audio data of the consumers subscribed by the client.

[0085] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0086] like Figure 7 As shown, an embodiment of the present application provides an electronic device 700, which includes: a memory 710, a processor 720, and a computer program stored in the memory 710 and executable on the processor 720, wherein the processor 720 reads the program from the memory 710 through a bus 730 and executes the program to implement the method executed by the client or the method executed by the media forwarding server in any of the above embodiments. Optionally, Figure 7The device shown may also include a transceiver, which may be used to send and / or receive data streams. Figure 7 The device shown may also include a collector, which may be used to collect audio / video data. The optional electronic device 700 may also include other input devices or other hardware devices, etc., but the embodiments of the present application are not limited thereto.

[0087] Processor 720 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 720 can be a microprocessor.

[0088] The memory 710 may be used to store instructions executed by the processor 720 or data related to the execution of instructions. These instructions and / or data may include codes for implementing some functions or all functions of one or more modules described in the embodiments of the present application. The processor 720 of the disclosed embodiment may be used to execute instructions in the memory 710 to implement the method shown above to be performed by the client or the media forwarding server. The memory 710 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memory well known to those skilled in the art.

[0089] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement a method executed by a client or a media forwarding server in any of the above methods provided in the above embodiments.

[0090] An embodiment of the present application further provides a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the method performed by a client or a media forwarding server in any of the above methods provided in the above embodiments.

[0091] It should be noted that the processor in the embodiment of the present invention (eg, Figure 7The processor in the embodiment of the present invention may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method embodiment may be completed by an integrated logic circuit of hardware in the processor or an instruction in software form. The above processor may be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. The general processor may be a microprocessor or the processor may be any conventional processor, etc. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0092] It can be understood that the memory in the embodiment of the present invention (for example, Figure 7The memory in the memory may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0093] The application programs involved in the embodiments of the present application include any application installed on the requesting end, including but not limited to browsers, email, instant messaging services, word processing, keyboard virtualization, widgets, encryption, digital rights management, voice recognition, voice replication, positioning (such as functions provided by the global positioning system), music playback, etc.

[0094] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instruction is loaded and executed on a computer, the process or function according to the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instruction can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0095] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present invention, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0096] Additionally, the terms "system" and "network" are often used interchangeably herein.

[0097] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0098] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0099] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0101] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0102] In short, the above is only a preferred embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for recording audio in a video conference, characterized in that: Applied to a client, the method comprises: Sending an audio recording request message to a media forwarding server, wherein the audio recording request message is used to instruct the media forwarding server to switch from an audio routing forwarding mode to an audio recording mode; The audio data is received by the routing audio receiving channel corresponding to the audio routing forwarding mode, and the audio data is switched to the subscription audio receiving channel corresponding to the consumers subscribed by the client, and the audio data of the consumers subscribed by the client sent by the media forwarding server; The received audio data is recorded to obtain recorded audio.

2. The method according to claim 1, characterized in that The method further comprises: The received audio data is routed and mixed to obtain and play the mixed audio.

3. The method according to claim 1 or 2, characterized in that: The audio data is received by the audio receiving channel corresponding to the audio routing forwarding mode, and the audio receiving channel is switched to the audio receiving channel corresponding to the consumers subscribed by the client, and the audio data of the consumers subscribed by the client sent by the media forwarding server includes: Create a subscription audio receiving channel corresponding to each consumer subscribed by the client; The receiving of audio data through the routing audio receiving channel is suspended, and the receiving of audio data through the subscription audio receiving channel is started.

4. The method according to claim 1 or 2, characterized in that: The method further comprises: Copying the audio data packets in the buffer of the routing audio receiving channel and the decoded data after decoding to the buffer of the subscription audio receiving channel; and / or, The negative confirmation request message in the buffer of the routing audio receiving channel is copied to the buffer of the subscription audio receiving channel.

5. The method according to claim 1 or 2, characterized in that: The method further comprises: Copying the correspondence between the real-time transport protocol RTP timestamp and the NTP timestamp in the cache of the routing audio receiving channel to the cache of the subscription audio receiving channel; The audio data received by the routing audio receiving channel is synchronized with the video data using the corresponding relationship between the RTP timestamp and the NTP timestamp.

6. The method according to claim 1 or 2, characterized in that: The method further comprises: Sending a request message for closing audio recording to a media forwarding server, wherein the request message for closing audio recording is used to instruct the media forwarding server to switch from an audio recording mode to an audio routing forwarding mode; The audio data of the consumers subscribed by the client sent by the media forwarding server is received through the subscription audio receiving channel corresponding to the consumers subscribed by the client, and the audio data is received by the routing audio receiving channel corresponding to the audio routing forwarding mode.

7. A method for recording audio in a video conference, characterized in that: Applied to a media forwarding server, the method comprises: Receive an audio recording request message sent by a client, where the audio recording request message is used to instruct the media forwarding server to switch from an audio routing forwarding mode to an audio recording mode; The audio data is transmitted by the active speaking list consumers corresponding to the audio routing forwarding mode, and switched to ordinary consumers corresponding one-to-one to the consumers subscribed by the client, and the audio data of the consumers subscribed by the client is transmitted.

8. A device for recording audio in a video conference, characterized in that: Applied to a client, the device comprises: A sending unit, used to send an audio recording request message to a media forwarding server, wherein the audio recording request message is used to instruct the media forwarding server to switch from an audio routing forwarding mode to an audio recording mode; A processing unit, configured to receive audio data through a routing audio receiving channel corresponding to the audio routing forwarding mode, switch to a subscription audio receiving channel corresponding one-to-one to consumers subscribed by the client, and receive audio data of consumers subscribed by the client sent by the media forwarding server; The received audio data is recorded to obtain recorded audio.

9. A device for recording audio in a video conference, characterized in that: Applied to a media forwarding server, the device comprises: A receiving unit, configured to receive an audio recording request message sent by a client, wherein the audio recording request message is used to instruct the media forwarding server to switch from an audio routing forwarding mode to an audio recording mode; The processing unit is used to transmit audio data from the active speaking list consumers corresponding to the audio routing forwarding mode, switch to ordinary consumers corresponding one-to-one to the consumers subscribed by the client, and transmit the audio data of the consumers subscribed by the client.

10. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the computer program executes the method according to any one of claims 1 to 7 when being run by the processor.

Images