Audio processing method, device, storage medium and system based on online conference

Abstract

The present application relates to an audio processing method, device, storage medium and system based on online conferences, belonging to the field of computer technology, the method comprising: obtaining audio data of at least two participating terminals during an online conference; determining a channel of target audio data corresponding to the main speaker from at least two channels of audio data; obtaining first text data corresponding to the target audio data; the problem of consuming a large amount of transcription resources when each channel of audio data is converted into text data can be solved; by predicting the current main speaker, only one channel of audio data corresponding to the main speaker is processed, and an online conference only needs one channel of voice transcription resources, which can reduce the transcription resources occupied by the online conference. At the same time, the audio actually used for transcription is the original audio, so there is no signal loss caused by intermediate processing, which can improve the accuracy of voice transcription.

Description

[Technical field]

[0001] The present application relates to an audio processing method, device, storage medium and system based on online conference, belonging to the field of computer technology. [Background technology]

[0002] During an online meeting, the voice of the online meeting needs to be processed to display subtitles or generate meeting minutes. The audio processing includes converting the voice data in the audio data into text data, that is, the voice transcription process.

[0003] The traditional voice transcription method in video conferencing includes: separately transcribing the audio of all participating terminals to obtain text data; merging each text data in chronological order to obtain a text file corresponding to the online conference.

[0004] However, transcribing multiple audio channels separately requires setting up a corresponding voice transcription module for each audio channel, which consumes a large amount of transcription resources. [Summary of the invention]

[0005] The present application provides an audio processing method, device, storage medium and system based on online conferences, which can solve the problem of consuming a large amount of transcription resources when converting each channel of audio data into text data. The technical solution provided by the present application is as follows:

[0006] In a first aspect, an audio processing method based on an online conference is provided, which is used in a conference intelligent server, and the method comprises:

[0007] During an online conference, audio data of at least two participating terminals are obtained; the at least two participating terminals are connected to the same online conference, and each participating terminal corresponds to one channel of audio data;

[0008] Determine a channel of target audio data corresponding to a main speaker from at least two channels of audio data;

[0009] Acquire first text data corresponding to the target audio data.

[0010] Optionally, determining a channel of target audio data corresponding to a main speaker from at least two channels of audio data includes:

[0011] Determine whether each channel of audio data includes voice data;

[0012] When there are at least two channels of audio data including voice data, obtaining voice features of the voice data;

[0013] The target audio data is determined from at least two channels of audio data including the voice data according to the voice feature.

[0014] Optionally, determining the target audio data from at least two channels of audio data including the voice data according to the voice feature includes:

[0015] The speech feature includes speech energy, and a channel of audio data with the largest speech energy is determined as the target audio data;

[0016] or,

[0017] The speech features include speech energy and speech duration, and a channel of audio data whose speech energy exceeds a preset threshold and whose speech duration is the largest is determined as the target audio data;

[0018] or,

[0019] The speech feature includes speech duration, and a channel of audio data whose speech duration exceeds a preset duration threshold and has the largest speech duration is determined as the target audio data.

[0020] Optionally, the step of determining a channel of target audio data corresponding to a main speaker from at least two channels of audio data further includes:

[0021] Obtaining the principal speakers designated for said online conference;

[0022] A channel of audio data corresponding to the designated main speaker is determined from at least two channels of audio data to obtain the target audio data.

[0023] Optionally, the acquiring the first text data corresponding to the target audio data includes:

[0024] Processing the target audio data using a speech transcription algorithm to obtain the first text data;

[0025] or,

[0026] The target audio data is sent to a designated device so that the designated device processes the target audio data using a speech transcription algorithm to obtain first text data corresponding to the target audio data; and the first text data corresponding to the target audio data sent by the designated device is received.

[0027] Optionally, the acquiring audio data of the at least two participating terminals includes:

[0028] When the conference control platform turns on the mixing function, a mixing list sent by the conference control platform is obtained, wherein the mixing list includes data identifiers of the at least two channels of audio data; the conference control platform is used to provide online conference services for the at least two participating terminals, and the number of data identifiers in the mixing list is determined by the conference control platform according to a preset mixing depth;

[0029] Receiving N channels of audio data sent by the conference control platform, where the N channels of audio data refer to audio data corresponding to all participating terminals accessing the online conference, where N is greater than or equal to 2;

[0030] The at least two channels of audio data indicated by the data identifier are obtained from the N channels of audio data.

[0031] Optionally, after obtaining the first text data corresponding to the target audio data, the method further includes:

[0032] After the online conference ends, obtaining other audio data of the online conference, where the other audio data refers to audio data other than the target audio data in the at least two channels of audio data;

[0033] Acquire second text data corresponding to the other audio data;

[0034] The first text data and the second text data are merged in chronological order to obtain a text file corresponding to the online conference.

[0035] In a second aspect, an audio processing device based on an online conference is provided, the device comprising:

[0036] An audio acquisition module is used to acquire audio data of at least two participating terminals during an online conference; the at least two participating terminals are connected to the same online conference, and each participating terminal corresponds to one channel of audio data;

[0037] An audio determination module, used to determine a channel of target audio data corresponding to a main speaker from at least two channels of audio data;

[0038] The text acquisition module is used to acquire first text data corresponding to the target audio data.

[0039] According to a third aspect, an electronic device is provided, comprising a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement the online conference-based audio processing method provided in the first aspect.

[0040] In a fourth aspect, a computer-readable storage medium is provided, wherein a program is stored in the storage medium, and when the program is executed by a processor, the program is used to implement the audio processing method based on online conference provided in the first aspect.

[0041] In a fifth aspect, an audio processing system based on an online conference is provided, the system comprising N participating terminals, a conference control platform communicatively connected to each participating terminal, and a conference intelligent server connected to the conference control platform; the N participating terminals access the same online conference, and N is an integer greater than 1;

[0042] Each of the N participating terminals is used to collect audio data during the online conference to obtain N channels of audio data; and send the N channels of audio data to the conference control platform;

[0043] The conference control platform is used to receive the N-channel audio data and forward the N-channel audio data to the conference intelligent server;

[0044] The conference intelligent server is used to obtain audio data from at least two participating terminals during an online conference; determine a channel of target audio data corresponding to a main speaker from the at least two channels of audio data; and obtain first text data corresponding to the target audio data.

[0045] The beneficial effects of the present application include at least: obtaining audio data of at least two participating terminals during an online conference; determining a target audio data channel corresponding to a main speaker from at least two audio data channels; obtaining first text data corresponding to the target audio data; solving the problem of consuming a large amount of transcription resources when converting each audio data channel into text data; and predicting the current main speaker and only processing the audio data channel corresponding to the main speaker. At this time, an online conference only needs one voice transcription resource, which can reduce the transcription resources occupied by the online conference.

[0046] In addition, the audio actually used for transcription in this application is the original audio rather than the mixed audio. Therefore, there is no signal loss caused by intermediate processing, which can improve the accuracy of speech transcription.

[0047] In addition, the accuracy of determining the primary speaker can be improved by combining speech detection and speech features to determine the primary speaker instead of determining the speaker only by speech detection.

[0048] In addition, by obtaining the designated main speaker for the online meeting, the efficiency of determining the main speaker can be improved without having to determine it through decision-making by oneself.

[0049] In addition, when the mixing function is turned on on the conference control platform, since the audio data that does not enter the mixing will not be heard by the participants, by determining the main speaker from at least two channels of audio data corresponding to the mixing list, the number of audio data that needs to be judged can be reduced, saving equipment resources.

[0050] In addition, by transcribing the other audio data at the end of the meeting to obtain the second text data, and merging the first text data with the second text data, it can be ensured that the user can obtain a complete meeting record.

[0051] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0052] Figure 1 It is a structural diagram of an audio processing system based on online conference provided by an embodiment of the present application;

[0053] Figure 2 is a flowchart of an audio processing method based on an online conference provided by an embodiment of the present application;

[0054] Figure 3 is a flowchart of an audio processing method based on an online conference provided by another embodiment of the present application;

[0055] Figure 4 This is a flowchart of an audio processing method after an online conference is completed provided by an embodiment of the present application;

[0056] Figure 5 is a block diagram of an audio processing device based on an online conference provided by an embodiment of the present application;

[0057] Figure 6 It is a block diagram of an electronic device provided by an embodiment of the present application. [Specific implementation method]

[0058] The specific implementation methods of the present application are further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application but are not intended to limit the scope of the present application.

[0059] First, some nouns involved in some applications are introduced.

[0060] Online conference: Online conference is also called network conference or remote collaborative office. Users use the Internet to share data among multiple users in different locations.

[0061] Voice Activity Detection (VAD), also known as voice endpoint detection and voice boundary detection, is used to detect whether the current audio signal contains a voice signal, that is, to judge the current input audio signal and distinguish the voice signal from various background noise signals.

[0062] Speech transcription technology: refers to a technology that converts audio data into text data. In one example, speech transcription technology includes but is not limited to: Automatic Speech Recognition Technology (Automatic Speech Recognitio, ASR): is a technology that converts human speech into text.

[0063] Figure 1 1 is a schematic diagram of the structure of an audio processing system based on an online conference provided by an embodiment of the present application. In the present application, the online conference can be a multi-point conference or a point-to-point conference, and the present embodiment does not limit the type of online conference. Figure 1 As shown, the system at least includes a conference terminal 110 , a conference control platform 120 and a conference intelligent server 130 .

[0064] The number of the participating terminals 110 is N, and the number N is greater than or equal to 2. The N participating terminals 110 join the same online conference through the conference control platform 120 .

[0065] Optionally, the participant terminal 110 may be a device with an online conference function, such as a mobile phone, a tablet computer, or a computer. This embodiment does not limit the device type of the participant terminal 110 .

[0066] During the online conference, the participant terminal 110 has the function of collecting audio data of the participants and sending the audio data to the conference control platform 120. At this time, each participant terminal 110 will send one channel of audio data to the conference control platform 120. In other words, each participant terminal 110 corresponds to one channel of audio data, and different participant terminals 110 have different channels of audio data.

[0067] In addition, the participant terminal 110 may also have other functions required for the online conference, such as image acquisition function, communication function, etc., which are not listed one by one in this embodiment.

[0068] The conference terminal 110 is connected to the conference control platform 120 for communication.

[0069] Optionally, the conference control platform 120 may be a multi-point control unit (MCU), or a terminal or server equipped with an MCU. This embodiment does not limit the implementation of the conference control platform 120 .

[0070] The conference control platform 120 is used to provide online conference services for the participating terminals 110. During the online conference, the conference control platform 120 is used to receive audio data sent by N participating terminals 110 to obtain N channels of audio data.

[0071] Optionally, the conference control platform 120 has a mixing function, and when the conference control platform 120 turns on the mixing function, M channels of audio data among N channels of audio data can be mixed based on a preset mixing depth; and the mixed audio data is sent to each participating terminal 110. N≥M, M is greater than or equal to 2. The value of M is less than or equal to the preset mixing depth.

[0072] Correspondingly, the conference terminal 110 receives and plays the audio data mixed by the conference control platform 120. At this time, the participants corresponding to each conference terminal 110 can only hear the M channels of audio data added into the mix.

[0073] The preset mixing depth refers to the maximum number of mixing paths of the conference control platform 120 , for example, the preset mixing depth is 8 or 5, etc. This embodiment does not limit the value of the preset mixing depth.

[0074] Since there may be many participating terminals 110 collecting audio data at the same time, not all audio data of participating terminals 110 need to be mixed. Based on this, in this embodiment, by setting a preset mixing depth, when the number of participating terminals 110 sending audio data at the same time exceeds the preset mixing depth, M (equal to the preset mixing depth) channels of audio data are selected from each channel of audio data for mixing, which can save transmission resources and improve the conference effect.

[0075] The conference control platform 120 selects M channels of audio data according to the mixing depth, including: performing voice detection on each of the N channels of audio data received; determining at least two channels of audio data including voice data from the N channels of audio data; determining whether the number of channels of the at least two channels of audio data including voice data is greater than the mixing depth; if so, obtaining the voice energy of the at least two channels of audio data including voice data, and selecting the M channels of audio data with the highest voice energy as the audio data to be mixed.

[0076] Optionally, in the present application, speech energy may be the maximum volume value of audio data including speech data; or the average volume value of audio data including speech data. The present application does not limit the setting method of speech energy.

[0077] The method of performing voice detection on the audio data may be VAD detection. In this case, the conference control platform 120 also has a VAD detection function.

[0078] It should be additionally explained that, when the number of the participating terminals 110 simultaneously sending audio data is less than or equal to the preset mixing depth, the conference control platform 120 may mix all the received audio data without selecting the audio data.

[0079] Optionally, when the conference control platform 120 does not have a mixing function, or has a mixing function but the mixing function is not turned on, the conference control platform 120 forwards the audio data to other participating terminals 110 connected to the same online conference each time it receives audio data sent by a participating terminal 110.

[0080] In this embodiment, the conference control platform 120 is also communicatively connected to the conference intelligent server 130 .

[0081] Optionally, the conference intelligent server 130 can be a computer, a server cluster, etc. The conference intelligent server 130 can be implemented in the same device as the conference control platform 120, or can be implemented in a different device from the conference control platform 120. This embodiment does not limit the implementation method of the conference intelligent server 130.

[0082] In other implementations, the conference intelligent server 130 may also be called an audio processing device, an intelligent conference management server, etc. This embodiment does not limit the name of the conference intelligent server 130.

[0083] During the online conference, after receiving N channels of audio data sent by N participating terminals 110 , the conference control platform 120 sends the N channels of audio data to the conference intelligent server 130 .

[0084] Correspondingly, after receiving N channels of audio data, the conference intelligent server 130 obtains audio data from at least two participating terminals; determines a channel of target audio data corresponding to the main speaker from the at least two channels of audio data; and obtains first text data corresponding to the target audio data.

[0085] The audio data of at least two participating terminals acquired by the conference intelligent server 130 refers to the audio data to be processed. The number of the audio data to be processed is less than or equal to N.

[0086] The main speaker refers to the participant who is mainly speaking in the current time period predicted by the conference intelligent server 130.

[0087] Because in some conference scenarios, the conference mode is mainly based on a speaker speaking at the rostrum, and rarely involves dialogue with the audience. Even if there is a need for dialogue, it is conducted in the form of terminals taking turns to speak. Based on this, in a specific conference scenario where a single speaker is the main speaker, the current main speaker is predicted, and then only the audio data corresponding to the main speaker is processed to obtain the first text data. At this time, an online meeting only needs one voice transcription resource; and because the audio actually used for transcription is the original audio (rather than the mixed audio), there is no signal loss caused by intermediate processing, which can improve the accuracy of voice transcription.

[0088] The following is a detailed introduction to the audio processing method for online conferences involved in this application.

[0089] Figure 2 This is a flowchart of an audio processing method for an online conference provided by an embodiment of the present application. This embodiment uses the method for Figure 1 Taking the conference intelligent server 130 of the system as an example, the method includes at least the following steps:

[0090] Step 201: During an online conference, audio data of at least two participating terminals are obtained.

[0091] Among them, at least two participating terminals are terminals accessing the same online conference, and each participating terminal corresponds to one channel of audio data.

[0092] In this embodiment, the audio data acquired by the conference intelligent server are all audio data forwarded by the conference control platform, that is, after the conference control platform receives the audio data sent by each conference terminal, it will forward the original audio data to the conference intelligent server.

[0093] In this embodiment, the audio data of at least two participating terminals acquired by the conference intelligent server are the audio data to be processed. In other words, the conference intelligent server may receive N channels of audio data, but only processes M channels of audio data.

[0094] For example, when the conference control platform turns on the mixing function, if the conference control platform receives N channels of audio data sent by N participating terminals, it will first mix the N channels of audio data according to the preset mixing depth, and send the mixing list and the N channels of audio data to the conference intelligent server. The mixing list includes data identifiers of at least two channels of audio data; the number of data identifiers in the mixing list is determined by the conference control platform according to the preset mixing depth.

[0095] Optionally, the data identifier is used to uniquely identify one channel of audio data, and may be the number, device number, participant name or IP address of the participant terminal sending the audio data. This embodiment does not limit the implementation method of the data identifier.

[0096] Accordingly, the conference intelligent server obtains the mixing list sent by the conference control platform, receives N channels of audio data sent by the conference control platform, obtains at least two channels of audio data indicated by the data identifier from the N channels of audio data, and obtains M channels of audio data to be processed.

[0097] When the conference control platform does not enable the audio mixing function or does not have the audio mixing function, the conference intelligent server may treat all received audio data as at least two channels of audio data to be processed, that is, M=N.

[0098] Step 202: determining a channel of target audio data corresponding to a main speaker from at least two channels of audio data.

[0099] In this embodiment, the number of main speakers determined by the intelligent conference server is one, and therefore, one channel of target audio data corresponding to the main speaker can be obtained.

[0100] A method for determining a channel of target audio data corresponding to the main speaker from at least two channels of audio data includes but is not limited to one of the following:

[0101] The first method is to obtain a main speaker designated in an online conference; determine one channel of audio data corresponding to the designated main speaker from at least two channels of audio data to obtain target audio data.

[0102] The main speaker designated in the online conference is designated by the administrator of the online conference or decided by voting of each participant. This embodiment does not limit the designation method of the designated main speaker. After the conference control platform obtains the designated main speaker, it will send the speaker information of the designated speaker to the intelligent conference server. Correspondingly, the intelligent conference server obtains the speaker information sent by the conference control platform and obtains the main speaker corresponding to the speaker information.

[0103] The speaker information may be the same as the data identifier or different from the data identifier. The speaker information may be the name of the participant, the IP address of the participant terminal, the device number, etc. This embodiment does not limit the implementation method of the speaker information.

[0104] The second method is to determine whether each channel of audio data includes voice data; when there are at least two channels of audio data including voice data, obtain voice features of the voice data; and determine target audio data from at least two channels of audio data including voice data based on the voice features.

[0105] In one example, determining whether each channel of audio data includes voice data includes: using a VAD algorithm to detect each channel of audio data respectively to obtain a detection result, where the detection result is used to indicate whether the audio data includes voice data.

[0106] At this time, each channel of audio data corresponds to a VAD detection algorithm. In other words, VAD detection is performed in parallel on each channel of audio data in the same time period to obtain a detection result.

[0107] In another example, the conference control platform performs VAD detection on each channel of audio data during mixing, obtains a detection result, and sends the detection result to the conference intelligent server. Accordingly, the conference intelligent server determines whether each channel of audio data includes voice data, including: receiving the detection result sent by the conference control platform; and determining whether each channel of audio data includes voice data according to the detection result.

[0108] Optionally, the speech feature includes speech energy and / or speech duration. For different speech features, different methods for determining the target audio data are used. The following describes each method for determining the target audio data.

[0109] Case 1: When the speech feature includes speech energy, the audio data having the maximum speech energy is determined as the target audio data.

[0110] Case 2: The speech features include speech energy and speech duration. The audio data whose speech energy exceeds a preset threshold and whose speech duration is the longest is determined as the target audio data.

[0111] Since a certain audio data channel may have a sudden change in speech energy, for example, a participant sneezes, causing the speech energy to suddenly increase or decrease. At this time, it may not be accurate enough to determine the target audio data only by speech energy. Based on this, in the second case, the target audio data is determined by combining speech energy and speech duration, which can improve the accuracy of determining the target audio data.

[0112] Case 3: The speech feature includes the speech duration, and the audio data whose speech duration exceeds the preset duration threshold and has the largest speech duration is determined as the target audio data.

[0113] It should be noted that, in actual implementation, the voice features may also include other features, such as voice frequency, etc. This embodiment does not limit the implementation method of the voice features and the method of determining the target audio data based on the voice features.

[0114] Step 203: Acquire first text data corresponding to the target audio data.

[0115] In this embodiment, obtaining the first text data corresponding to the target audio data includes but is not limited to the following implementation methods:

[0116] The first method is to use a speech transcription algorithm to process the target audio data to obtain first text data.

[0117] For example: use the ASR algorithm to transcribe the target audio data.

[0118] The second method: sending the target audio data to a designated device so that the designated device processes the target audio data using a speech transcription algorithm to obtain first text data corresponding to the target audio data; receiving the first text data corresponding to the target audio data sent by the designated device.

[0119] At this time, the conference intelligent server does not have a voice transcription function. The conference intelligent server can send the target audio data to a designated device, which is communicatively connected to the conference intelligent server and has a voice transcription function.

[0120] Optionally, after acquiring the first text data, the intelligent conference server can send the first text data to each participating terminal through the conference control platform, so that the participating terminal can display the first text data in real time during the conference. At this time, each participant can see the text information corresponding to the speech content of the main speaker.

[0121] Since only one channel of target audio data corresponding to the main speaker is transcribed in the above embodiment, the information of other speakers is not converted into text data at this time. In order to ensure that the user can obtain the text data corresponding to the speech content of each speaker, in this embodiment, after step 103, it also includes: when the online meeting is over, obtaining other audio data of the online meeting, the other audio data refers to the audio data other than the target audio data in at least two channels of audio data; obtaining the second text data corresponding to the other audio data; merging the first text data and the second text data in chronological order to obtain the text file corresponding to the online meeting.

[0122] The related description of obtaining the second text data corresponding to the other audio data is detailed in the two implementation methods in step 103, which will not be repeated in this embodiment.

[0123] The first text data and the second text data are merged in chronological order to obtain a text file corresponding to the online meeting, including: when the same time period includes both the first text data and the second text data, a corresponding relationship between the time period and the first text data and the second text data is established respectively, and the first text data and the second text data are merged according to the corresponding relationship.

[0124] In other words, when the speakers overlap in time, the conference intelligent server will process the original audio data of each speaker to obtain the text data corresponding to each speaker, and then save the text data of each speaker into a text file.

[0125] Optionally, the text file can be sent to each participating terminal as the final meeting minutes.

[0126] To summarize, the audio processing method based on online conference provided in this embodiment obtains audio data of at least two participating terminals during the online conference; determines a channel of target audio data corresponding to the main speaker from at least two channels of audio data; obtains first text data corresponding to the target audio data; and can solve the problem of consuming a large amount of transcription resources when each channel of audio data is converted into text data; and in a specific conference scenario where a single speaker is the main speaker, by predicting the current main speaker, only the channel of audio data corresponding to the main speaker is processed. At this time, an online conference only needs one voice transcription resource, which can reduce the transcription resources occupied by the online conference.

[0127] In addition, the audio actually used for transcription in this application is the original audio rather than the mixed audio. Therefore, there is no signal loss caused by intermediate processing, which can improve the accuracy of speech transcription.

[0128] In addition, the accuracy of determining the primary speaker can be improved by combining speech detection and speech features to determine the primary speaker instead of determining the speaker only by speech detection.

[0129] In addition, when the conference control platform turns on the mixing function, since the audio data that does not enter the mixing will not be heard by the participants, by determining the main speaker from at least two channels of audio data corresponding to the mixing list, the number of audio data that needs to be judged can be reduced, saving equipment resources.

[0130] In addition, by transcribing the other audio data at the end of the meeting to obtain the second text data, and merging the first text data with the second text data, it can be ensured that the user can obtain a complete meeting record.

[0131] In order to more clearly understand the audio processing method based on online conference provided by this application, the method is described below with an example. Figure 3 In this embodiment, the number of participating terminals accessing the online conference is 3, namely, participating terminal 1, participating terminal 2, and participating terminal 3. The method includes at least the following steps:

[0132] In step 31, the participating terminal 1 joins the conference and sends one channel of audio data to the conference control platform; the participating terminal 2 joins the conference and sends one channel of audio data to the conference control platform; the participating terminal 3 joins the conference and sends one channel of audio data to the conference control platform.

[0133] This embodiment does not limit the order in which participant terminal 1 , participant terminal 2 , and participant terminal 3 join the same online conference.

[0134] Step 32: The conference control platform forwards all three channels of audio data to the conference intelligent server.

[0135] The conference control platform forwards all audio data from the participating terminals to the conference intelligent server. If the mixing function is enabled on the conference control platform, the conference control platform will also send a list of all mixed audio to the conference intelligent server according to the preset mixing depth.

[0136] Step 33: When there is a designated main speaker for the current online conference, the conference control platform forwards the speaker information of the designated main speaker to the conference intelligent server.

[0137] Optionally, the designated main speaker may be mandatorily designated at the beginning of the online conference, or may be selected by the conference control platform according to its own decision-making strategy. This embodiment does not limit the method for determining the designated main speaker.

[0138] When there is no designated main speaker in the current online conference, step 33 is not performed.

[0139] Step 33 may be executed after step 32, or before step 32, or simultaneously with step 32. This embodiment does not limit the execution order of step 33 and step 32.

[0140] In step 34, the conference intelligent server determines whether there is a designated main speaker for the online conference; if so, execute step 37; if not, execute step 35.

[0141] Specifically, if the conference intelligent server receives the speaker information of the designated main speaker, it determines that there is a designated main speaker in the online meeting; if the conference intelligent server does not receive the speaker information of the designated main speaker, it determines that there is no designated main speaker in the online meeting.

[0142] Step 35: The conference intelligent server performs VAD detection on the received 3-channel audio data.

[0143] In the case of the audio mixing function enabled on the conference control platform, this embodiment takes the example of all three audio data received by the conference intelligent server entering the audio mixing. In actual implementation, if one audio data channel does not enter the audio mixing, two audio data channels can be selected from the three audio data channels for VAD detection according to the audio mixing list. For details, see Figure 2 The described embodiment.

[0144] Step 36, the conference intelligent server determines, based on the result of the VAD detection, a channel of audio data that currently has voice data, has a voice duration of more than 3 seconds, and has the longest voice duration, and obtains a channel of target audio data corresponding to the main speaker.

[0145] In this embodiment, the preset duration threshold is taken as 3 seconds for illustration. In actual implementation, it may also be other values. This application does not limit the value of the preset duration threshold.

[0146] Step 35 and step 36 are processes that are continuously iterated over time, that is, VAD detection is continuously performed on the three-channel audio data in each current time period, and the main speaker is determined until the online meeting ends.

[0147] In step 37, the conference intelligent server sends one channel of target audio data corresponding to the main speaker to the ASR transcription server for transcription.

[0148] In this embodiment, transcription through an ASR transcription server is used as an example for explanation. In actual implementation, the transcription process can also be completed in the conference intelligent server. This embodiment does not limit the implementation method of the transcription process.

[0149] Step 38: The ASR transcription server transcribes the speaker's audio to obtain first text data; and returns the first text data to the conference intelligent server.

[0150] Step 39: After receiving the first text data returned by the ASR server, the conference intelligent server uses the first text data as the real-time text corresponding to the online conference and forwards the first text data to the conference control platform.

[0151] Step 310: After receiving the first text data, the conference control platform forwards the first text data to the conference participating terminals 1, 2 and 3, so that the conference participating terminals 1, 2 and 3 can display simultaneous subtitles.

[0152] At the end of the online meeting, refer to Figure 4 , at least the following steps are included:

[0153] Step 311: The conference intelligent server sends other audio data in the VAD to the ASR transcription server.

[0154] Step 312: The ASR transcription server transcribes other audio data to obtain second text data; and the second text data is returned to the conference intelligent server.

[0155] Step 313: The conference intelligent server receives the second text data, and merges the second text data with the first text data in chronological order to obtain a text file.

[0156] Optionally, the conference intelligent server may also mix the audio data of each channel to obtain a mixed audio file.

[0157] According to the above steps, the audio processing method provided in this embodiment can achieve:

[0158] 1. In real-time scenarios (simultaneous subtitles), only one transcription is done (the one corresponding to the designated main speaker, or the main speaker determined by itself) and output to the screen.

[0159] The audio data of non-main speakers is not transcribed in real time, but is transcribed when the text (or meeting minutes) is generated, which can save transcription resources. The resource requirements for real-time transcription are much greater than those for non-real-time transcription.

[0160] 2. Record the complete meeting speeches in a non-real-time scenario, and store what is said in each meeting in a text file.

[0161] Figure 5 is a block diagram of an audio processing device based on online conference provided by an embodiment of the present application. Figure 1 The conference intelligent server 130 of the system is used as an example for explanation. The device includes at least the following modules: an audio acquisition module 510 , an audio determination module 520 and a text acquisition module 530 .

[0162] The audio acquisition module 510 is used to acquire audio data of at least two participating terminals during the online conference; the at least two participating terminals are connected to the same online conference, and each participating terminal corresponds to one channel of audio data;

[0163] An audio determination module 520, configured to determine a channel of target audio data corresponding to a main speaker from at least two channels of audio data;

[0164] The text acquisition module 530 is used to acquire first text data corresponding to the target audio data.

[0165] For relevant details, refer to the above method embodiment.

[0166] It should be noted that: the audio processing device based on online conferences provided in the above embodiments only uses the division of the above functional modules as an example when performing audio processing based on online conferences. In actual applications, the above functional distribution can be completed by different functional modules as needed, that is, the internal structure of the audio processing device based on online conferences is divided into different functional modules to complete all or part of the functions described above. In addition, the audio processing device based on online conferences provided in the above embodiments and the audio processing method embodiment based on online conferences belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0167] Figure 6 is a block diagram of an electronic device provided by an embodiment of the present application. The device can be Figure 1 The conference intelligent server 130 of the system includes at least a processor 601 and a memory 602 .

[0168] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0169] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction, which is used to be executed by the processor 601 to implement the audio processing method based on online conferences provided in the method embodiment of the present application.

[0170] In some embodiments, the electronic device may further optionally include: a peripheral device interface and at least one peripheral device. The processor 601, the memory 602 and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply.

[0171] Of course, the electronic device may also include fewer or more components, which is not limited in this embodiment.

[0172] Optionally, the present application also provides a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the audio processing method based on online conferences of the above method embodiment.

[0173] Optionally, the present application also provides a computer product, which includes a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the audio processing method based on online conferences of the above method embodiment.

[0174] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0175] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An audio processing method based on online conference, characterized in that: Used in a conference intelligent server, the method includes: During an online conference, audio data of at least two participating terminals are obtained; the at least two participating terminals are connected to the same online conference, and each participating terminal corresponds to one channel of audio data; Determining a channel of target audio data corresponding to a main speaker from at least two channels of audio data includes: Perform VAD loop iteration detection on the at least two channels of audio data, and determine a channel of target audio data corresponding to the main speaker according to the VAD loop iteration detection result until the online conference ends; Acquire first text data corresponding to the target audio data, where the target audio data is original audio data, and send the first text data to each participating terminal so that the participating terminal can display the first text data in real time during the conference; After obtaining the first text data corresponding to the target audio data, the method further includes: After the online conference ends, other audio data of the online conference is obtained, where the other audio data refers to other audio data with VAD in the at least two channels of audio data except the target audio data; Acquire second text data corresponding to the other audio data; The first text data and the second text data are combined in chronological order to obtain a text file corresponding to the online meeting, including: When the same time period includes both the first text data and the second text data, respectively establishing a correspondence between the time period and the first text data and the second text data, and merging the first text data and the second text data according to the correspondence; The text file is sent as the final meeting minutes to each participating terminal.

2. The method according to claim 1, characterized in that The step of determining a channel of target audio data corresponding to the main speaker from at least two channels of audio data includes: Determine whether each channel of audio data includes voice data; When there are at least two channels of audio data including voice data, obtaining voice features of the voice data; The target audio data is determined from at least two channels of audio data including the voice data according to the voice feature.

3. The method according to claim 2, characterized in that The step of determining the target audio data from at least two channels of audio data including the voice data according to the voice feature comprises: The speech feature includes speech energy, and a channel of audio data with the largest speech energy is determined as the target audio data; or, The speech features include speech energy and speech duration, and a channel of audio data whose speech energy exceeds a preset threshold and whose speech duration is the largest is determined as the target audio data; or, The speech feature includes speech duration, and a channel of audio data whose speech duration exceeds a preset duration threshold and has the largest speech duration is determined as the target audio data.

4. The method according to claim 1, characterized in that: The step of determining a channel of target audio data corresponding to the main speaker from at least two channels of audio data further includes: Obtaining the principal speakers designated for said online conference; A channel of audio data corresponding to the designated main speaker is determined from at least two channels of audio data to obtain the target audio data.

5. The method according to claim 1, characterized in that The obtaining of first text data corresponding to the target audio data includes: Processing the target audio data using a speech transcription algorithm to obtain the first text data; or, The target audio data is sent to a designated device so that the designated device processes the target audio data using a speech transcription algorithm to obtain first text data corresponding to the target audio data; and the first text data corresponding to the target audio data sent by the designated device is received.

6. The method according to claim 1, characterized in that The obtaining of audio data of the at least two participating terminals includes: When the conference control platform turns on the mixing function, a mixing list sent by the conference control platform is obtained, wherein the mixing list includes data identifiers of the at least two channels of audio data; the conference control platform is used to provide online conference services for the at least two participating terminals, and the number of data identifiers in the mixing list is determined by the conference control platform according to a preset mixing depth; Receiving N channels of audio data sent by the conference control platform, where the N channels of audio data refer to audio data corresponding to all participating terminals accessing the online conference, where N is greater than or equal to 2; The at least two channels of audio data indicated by the data identifier are obtained from the N channels of audio data.

7. An audio processing device based on online conference, characterized in that: The device includes a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement the audio processing method based on online conference as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The storage medium stores a program, and when the program is executed by the processor, it is used to implement the audio processing method based on online conference as described in any one of claims 1 to 6.

9. An audio processing system based on online conference, characterized in that: The system includes N participating terminals, a conference control platform connected to each participating terminal in communication, and a conference intelligent server connected to the conference control platform; the N participating terminals access the same online conference, where N is an integer greater than 1; Each of the N participating terminals is used to collect audio data during the online conference to obtain N channels of audio data; Sending the N channels of audio data to the conference control platform; The conference control platform is used to receive the N-channel audio data and forward the N-channel audio data to the conference intelligent server; The conference intelligent server is used to obtain audio data of at least two participating terminals during the online conference; determine a channel of target audio data corresponding to the main speaker from the at least two channels of audio data; and obtain first text data corresponding to the target audio data, wherein the target audio data is the original audio data, and send the first text data to each participating terminal so that the participating terminal can display the first text data in real time during the conference; The step of determining a channel of target audio data corresponding to a main speaker from at least two channels of audio data includes: Perform VAD loop iteration detection on the at least two channels of audio data, and determine a channel of target audio data corresponding to the main speaker according to the VAD loop iteration detection result until the online conference ends; After obtaining the first text data corresponding to the target audio data, the method further includes: After the online conference ends, other audio data of the online conference is obtained, where the other audio data refers to other audio data with VAD in the at least two channels of audio data except the target audio data; Acquire second text data corresponding to the other audio data; The first text data and the second text data are combined in chronological order to obtain a text file corresponding to the online meeting, including: When the same time period includes both the first text data and the second text data, respectively establishing a correspondence between the time period and the first text data and the second text data, and merging the first text data and the second text data according to the correspondence; The text file is sent as the final meeting minutes to each participating terminal.

Images