Audio processing method, device and system, storage medium and electronic equipment

Abstract

The embodiment of the invention provides an audio processing method, device and system, a storage medium and electronic equipment. The method comprises the following steps: acquiring audio data of each participant, participant identifiers corresponding to the audio data, first time identifiers of the audio data, mixed stream audio data and second time identifiers of the mixed stream audio data; the mixed stream audio data is obtained by mixing audio data of each conference participant; identifying each piece of audio data to obtain speaking text data; and according to each first time identifier and each second time identifier, aligning each participant identifier, the speaking text data and the mixed stream audio data to obtain mixed stream speaking text data associated with the participant identifier, thereby effectively distinguishing the speaking content of each participant based on the mixed stream speaking text number, and improving the speaking efficiency. And users can understand the conference content conveniently.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio conferencing, and in particular, to an audio processing method, device, system, storage medium, and electronic device. Background Art

[0002] With the development of the Internet, more and more users hold audio conferences and audio and video conferences through the Internet. In audio conferences and audio and video conferences, it is often necessary to record the speech content of the participants. In related technologies, the audio in the meeting is generally recorded, and users can obtain the meeting content through the recorded audio file.

[0003] However, due to poor recording quality or unclear audio from the speaker, users cannot accurately understand the content of the meeting through the audio file, which affects the users' understanding of the content of the meeting. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present application provides an audio processing method, device, system, storage medium and electronic device to effectively distinguish the speech content of each participant and facilitate users to understand the content of the meeting.

[0005] According to a first aspect of an embodiment of the present application, there is provided an audio processing method, comprising the following steps:

[0006] Acquire audio data of each participant, a participant identifier corresponding to each audio data, a first time identifier of each audio data, mixed audio data, and a second time identifier of the mixed audio data; the mixed audio data is obtained by mixing the audio data of each participant;

[0007] Identify each of the audio data to obtain speech text data;

[0008] According to each of the first time markers and the second time markers, each of the participant identifiers, the speech text data and the mixed audio data are aligned to obtain the mixed speech text data associated with the participant identifier.

[0009] According to a second aspect of an embodiment of the present application, there is provided an audio processing device, including:

[0010] A data acquisition module, used to acquire audio data of each participant, a participant identifier corresponding to each audio data, a first time identifier of each audio data, mixed audio data, and a second time identifier of the mixed audio data; the mixed audio data is obtained by mixing the audio data of each participant;

[0011] A text recognition module, used to recognize each of the audio data and obtain speech text data;

[0012] The speech text acquisition module is used to align each of the participant identifiers, the speech text data and the mixed audio data according to each of the first time identifiers and the second time identifier, and obtain the mixed speech text data associated with the participant identifier.

[0013] According to a third aspect of the embodiment of the present application, there is provided an audio processing system, comprising: a server and a plurality of conference participating terminals; each of the conference participating terminals is connected to the server;

[0014] Each of the participating terminals sends the audio data of each participant and the participant identifier corresponding to each of the audio data to the server;

[0015] The server is used to obtain audio data of each participant, a participant identifier corresponding to each audio data, a first time identifier of each audio data, mixed audio data, and a second time identifier of the mixed audio data; the mixed audio data is obtained by mixing the audio data of each participant;

[0016] The server is used to identify each of the audio data and obtain speech text data;

[0017] The server is used for aligning each of the participant identifiers, the speech text data and the mixed audio data according to each of the first time identifiers and the second time identifier, and obtaining the mixed speech text data associated with the participant identifier.

[0018] According to a fourth aspect of an embodiment of the present application, there is provided an electronic device, comprising a processor and a memory; the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the audio processing method as described above.

[0019] According to a fifth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the audio processing method as described above is implemented.

[0020] The embodiment of the present application obtains the audio data of each participant, the participant identifier corresponding to each of the audio data, the first time identifier of each of the audio data, the mixed audio data and the second time identifier of the mixed audio data; the mixed audio data is obtained by mixing the audio data of each participant; each of the audio data is identified to obtain speech text data; according to each of the first time identifier and the second time identifier, each of the participant identifiers, the speech text data and the mixed audio data are aligned to obtain the mixed speech text data associated with the participant identifier, so that the speech content of each participant can be effectively distinguished based on the number of mixed speech texts, so that users can understand the content of the meeting more conveniently.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0022] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A schematic diagram of an application scenario of an audio processing method shown in an embodiment of the present application;

[0025] Figure 2 This is a flowchart of an audio processing method shown in one embodiment of the present application;

[0026] Figure 3 A flowchart of a method for determining audio data of each participant and a participant identifier corresponding to each audio data, shown as an embodiment of the present application;

[0027] Figure 4 A flowchart of a method for determining a first time identifier is shown as an embodiment of the present application;

[0028] Figure 5 A flowchart of a method for determining a second time marker is shown as an embodiment of the present application;

[0029] Figure 6 A flowchart of a method for determining mixed-stream speech text data shown in one embodiment of the present application;

[0030] Figure 7 A schematic diagram showing the principle of determining mixed-stream speech text data according to an embodiment of the present application;

[0031] Figure 8 A flowchart of an audio processing method according to another embodiment of the present application;

[0032] Fig. 9 This is a schematic diagram showing the effect of an audio processing method according to an embodiment of the present application;

[0033] Fig.10 A schematic block diagram of an audio processing device according to an embodiment of the present application;

[0034] Fig.11This is a schematic diagram of the structure of an electronic device shown in one embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present application clearer, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. Wherein, when the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.

[0036] It should be clear that the implementation methods described in the embodiments described below do not represent all implementation methods consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0037] The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit this application. The singular forms of "a", "the" and "the" used in this application are also intended to include plural forms, unless the context clearly indicates other meanings. In addition, in the description of this application, unless otherwise specified, "multiple" refers to two or more. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone; the character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0038] It should be understood that, although the terms first, second, third, etc. may be used in the present application to describe various information, such information should not be limited to these terms, and these terms are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. Depending on the context, the words "if" / "if" used in the present application can be interpreted as "at the time of" or "when" or "in response to determination".

[0039] It should be understood that although the method of the present application identifies the step sequence, and the various steps in the method flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order of the step sequence identification and the arrow indication. Unless clearly stated in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Among them, each step can include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0040] The audio processing method of the present application is mainly applicable to scenarios involving multi-terminal audio interaction, such as audio conference scenarios, audio and video conference scenarios, remote classroom teaching scenarios, etc. The present application embodiment is exemplified by an audio conference scenario.

[0041] See also Figure 1 , which is a schematic diagram of an application scenario of an audio processing method according to an embodiment of the present application. The application scenario includes a server 110 and a plurality of conference terminals 120 ; the plurality of conference terminals 120 interact with each other through the server 110 .

[0042] Among them, several conference terminals 120 can access the Internet through network access, establish a data transmission channel with the server 110, and thus realize audio data interaction. Among them, the network can be a communication medium of various connection types that can realize communication between the conference terminal 120 and the server 110, for example, it can be a wired communication link, a wireless communication link or an optical fiber cable, etc., and this application does not limit it here.

[0043] The server 110 can be used as a service server, which can be responsible for further connecting related audio data servers, video streaming servers and other servers that provide related support, so as to form a logically related service cluster to serve related terminal devices, such as Figure 1 The conference participant terminal 120 shown in FIG. 1 provides services.

[0044] The conference participant terminal 120 can be understood as an application installed in a computer device, or it can also be understood as a hardware device corresponding to a server. More specifically, the hardware device corresponding to a server refers to a computer device, such as a smart phone, a smart interactive tablet, and a personal computer. It should be understood that when the conference participant terminal 120 is a hardware device such as a smart phone or a smart interactive tablet, the user can install a matching application on the conference participant terminal 120, or access a Web-side application on the conference participant terminal 120.

[0045] In the embodiment of the present application, each participant terminal 120 is installed with an audio conference application, and each participant clicks to access the audio conference application installed on the participant terminal 120 to join the same conference room to conduct a meeting, especially a remote meeting. The above-mentioned conference room refers to a chat room implemented by Internet technology and a server, which usually has audio and video broadcast control functions. Each participant can speak and discuss in the conference room through their respective participant terminals 120.

[0046] Specifically, the participant speaking in the conference room collects his or her own audio data in real time through the participant terminal 120 where he or she is located, and then sends the audio data to the server 110; the server 110 forwards the audio data to other participant terminals 120 for playback, thereby realizing remote conference discussions among all participants.

[0047] It is worth mentioning that Figure 1 The application scenario in is only an exemplary application scenario and is not used to limit the solution of the present invention. The solution of the present invention can also be applied to other forms of audio processing application scenarios, and will be processed using existing data processing methods according to the actual application scenarios. For example, for audio and video conferencing, each participant terminal 120 also collects the participant's own picture data and sends the picture data to the server 110, which mixes the picture data and forwards it to each participant terminal 120 for display, which will not be explained one by one.

[0048] During an audio conference, it is often necessary to record the speech content of the participants. In the related art, the audio in the conference is generally recorded, and the user can obtain the conference content through the recorded audio file.

[0049] However, due to poor recording quality or unclear audio from the speaker, users cannot accurately understand the content of the meeting through the audio file, which affects the users' understanding of the content of the meeting.

[0050] To solve the above technical problems, in another related technology, after the server 110 obtains the audio data of each participant, it also mixes the audio data of each participant. After the meeting, a mixed audio data is generated, and finally speech recognition is performed based on the mixed audio data to obtain the conference text.

[0051] However, the conference text obtained by mixing the audio data cannot effectively distinguish the speech content of each participant, which still affects the user's understanding of the conference content.

[0052] To this end, embodiments of the present application propose an audio processing method, device, storage medium, and electronic device.

[0053] The following will be combined with the attached Figures 2 to 9 , the audio processing method provided in the embodiment of the present application is introduced in detail.

[0054] See also Figure 2 , which is a flowchart of an audio processing method provided by an embodiment of the present application. The audio processing method provided by the embodiment of the present application comprises the following steps:

[0055] Step S101: Acquire the audio data of each participant, the participant identifier corresponding to each audio data, the first time identifier of each audio data, the mixed audio data, and the second time identifier of the mixed audio data; the mixed audio data is obtained by mixing the audio data of each participant.

[0056] It is understandable that the audio processing method of the embodiment of the present application can be executed by one of the participating terminals as the execution subject, or by the server as the execution subject, or even by other electronic devices such as recording devices connected to the server or the participating terminals as the execution subject. The embodiment of the present application is exemplified by taking the server as the execution subject.

[0057] When the server detects the start of the meeting, it records the audio conference. Specifically, when the server detects that at least one participant has entered the conference room or has arrived at the scheduled meeting start time, it starts to receive the audio data transmitted by the participants who have joined the conference room through their respective participants, and obtains the participant identifier corresponding to the audio data and the first time identifier of each audio data. The server also mixes the audio data of each participant to obtain mixed audio data, and at the same time, the server also determines the second time identifier of the mixed audio data.

[0058] It should be understood that the server can mix the audio data in real time, thereby obtaining the mixed speech text data associated with the participant identification in real time, and then displaying the mixed speech text data in real time in the conference room. The server can also mix the acquired audio data after the meeting, thereby obtaining the mixed speech text data recording the content of the meeting after the meeting. The embodiments of the present application are not limited to this.

[0059] The first time identifier is used to identify the audio data. In an optional embodiment, the first time identifier may be a time point; specifically, the first time identifier may be the time point when the participant joins the conference, the time point when the participant starts to mic, or even the middle time point of the entire audio data. The first time identifier may be a time determined based on the system time of the server, or a time determined based on the system time of the participant terminal where the audio data is located.

[0060] It should be understood that the first time mark is the time point when the participant joins the meeting. Considering that the participant may exit the meeting after joining the meeting, and then join and exit the meeting many times, an audio data will be recorded for each time the participant joins the meeting, and the corresponding first time mark will be recorded. Similarly, when the first time mark is the time point when the participant opens the microphone, considering that the participant may open and close the microphone many times during the meeting, an audio data will be recorded for each time the participant opens the microphone, and the corresponding first time mark will be recorded.

[0061] The second time identifier is used to identify the mixed audio data. In an optional embodiment, the second time identifier may be a time point; specifically, the second time identifier may be the time point when the meeting starts, or the time point when the first participant enters the meeting room. The second time identifier may be a time determined based on the system time of the server, or a time determined based on the system time of the participant terminal that triggers the start of the meeting.

[0062] It should be understood that the audio data and mixed audio data of the present application include not only the audio itself, but also the audio time, which includes at least the various times when this audio is played, which can be a time marked by a duration, or a specific time. For example, if the audio is 10 seconds, then when the audio is played from the beginning to the end, the time marked with a duration of 10 seconds as a reference can be displayed, such as the 1st second, the 2nd second... the 10th second; or, when the audio is played from the beginning to the end, a specific time can be displayed, such as January 1st 1:01 second, January 1st 1:02 seconds... January 1st 1:10 seconds.

[0063] Step S102: Identify each audio data and obtain speech text data.

[0064] In an optional implementation, the server stores a speech recognition algorithm, and the server recognizes the audio data of each participant according to the speech recognition algorithm to obtain the speech text data of each participant.

[0065] In another optional implementation, the server stores a trained speech recognition model, and the server inputs the audio data of each participant into the speech recognition model to obtain the speech text data of each participant. The trained speech recognition model can be obtained by training an existing neural network model with an existing speech data set.

[0066] In an optional embodiment, in order to speed up the efficiency of acquiring speech text data and reduce the processing burden of a single server, at least two servers are used for processing, one of which is a business processing server and the other is an audio data processing server; the business server receives the audio data of each participant in real time, mixes the audio data of each participant, and forwards the audio data to the audio data processing server; the audio data processing server identifies the audio data of each participant to obtain the speech text data of each participant.

[0067] It is understandable that, since the audio data includes the audio itself and the audio time, the speech text data obtained will also be the text corresponding to the audio time.

[0068] Step S103: aligning each participant identifier, speech text data and mixed audio data according to each first time identifier and second time identifier, and obtaining mixed speech text data associated with the participant identifier.

[0069] It can be understood that since the mixed audio data is obtained by mixing the audio data of each participant, the corresponding position of each audio data in the mixed audio data can be determined according to each first time marker and the second time marker, so that the participant identifier and speech text data corresponding to each audio data are aligned with the mixed audio data, and the aligned participant identifier and speech text data are spliced ​​and integrated to obtain the mixed speech text data associated with the participant identifier.

[0070] The embodiment of the present application obtains the audio data of each participant, the participant identifier corresponding to each audio data, the first time identifier of each audio data, the mixed audio data and the second time identifier of the mixed audio data; the mixed audio data is obtained by mixing the audio data of each participant; each audio data is identified to obtain speech text data; according to each first time identifier and second time identifier, each participant identifier, the speech text data and the mixed audio data are aligned to obtain the mixed speech text data associated with the participant identifier, so that the speech content of each participant can be effectively distinguished based on the mixed speech text number, so as to facilitate users to understand the content of the meeting.

[0071] See also Figure 3 In an optional embodiment, the step of obtaining the audio data of each participant and the participant identifier corresponding to each audio data in step S101 includes:

[0072] Step S111: receiving a speech operation request from each participant; the speech operation request includes a participant identifier.

[0073] In an optional embodiment, the speech operation request may be a request from each participant to join the meeting. Specifically, after the participant joins the meeting according to the conference identifier, the participant terminal will obtain a pre-stored participant identifier or receive a participant identifier input by the participant, and generate a speech operation request based on the participant identifier and send it to the server.

[0074] In another optional embodiment, the speaking operation request may also be a microphone opening request of each participant. Specifically, after the participant triggers the microphone opening control, the participant terminal will obtain the pre-stored participant identifier, generate a speaking operation request based on the participant identifier, and send it to the server.

[0075] Step S112: determining the audio data of each participant and the participant identifier corresponding to each audio data according to the participant identifier of each participant, the data transmission channel where each speech operation request is located, and the audio data transmitted by each data transmission channel.

[0076] In an optional embodiment, each participant joining the conference will establish a data transmission channel with the server. The server can identify the data transmission channel established with each participant to determine the corresponding speech operation request, and then based on the participant identification carried in the speech operation request and the audio data transmitted by the data transmission channel, determine the audio data of each participant and the participant identification corresponding to each audio data.

[0077] In another optional embodiment, each participant joining the conference will establish a data transmission channel with the server. Each time each participant transmits audio data, it will carry its participant identification. The data transmission channel for the audio data can be determined based on the participant identification, thereby determining the corresponding speech operation request. Based on the participant identification carried in the speech operation request and the audio data transmitted through the data transmission channel, the audio data of each participant and the participant identification corresponding to each audio data can be determined.

[0078] The embodiment of the present application can conveniently and accurately obtain the audio data of each participant and the participant identifier corresponding to each audio data based on the participant identifier of each participant, the data transmission channel where each speech operation request is located, and the audio data transmitted by each data transmission channel.

[0079] See also Figure 4 In an optional embodiment, the step of obtaining the first time mark of each audio data in step S101 includes:

[0080] Step S121: receiving a speaking operation request from each participant.

[0081] Step S122: determining the time of receiving the speech operation request of each participant as the first time mark of each audio data transmitted based on the speech operation request.

[0082] In an optional embodiment, the speaking operation request may be a request for each participant to join the conference, and the first time identifier of each audio data is the time when the server receives the time when each participant joins the conference.

[0083] In another optional embodiment, the speaking operation request may also be a request for each participant to turn on the microphone, and the first time identifier of each audio data is the time when the server receives the request for each participant to turn on the microphone.

[0084] The embodiment of the present application takes into account network delays and differences in system time between different devices, and uniformly sets the first time identifier of each audio data to be determined by the server system time, thereby avoiding time errors caused by device differences and network delays, thereby ensuring that the participant identifiers, speech text data and mixed audio data of subsequent audio data are aligned.

[0085] See also Figure 5 In an optional embodiment, the step of obtaining the second time mark of the mixed audio data in step S101 includes:

[0086] Step S131: monitoring the conference start operation.

[0087] Step S132: Determine the time corresponding to the start of the conference as the second time marker of the mixed audio data.

[0088] It is understandable that the conference start operation may be the time when the scheduled conference start time arrives, or the time when the first participant joins the conference room.

[0089] In the embodiment of the present application, the server determines the time corresponding to the start of the meeting as the second time identifier of the mixed audio data, and then sets the second time identifier of the mixed audio data to be determined according to the server system time, which can ensure that the participant identifiers and speech text data of subsequent audio data are aligned with the mixed audio data.

[0090] See also Figure 6 In an optional embodiment, in step S103, the step of aligning each participant identifier, speech text data and mixed audio data according to each first time identifier and second time identifier to obtain mixed speech text data associated with the participant identifier includes:

[0091] Step S1031: Obtain a time offset of each audio data relative to the mixed stream audio data according to each first time marker and the second time marker.

[0092] Step S1032: according to each time offset, align the participant identifier corresponding to each audio data, the corresponding speech text data and the mixed audio data to obtain the mixed speech text data associated with the participant identifier.

[0093] It can be understood that since the mixed audio data is obtained by mixing individual audio data, and each audio data and the mixed audio data themselves carry time data, therefore, after obtaining the time offset of each audio data relative to the mixed audio data according to each first time marker and the second time marker, the individual audio data and the mixed audio data can be aligned with each other based on time. Since each audio data can be identified as speech text data, and each audio data corresponds to a participant identifier, the participant identifier corresponding to each audio data and the corresponding speech text data can be aligned with the mixed audio data to obtain the mixed speech text data associated with the participant identifier.

[0094] For example, Figure 7 As shown, when the first time identifier is the microphone start time of each participant and the second time identifier is the start time of the meeting, the time offset of each audio data relative to the mixed audio data is obtained according to each first time identifier and the second time identifier, and then the audio data of each participant A, participant B, participant C and participant D can be aligned with the mixed audio data based on time, so that the participant identifiers of participant A, participant B, participant C and participant D, the corresponding speech text data and the mixed audio data are aligned to obtain the mixed speech text data associated with the participant identifier.

[0095] It is understandable that if Figure 7 As shown, considering the recording time and volume of the mixed audio data, the mixed audio will be segmented at intervals to obtain several mixed audio segments, and then the alignment operation will be performed after the mixed audio segments are spliced.

[0096] The embodiment of the present application aligns each audio data with the mixed audio data based on the time offset of each audio data relative to the mixed audio data, and aligns the corresponding participant identification and the corresponding speech text data with the mixed audio data. Compared with the solution of directly identifying the mixed audio data as mixed speech text data, the present application can avoid the problem that when each participant speaks at the same time point, the speech content of each participant cannot be effectively distinguished based on the mixed audio data. Compared with the solution of directly splicing the speech text data of each audio data into mixed speech text data, the present application does not need to find the splicing time point, and can also avoid the problem of mixed speech text errors caused by splicing time point errors.

[0097] In an optional embodiment, the step of obtaining the time offset of each audio data relative to the mixed stream audio data according to each first time identifier and the second time identifier in step S1032 includes:

[0098] Step S1321: Subtract the second time marker from each first time marker to obtain a time offset of each audio data relative to the mixed stream audio data.

[0099] In the embodiment of the present application, the time offset of each audio data relative to the mixed audio data can be conveniently obtained by subtracting the second time marker from each first time marker.

[0100] See also Figure 8 In an optional embodiment, the audio processing method of the embodiment of the present application further includes:

[0101] Step S104: when playing the mixed audio data, the mixed speech text data associated with the participant's ID is processed into subtitles with the participant's ID added, and the subtitles with the participant's ID added are displayed.

[0102] The mixed speech text data is used as the subtitles of the video stream data and displayed, so that the corresponding participant identification is added to each subtitle to distinguish the subtitles corresponding to the audio data of different participants. Specifically, when playing the mixed audio data of the audio conference, a subtitle interface can be displayed, and the corresponding subtitles can be played on the subtitle interface. Fig. 9 As shown, the participant logos and subtitles may be displayed below the played video stream data, and / or on the right side of the played video stream data, and this is not necessarily limited in this embodiment of the present invention.

[0103] When playing mixed audio data, the embodiment of the present application processes the mixed speech text data associated with the participant's identifier into subtitles with the participant's identifier added, and displays the subtitles with the participant's identifier added, so that the user can accurately obtain the conference content by combining the mixed audio data and the subtitles.

[0104] The embodiment of the present application also provides an audio processing system, including: a server and a plurality of conference terminals; each conference terminal is connected to the server;

[0105] Each participant terminal sends the audio data of each participant and the participant identifier corresponding to each audio data to the server;

[0106] The server is used to obtain audio data of each participant, a participant identifier corresponding to each audio data, a first time identifier of each audio data, mixed audio data, and a second time identifier of the mixed audio data; the mixed audio data is obtained by mixing the audio data of each participant;

[0107] The server is used to identify each audio data and obtain speech text data;

[0108] The server is used to align each participant identifier, speech text data and mixed audio data according to each first time identifier and second time identifier, and obtain the mixed speech text data associated with the participant identifier.

[0109] The audio processing system provided in the embodiment of the present application and the audio processing method provided in the embodiment of the present application belong to the same concept, and the implementation process thereof is detailed in the method embodiment and will not be repeated here.

[0110] See also Fig.10 , which is a schematic diagram of the structure of an audio processing device provided by an embodiment of the present application. The device 200 includes:

[0111] The data acquisition module 201 is used to acquire the audio data of each participant, the participant identifier corresponding to each audio data, the first time identifier of each audio data, the mixed audio data and the second time identifier of the mixed audio data; the mixed audio data is obtained by mixing the audio data of each participant;

[0112] A text recognition module 202 is used to recognize each audio data and obtain speech text data;

[0113] The speech text acquisition module 203 is used to align each participant identifier, speech text data and mixed audio data according to each first time identifier and second time identifier, and obtain the mixed speech text data associated with the participant identifier.

[0114] It should be noted that when the audio processing device provided in the embodiment of the present application executes the audio processing method, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the audio processing device provided in the embodiment of the present application and the audio processing method provided in the embodiment of the present application belong to the same concept. The implementation process is detailed in the method embodiment and will not be repeated here.

[0115] The embodiment of the audio processing device provided in the embodiment of the present application can be applied on a computer device, such as a server, and the device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the file processing in which it is located reading the corresponding computer program instructions in the memory and running. From the hardware level, the computer device in which it is located may include a processor and a memory, and the processor and the memory are connected via a data bus or other known methods.

[0116] See also Fig.11 , is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Fig.11 As shown, the electronic device 300 can be specifically a computer, a mobile phone, a tablet computer, an interactive tablet, etc. In an exemplary embodiment of the present application, the electronic device 300 is a server, and the electronic device 300 may include: at least one processor 310, at least one memory 320, at least one network interface 330, a user interface 340 and at least one communication bus 350.

[0117] The communication bus 350 is used to realize the connection and communication between these components.

[0118] The user interface 340 may include a display screen and a camera; the user interface 340 may also include a standard wired interface and a wireless interface.

[0119] The network interface 330 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0120] Among them, the processor 310 may include one or more processing cores. The processor 310 uses various interfaces and lines to connect various parts within the entire electronic device 300, and executes various functions and processes data of the electronic device 300 by running or executing instructions, programs, code sets or instruction sets stored in the memory 320, and calling data stored in the memory 320. Optionally, the processor 310 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 310 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed by the display layer; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 310, but implemented separately by a chip.

[0121] Among them, the memory 320 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 320 includes a non-transitory computer-readable storage medium. The memory 320 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 320 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 320 may also be optionally at least one storage device located away from the aforementioned processor 310. As Fig.11 As shown, the memory 320 as a computer storage medium may include an operating system, a network communication module, and a user.

[0122] exist Fig.11 In the electronic device 300 shown, the user interface 340 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 310 can be used to call the server operation application stored in the memory 320, such as: an audio processing program; and execute the relevant operations of any audio processing method in the above-mentioned embodiments, with corresponding functions and beneficial effects.

[0123] The present application also provides a computer-readable storage medium on which a computer program is stored, and the instructions are suitable for being loaded by a processor and executing the steps of the audio processing method shown above. The specific execution process can refer to the specific description shown in the embodiment, and will not be repeated here. The device where the storage medium is located can be a personal computer, a laptop computer, a smart phone, a tablet computer and other electronic devices.

[0124] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0125] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0126] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the function selected in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 function selected in a box or multiple boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 steps for the function selected in a box or multiple boxes.

[0128] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0129] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0130] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0131] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0132] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. An audio processing method, characterized in that: The steps include: Acquire audio data of each participant, a participant identifier corresponding to each audio data, a first time identifier of each audio data, mixed audio data, and a second time identifier of the mixed audio data; the mixed audio data is obtained by mixing the audio data of each participant; Identify each of the audio data and obtain speech text data; According to each of the first time markers and the second time markers, each of the participant identifiers, the speech text data and the mixed audio data are aligned to obtain the mixed speech text data associated with the participant identifier.

2. The audio processing method according to claim 1, characterized in that: The step of aligning each of the participant identifiers, the speech text data and the mixed audio data according to each of the first time identifiers and the second time identifier to obtain the mixed speech text data associated with the participant identifier includes: Obtaining, according to each of the first time markers and the second time markers, a time offset of each of the audio data relative to the mixed stream audio data; According to each of the time offsets, the participant identifiers corresponding to each of the audio data and the corresponding speech text data are aligned with the mixed audio data to obtain the mixed speech text data associated with the participant identifiers.

3. The audio processing method according to claim 2, characterized in that: The step of obtaining the time offset of each audio data relative to the mixed audio data according to each first time identifier and the second time identifier includes: The second time marker is subtracted from each of the first time markers to obtain a time offset of each of the audio data relative to the mixed audio data.

4. The audio processing method according to any one of claims 1 to 3, characterized in that , The step of obtaining the first time mark of each of the audio data comprises: Receive speaking operation requests from all participants; The time of receiving the speech operation request of each conference participant is determined as the first time identifier of each audio data transmitted based on the speech operation request.

5. The audio processing method according to any one of claims 1 to 3, characterized in that: The step of obtaining a second time mark of the mixed stream audio data comprises: Monitor the conference and start the operation; The time corresponding to the start of the conference is determined as the second time mark of the mixed audio data.

6. The audio processing method according to claim 1, characterized in that: The step of obtaining the audio data of each participant and the participant identifier corresponding to each audio data includes: Receiving a speech operation request from each participant; the speech operation request includes a participant identifier; The audio data of each participant and the participant identifier corresponding to each audio data are determined according to the participant identifier of each participant, the data transmission channel where each speech operation request is located, and the audio data transmitted by each data transmission channel.

7. The audio processing method according to claim 1, characterized in that: The following steps are also included: When playing the mixed audio data, the mixed speech text data associated with the participant identifier is processed into subtitles added with the participant identifier, and the subtitles added with the participant identifier are displayed.

8. An audio processing device, characterized in that: include: A data acquisition module, used to acquire audio data of each participant, a participant identifier corresponding to each audio data, a first time identifier of each audio data, mixed audio data, and a second time identifier of the mixed audio data; the mixed audio data is obtained by mixing the audio data of each participant; A text recognition module, used to recognize each of the audio data and obtain speech text data; The speech text acquisition module is used to align each of the participant identifiers, the speech text data and the mixed audio data according to each of the first time identifiers and the second time identifier, and obtain the mixed speech text data associated with the participant identifier.

9. An audio processing system, comprising: A server and a plurality of conference terminals; each of the conference terminals is connected to the server; characterized in that: Each of the conference participants sends the audio data of each participant and the participant identifier corresponding to each of the audio data to the server; The server is used to obtain audio data of each participant, a participant identifier corresponding to each audio data, a first time identifier of each audio data, mixed audio data, and a second time identifier of the mixed audio data; the mixed audio data is obtained by mixing the audio data of each participant; The server is used to identify each of the audio data and obtain speech text data; The server is used for aligning each of the participant identifiers, the speech text data and the mixed audio data according to each of the first time identifiers and the second time identifier, and obtaining the mixed speech text data associated with the participant identifier.

10. An electronic device comprising a processor and a memory; characterized in that: The memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the audio processing method according to any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the audio processing method according to any one of claims 1 to 7 is implemented.

Images