Audio data processing method and device

Abstract

The invention relates to the field of data processing, in particular to an audio data processing method and device. The method comprises the following steps: acquiring first audio data; the first audio data comprises conversation records between the human and the artificial intelligence voice system. And dividing second audio data and third audio data from the first audio data. The second audio data comprises a part in which the human and the artificial intelligence voice system make sounds simultaneously, and the third audio data comprises a part in which the human makes sounds independently. Inputting the second audio data into the first model to obtain fourth audio data output by the first model; the first model is a deep neural network model used for removing the sound part of the artificial intelligence voice system. And inputting the fourth audio data into the second model to obtain fifth audio data output by the second model. The second model is a long-short-term memory network model used for removing the sound part of the artificial intelligence voice system. And splicing the fifth audio data and the third audio data to obtain sixth audio data.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method and device for processing audio data. Background Art

[0002] At present, with the development of artificial intelligence (AI) technology, more and more scenarios (such as voice assistants, telephone customer service, and smart homes) are beginning to use artificial intelligence voice systems to perform corresponding tasks.

[0003] Specifically, when using an artificial intelligence voice system to perform corresponding tasks, it is usually necessary to first collect a recording of the conversation between humans and the artificial intelligence voice system, then obtain the content of what the human said from the conversation recording, and then perform the corresponding task based on what the human said.

[0004] However, the audio data collected by the device, including recordings of conversations between humans and artificial intelligence voice systems, may include situations where humans and the artificial intelligence voice system speak at the same time, which will affect the accuracy of the content spoken by humans obtained from the audio data. Summary of the Invention

[0005] In order to solve the above technical problems, the present application provides an audio data processing method and device.

[0006] In a first aspect, the present application provides an audio data processing method, the method comprising: obtaining first audio data; the first audio data includes a recording of a conversation between a human and an artificial intelligence voice system; dividing second audio data and third audio data from the first audio data; the second audio data includes the part of the first audio data in which the human and the artificial intelligence voice system speak simultaneously, and the third audio data includes the part of the first audio data in which the human speaks alone; inputting the second audio data into a first model to obtain fourth audio data output by the first model; the first model is: a deep neural network model for removing the part of the artificial intelligence voice system from the audio data in which the human and the artificial intelligence voice system speak simultaneously; inputting the fourth audio data into a second model to obtain fifth audio data output by the second model; the second model is: a long short-term memory network model for removing the part of the artificial intelligence voice system from the audio data in which the human and the artificial intelligence voice system speak simultaneously; splicing the fifth audio data and the third audio data to obtain sixth audio data.

[0007] In the above method of the present application, on the one hand, after obtaining the audio data (i.e., the first audio data) including the recording of the conversation between humans and the artificial intelligence voice system, the part in which humans and the artificial intelligence voice system speak at the same time (i.e., the second audio data) and the part in which humans speak alone (i.e., the third audio data) can be divided from the first audio data. For the second audio data, the part in which humans speak in the second audio data (i.e., the fifth audio data) can be obtained by removing the part in which the artificial intelligence voice system speaks in the second audio data. Then, the sixth audio data can be obtained by splicing the fifth audio data and the third audio data. Among them, since the sixth audio data includes the third audio data (i.e., the part in which humans speak alone in the original first audio data) and the part in the second audio data in which humans speak (i.e., the fifth audio data), the sixth audio data can be used to obtain more accurate content spoken by humans.

[0008] On the other hand, in the process of removing the part of the second audio data produced by the artificial intelligence voice system, the Deep Neural Networks (DNNs) model can be used to remove the part of the second audio data produced by the artificial intelligence voice system to obtain the fourth audio data; and then the Long-Short Term Memory (LSTM) model can be used to remove the part of the fourth audio data produced by the artificial intelligence voice system to obtain the fifth audio data. Among them, since the DNNs model can extract high-dimensional features in the audio data, and the LSTM model can capture the temporal changes and dependencies in the audio data, it can better capture the dynamic characteristics of the audio. Therefore, by first using the DNNs model to remove the part of the second audio data produced by the artificial intelligence voice system to obtain the fourth audio data, and then using the LSTM model to remove the part of the fourth audio data produced by the artificial intelligence voice system to obtain the fifth audio data, the part of the second audio data produced by the artificial intelligence voice system can be more accurately removed from the second audio data, so that the fifth audio data includes a more accurate human voice. In addition, the present embodiment takes into account that the computational process of the DNNs model is relatively easy to parallelize. When processing large-scale audio data, the feature extraction task can be completed quickly, making the processing speed of the entire model faster. Therefore, the DNNs model can be used first and then the LSTM model to remove the part of the second audio data that is spoken by the artificial intelligence voice system. This can reduce the processing burden of the LSTM, help the LSTM to more accurately capture time series features, and improve the overall processing effect.

[0009] In some implementations, dividing the second audio data and the third audio data from the first audio data includes: dividing the second audio data and the third audio data from the first audio data based on one or more of short-time energy, spectrum distribution, and time domain waveform corresponding to the first audio data.

[0010] In some implementations, dividing the second audio data and the third audio data from the first audio data includes: inputting the first audio data into a third model to obtain the second audio data and the third audio data output by the third model; wherein the third model is: a voice activity detection model for dividing the audio data including the recording of the conversation between the human and the artificial intelligence voice system into the parts where the human and the artificial intelligence voice system speak simultaneously and the parts where the human speaks alone.

[0011] In some implementations, the method further includes: inputting the sixth audio data into a fourth model to obtain seventh audio data output by the fourth model; the fourth model is: a deep complex convolutional recurrent network model for eliminating noise in audio data.

[0012] In some implementations, the method is applied to a server; and obtaining the first audio data specifically includes: receiving the first audio data from a terminal device.

[0013] In some implementations, the first model is a deep neural network model obtained by supervised learning using a first training sample set; wherein the first training sample set includes: multiple first sample audio data including recordings of human speech, and multiple second sample audio data obtained by adding the sound of an artificial intelligence voice system to the multiple first sample audio data.

[0014] In some implementations, the second model is a long short-term memory network model obtained by supervised learning using a second training sample set; wherein the first training sample set includes: multiple third sample audio data including recordings of human speech, and multiple fourth sample audio data; wherein the multiple fourth sample audio data are audio data output by the first model after multiple fifth sample audio data are input into the first model; the multiple fifth sample audio data are audio data obtained by adding the sound of an artificial intelligence voice system to the multiple third sample audio data.

[0015] In a second aspect, an audio data processing device is provided, including: an acquisition unit for acquiring first audio data; the first audio data includes a recording of a conversation between a human and an artificial intelligence voice system; a processing unit for dividing the first audio data into second audio data and third audio data; the second audio data includes the part of the first audio data in which the human and the artificial intelligence voice system speak simultaneously, and the third audio data includes the part of the first audio data in which the human speaks alone; the processing unit is further used to input the second audio data into a first model to obtain fourth audio data output by the first model; the first model is: a deep neural network model for removing the part of the artificial intelligence voice system from the audio data in which the human and the artificial intelligence voice system speak simultaneously; the processing unit is further used to input the fourth audio data into a second model to obtain fifth audio data output by the second model; the second model is: a long short-term memory network model for removing the part of the artificial intelligence voice system from the audio data in which the human and the artificial intelligence voice system speak simultaneously; the processing unit is further used to splice the fifth audio data and the third audio data to obtain sixth audio data.

[0016] In one implementation, a processing unit, configured to separate second audio data and third audio data from the first audio data, includes: a processing unit, configured to separate the second audio data and the third audio data from the first audio data based on one or more of short-time energy, spectral distribution, and time domain waveform corresponding to the first audio data.

[0017] In one implementation, a processing unit, for separating second audio data and third audio data from the first audio data, includes: a processing unit, for inputting the first audio data into a third model to obtain second audio data and third audio data output by the third model; wherein the third model is: a voice activity detection model for separating, from audio data including a recording of a conversation between a human and an artificial intelligence voice system, the parts in which a human and the artificial intelligence voice system speak simultaneously and the parts in which a human speaks alone.

[0018] In one implementation, the processing unit is further used to input the sixth audio data into a fourth model to obtain seventh audio data output by the fourth model; the fourth model is: a deep complex convolutional recurrent network model for eliminating noise in audio data.

[0019] In one implementation, the audio data processing device is applied to a server. The acquiring unit is configured to acquire the first audio data, specifically comprising: an acquiring unit configured to receive the first audio data from a terminal device.

[0020] In one implementation, the first model is a deep neural network model obtained by supervised learning using a first training sample set; wherein the first training sample set includes: a plurality of first sample audio data including recordings of human speech, and a plurality of second sample audio data obtained by adding the sound of an artificial intelligence voice system to the plurality of first sample audio data.

[0021] In one implementation, the second model is a long short-term memory network model obtained by supervised learning using a second training sample set; wherein the first training sample set includes: multiple third sample audio data including recordings of human speech, and multiple fourth sample audio data; wherein the multiple fourth sample audio data are audio data output by the first model after multiple fifth sample audio data are input into the first model; the multiple fifth sample audio data are audio data obtained by adding the sound of an artificial intelligence voice system to the multiple third sample audio data.

[0022] The technical solution provided by the embodiments of the present application has the following advantages compared with the prior art:

[0023] In the above method of the present application, on the one hand, after obtaining the audio data (i.e., the first audio data) including the recording of the conversation between humans and the artificial intelligence voice system, the part in which humans and the artificial intelligence voice system speak at the same time (i.e., the second audio data) and the part in which humans speak alone (i.e., the third audio data) can be divided from the first audio data. For the second audio data, the part in which humans speak in the second audio data (i.e., the fifth audio data) can be obtained by removing the part in which the artificial intelligence voice system speaks in the second audio data. Then, the sixth audio data can be obtained by splicing the fifth audio data and the third audio data. Among them, since the sixth audio data includes the third audio data (i.e., the part in which humans speak alone in the original first audio data) and the part in the second audio data in which humans speak (i.e., the fifth audio data), the sixth audio data can be used to obtain more accurate content spoken by humans.

[0024] On the other hand, in the process of removing the part of the second audio data produced by the artificial intelligence voice system, the Deep Neural Networks (DNNs) model can be used to remove the part of the second audio data produced by the artificial intelligence voice system to obtain the fourth audio data; and then the Long-Short Term Memory (LSTM) model can be used to remove the part of the fourth audio data produced by the artificial intelligence voice system to obtain the fifth audio data. Among them, since the DNNs model can extract high-dimensional features in the audio data, and the LSTM model can capture the temporal changes and dependencies in the audio data, it can better capture the dynamic characteristics of the audio. Therefore, by first using the DNNs model to remove the part of the second audio data produced by the artificial intelligence voice system to obtain the fourth audio data, and then using the LSTM model to remove the part of the fourth audio data produced by the artificial intelligence voice system to obtain the fifth audio data, the part of the second audio data produced by the artificial intelligence voice system can be more accurately removed from the second audio data, so that the fifth audio data includes a more accurate human voice. In addition, the present embodiment takes into account that the computational process of the DNNs model is relatively easy to parallelize. When processing large-scale audio data, the feature extraction task can be completed quickly, making the processing speed of the entire model faster. Therefore, the DNNs model can be used first and then the LSTM model to remove the part of the second audio data that is spoken by the artificial intelligence voice system. This can reduce the processing burden of the LSTM, help the LSTM to more accurately capture time series features, and improve the overall processing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0027] Figure 1 One of the flowcharts of an audio data processing method provided in an embodiment of the present application;

[0028] Figure 2 A second flowchart of an audio data processing method provided in an embodiment of the present application;

[0029] Figure 3 A third flowchart of an audio data processing method provided in an embodiment of the present application;

[0030] Figure 4 A fourth flowchart of an audio data processing method provided in an embodiment of the present application;

[0031] Figure 5 A schematic structural diagram of a first model provided in an embodiment of the present application;

[0032] Figure 6 A schematic structural diagram of a second model provided in an embodiment of the present application;

[0033] Figure 7 A fifth flowchart of an audio data processing method provided in an embodiment of the present application;

[0034] Figure 8 This is one of the structural diagrams of an audio data processing device provided in an embodiment of the present application;

[0035] Figure 9 This is a second structural diagram of an audio data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to more clearly understand the above-mentioned objectives, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0037] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application can also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present application, not all of the embodiments.

[0038] The technical solutions provided in the embodiments of the present application are introduced below with reference to examples.

[0039] As mentioned in the background technology, in the related technology, when using an artificial intelligence voice system to perform corresponding tasks, since the audio data collected by the device may include situations where humans and the artificial intelligence voice system speak at the same time, it will affect the accuracy of the content spoken by humans obtained from the audio data.

[0040] In view of the above problems, the embodiments of the present application take into account:

[0041] On the one hand, after obtaining audio data including a recording of a conversation between a human and an artificial intelligence voice system (hereinafter referred to as the first audio data), the part in which the human and the artificial intelligence voice system speak simultaneously (hereinafter referred to as the second audio data) and the part in which the human speaks alone (hereinafter referred to as the third audio data) can be divided from the first audio data. For the second audio data, the part in which the human speaks in the second audio data (hereinafter referred to as the fifth audio data) can be obtained by removing the part in which the artificial intelligence voice system speaks in the second audio data. Then, the sixth audio data can be obtained by splicing the fifth audio data and the third audio data. Among them, since the sixth audio data includes the third audio data (that is, the part in which the human speaks alone in the original first audio data) and the part in the second audio data in which the human speaks (that is, the fifth audio data), the sixth audio data can be used to obtain more accurate content spoken by humans.

[0042] On the other hand, in the process of removing the part of the second audio data produced by the artificial intelligence voice system, the Deep Neural Networks (DNNs) model can be used to remove the part of the second audio data produced by the artificial intelligence voice system to obtain the fourth audio data; and then the Long-Short Term Memory (LSTM) model can be used to remove the part of the fourth audio data produced by the artificial intelligence voice system to obtain the fifth audio data. Among them, since the DNNs model can extract high-dimensional features in the audio data, and the LSTM model can capture the temporal changes and dependencies in the audio data, it can better capture the dynamic characteristics of the audio. Therefore, by first using the DNNs model to remove the part of the second audio data produced by the artificial intelligence voice system to obtain the fourth audio data, and then using the LSTM model to remove the part of the fourth audio data produced by the artificial intelligence voice system to obtain the fifth audio data, the part of the second audio data produced by the artificial intelligence voice system can be more accurately removed from the second audio data, so that the fifth audio data includes a more accurate human voice. In addition, the present embodiment takes into account that the computational process of the DNNs model is relatively easy to parallelize. When processing large-scale audio data, the feature extraction task can be completed quickly, making the processing speed of the entire model faster. Therefore, the DNNs model can be used first and then the LSTM model to remove the part of the second audio data that is spoken by the artificial intelligence voice system. This can reduce the processing burden of the LSTM, help the LSTM to more accurately capture time series features, and improve the overall processing effect.

[0043] The following is a detailed description of the technical solutions provided by the embodiments of the present application, with reference to examples. Specifically, the embodiments of the present application provide an audio data processing method. The audio data processing method may be executed by an audio data processing device. When the audio data processing device is running, it can be used to execute all or part of the steps of the audio data processing method provided by the embodiments of the present application.

[0044] In actual application, the functions of the audio data processing device can be implemented by a personal computer (including desktop computers, laptop computers, handheld computers, and notebook computers), or electronic devices such as smart phones and servers; or the functions of the above-mentioned audio data processing device can also be implemented by some hardware / software devices in the above-mentioned electronic devices. The specific form of the data processing device is not particularly limited in the embodiments of the present application.

[0045] like Figure 1 As shown, the method provided in the embodiment of the present application may include the following contents S101-S105:

[0046] S101: An audio data processing apparatus obtains first audio data.

[0047] Among them, the first audio data includes a recording of a conversation between a human and an artificial intelligence voice system.

[0048] For example, the first audio data may be audio data recorded by a terminal device when a user answers a call using a speakerphone in a scenario where an artificial intelligence voice system is used for telephone customer service. In this case, because the user answers the call using the speakerphone, the first audio data recorded by the terminal device includes both the human voice and the voice of the artificial intelligence voice system.

[0049] In addition, the first audio data may also include portions of the voices spoken simultaneously by the human and the artificial intelligence voice system. For example, if the user speaks while the artificial intelligence voice system is still active, the acquired first audio data will include portions of the voices spoken simultaneously by the human and the artificial intelligence voice system.

[0050] In some implementations, the audio data processing method provided in the embodiments of the present application can be applied in a server. The above-mentioned audio data processing device can be specifically a server or a part of the hardware / software device in the server. Figure 2 As shown, S101 may specifically include:

[0051] S101a: The audio data processing apparatus receives first audio data from a terminal device.

[0052] In the above implementation, it is taken into account that different terminal devices use different systems and hardware configurations, and therefore have different audio data processing capabilities, which leads to different audio effects after processing. Therefore, in the above implementation, by applying the audio data processing method provided in the embodiment of the present application to the server, the differences between different devices and systems can be effectively smoothed out, ensuring the consistency of the processed audio effects.

[0053] S102: The audio data processing apparatus divides the first audio data into second audio data and third audio data.

[0054] Among them, the second audio data includes the part of the first audio data where humans and the artificial intelligence voice system speak simultaneously, and the third audio data includes the part of the first audio data where humans speak alone.

[0055] Specifically, in some implementations, such as Figure 3 As shown, S102 may specifically include the following S102a:

[0056] S102a: The audio data processing apparatus divides the first audio data into second audio data and third audio data according to one or more of the short-time energy, spectrum distribution, and time-domain waveform corresponding to the first audio data.

[0057] The following describes the implementation process of S102a in three aspects:

[0058] First, consider that the short-term energy of the portion of the first audio data where both the human and the AI ​​speech system speak simultaneously is generally higher, while the short-term energy of the portion where the human speaks alone and the portion where the AI ​​speech system speaks alone is generally lower. Short-term energy is a key characteristic parameter in speech signal processing. It is a calculation of the energy of the speech signal within a short time window. Simply put, it is a physical quantity that measures the strength of the speech signal over a short period of time.

[0059] Therefore, the second audio data and the third audio data can be divided from the first audio data according to the short-time energy corresponding to the first audio data. For example, in actual application, on the one hand, the part with higher short-time energy (for example, the part with short-time energy higher than the first threshold) can be determined from the first audio data as the second audio data. On the other hand, the part with lower short-time energy (for example, the part with short-time energy lower than the second threshold) can be determined from the first audio data. This part may include the part where humans speak alone and the part where the artificial intelligence voice system speaks alone, and then the part where humans speak alone is divided from the part with lower short-time energy as the third audio data.

[0060] Secondly, considering that in the first audio data, the simultaneous sounds of the human and the AI ​​voice system will exhibit peaks or variations within a specific frequency range, while the spectra of the human and AI voice systems alone are relatively flat, the second and third audio data can be separated from the first audio data based on the corresponding spectral distribution of the first audio data.

[0061] Thirdly, considering that the time domain waveforms of the first audio data where both the human and the AI ​​voice system speak simultaneously may exhibit delays and repetitions, while the time domain waveforms of the human and AI voice system's individual sounds are relatively smooth, the second and third audio data can be separated from the first audio data based on the corresponding time domain waveforms.

[0062] In addition, in actual application, the audio data processing device can also separate the second audio data and the third audio data from the first audio data according to multiple items of short-time energy, spectrum distribution, and time domain waveform corresponding to the first audio data.

[0063] For example, the audio data processing device can first divide the first audio data according to the short-time energy corresponding to the first audio data to obtain an initial division result; then divide the initial division result according to the frequency spectrum distribution corresponding to the first audio data to obtain a second division result; then divide the second division result according to the time domain waveform corresponding to the first audio data, thereby obtaining second audio data and third audio data.

[0064] In addition, in some other implementations, such as Figure 4 As shown, S102 may further include the following S102b:

[0065] S102b: The audio data processing device inputs the first audio data into the third model to obtain second audio data and third audio data output by the third model.

[0066] Among them, the third model is a Voice Activity Detection (VAD) model used to separate the parts in which humans and artificial intelligence voice systems speak simultaneously and the parts in which humans speak alone from audio data including recordings of conversations between humans and artificial intelligence voice systems.

[0067] Among them, since the VAD model can extract energy features (specifically including short-time energy features), spectral features (specifically including spectral distribution features) and time domain features (specifically including time domain waveform features) from audio data, the second audio data and the third audio data can be divided from the first audio data based on the energy features, spectral features and time domain features.

[0068] In actual application, the third model can be built into a functional module of the audio data processing device. In this case, the VAD model can be trained using training samples in the audio data processing device to obtain the above-mentioned third model. In addition, the function of the third model can also be implemented by a software / hardware module independent of the audio data processing device. In this case, the audio data processing device can send the first audio data to the third model through an application programming interface (API), so that the third model outputs the second audio data and the third audio data.

[0069] S103: The audio data processing device inputs the second audio data into the first model to obtain fourth audio data output by the first model.

[0070] Among them, the first model is a deep neural network (DNNs) model used to remove the part of the artificial intelligence voice system from the audio data in which humans and the artificial intelligence voice system speak simultaneously.

[0071] For example, Figure 5 The figure shows a schematic diagram of the structure of a first model provided by an embodiment of the present application. The first model 20 may include: an input layer 201 for receiving input data, N hidden layers 202 for performing nonlinear transformation and feature extraction on the input data, and an output layer 203 for outputting the final result. In the process of training the first model of the DNNs architecture through training samples, the DNNs can use the backpropagation algorithm to adjust the connection weights between neurons. Specifically, the DNNs can calculate the output based on the input data, and compare it with the actual target value to calculate the error, and then the error is propagated from the output layer 203 to the input layer 201 through backpropagation. The weight is then adjusted according to the error to reduce the error.

[0072] In some implementations, the first model may be a deep neural network model obtained by performing supervised learning using the first training sample set.

[0073] Among them, the first training sample set includes: multiple first sample audio data including recordings of human speech, and multiple second sample audio data obtained by adding the sound of the artificial intelligence voice system to the multiple first sample audio data.

[0074] For example, assuming that multiple first sample audio data include first sample audio data a, second sample audio data b (i.e., one of multiple second sample audio data) can be obtained by adding the sound of an artificial intelligence voice system to the first sample audio data a.

[0075] S104: The audio data processing device inputs the fourth audio data into the second model to obtain fifth audio data output by the second model.

[0076] Among them, the second model is a long-short term memory network (LSTM) model used to remove the part of the artificial intelligence voice system from the audio data in which humans and the artificial intelligence voice system speak simultaneously.

[0077] For example, Figure 6 FIG. 3 is a schematic diagram of a second model structure provided by an embodiment of the present application, wherein the second model 30 may include: an input gate 301 , a cell state 302 , a forget gate 303 and an output layer 304 .

[0078] Among them, the input gate 301 is used to receive input data and control new data to enter the cell state 302; the forget gate 303 is used to decide which data to discard from the cell state 302; the output gate 304 is used to control the output of the cell state; the data in the cell state 302 can be stored for a long time to achieve modeling of long-term dependencies.

[0079] When the second model of the LSTM architecture is running, at each time step, the LSTM receives the current input and the hidden state and cell state from the previous time step. Input gate 301 determines what new information needs to be added to cell state 302 based on the current input and the previous hidden state. Forget gate 303 determines what information needs to be forgotten from cell state 302 based on the previous hidden state and the current input. Cell state 302 is updated through the interaction of input gate 301 and forget gate 303. Output gate 304 determines the output result for the current time step based on the updated cell state 302 and the previous hidden state.

[0080] During LSTM training using the backpropagation algorithm, the parameters of the gating mechanism and cell state are adjusted to minimize prediction error. During training, LSTM learns how to update the cell state and generate output based on the input sequence, thereby being able to capture long-term dependencies in the sequence. Therefore, compared to the first model of the DNNs architecture, which focuses on extracting high-dimensional features from the audio data, the second model of the LSTM architecture can capture temporal changes and dependencies in the audio data, thereby better capturing the dynamic characteristics of the audio. This allows the portion of the second audio data produced by the artificial intelligence speech system to be further removed from the output of the first model (i.e., the fourth audio data), so that the fifth audio data includes more accurate human voices.

[0081] In some implementations, the training sample set used during the training of the second model can be the same as the first training sample set used to train the first model. This effectively allows the DNN model to be trained using the audio data samples in the first training sample set to obtain the first model, while also using the audio data samples in the first training sample set to train the LSTM model to obtain the second model. This reduces the workload associated with collecting sample data.

[0082] In other implementations, the second model is a long short-term memory network model obtained by supervised learning using the second training sample set.

[0083] The first training sample set includes: a plurality of third sample audio data comprising recordings of human speech, and a plurality of fourth sample audio data. The plurality of fourth sample audio data is the audio data output by the first model after the plurality of fifth sample audio data is input into the first model; the plurality of fifth sample audio data is the audio data obtained by adding the sound of the artificial intelligence speech system to the plurality of third sample audio data.

[0084] For example, assuming that multiple third sample audio data include third sample audio data c (including a recording of human speech), fifth sample audio data d can be obtained by adding the sound of an artificial intelligence voice system to the third sample audio data c, and then the fifth sample audio data d is input into the first model of the above-mentioned DNNs architecture, and fourth sample audio data e (i.e., one of the multiple fourth sample audio data) is obtained.

[0085] In the above implementation method, considering that the second model in the embodiment of the present application is used to eliminate the part of the output result of the first model (i.e., the fourth audio data) that is produced by the artificial intelligence voice system, the data output by the first model (i.e., multiple fourth sample audio data) can be used to train the LSTM architecture to obtain the second model.

[0086] S105: The audio data processing device splices the fifth audio data and the third audio data to obtain sixth audio data.

[0087] It can be understood that the sixth audio data includes the human voice portion extracted from the first audio data.

[0088] Additionally, in some implementations, such as Figure 7 As shown, the method may further include:

[0089] S106: The audio data processing device inputs the sixth audio data into the fourth model to obtain the seventh audio data output by the fourth model.

[0090] The fourth model is a Deep Complex Convolutional Recurrent Network (DCCRN) model for eliminating noise in audio data.

[0091] Since the fourth model adopts the DCCRN architecture, the fourth model can combine the advantages of complex convolutional neural networks and recursive neural networks, and can more effectively process noise in speech signals. Specifically, after the audio data processing device inputs the sixth audio data into the fourth model, the fourth model can convert the time domain signal in the sixth audio data into a frequency domain signal, and extract the complex features of the speech signal (for example, the complex features include amplitude features and phase features). Then, the fourth model can adjust the complex features according to the time dependency in the speech signal, and generate an enhanced frequency domain signal based on the adjusted complex features, and then convert the frequency domain signal into a time domain signal to obtain the seventh audio data.

[0092] In some implementations, when the method provided in the embodiment of the present application is applied to a server, such as Figure 2 As shown, the method may further include:

[0093] S107: The audio data processing apparatus sends a feedback message to the terminal device.

[0094] The feedback message may include the sixth audio data or the seventh audio data.

[0095] For example, when the method provided in the embodiment of the present application is applied to the scenario of telephone customer service, the first audio data sent by the terminal device to the audio data processing device in S101a can be a recording of the conversation between the AI ​​voice system of the telephone customer service and the user (for example, the conversation between the telephone customer service and the user included in the first audio data is: "Hello, are you satisfied with the ** service at ** time? You can choose a score from 1 to 10 to evaluate" "Satisfied, ten points"), wherein, when the AI ​​voice system said "You can choose a score from 1 to 10 to evaluate", the user had already said the sentence "Satisfied, ten points", that is, the two sounds are mixed together, that is, the first audio data includes audio data spoken by humans and the artificial intelligence voice system at the same time. Furthermore, after the audio data processing device obtains the human-sounding part of the first audio data (which can be the sixth audio data or the seventh audio data) through all or part of the content in the above S102-S106, it carries the sixth audio data or the seventh audio data in the feedback message and sends it to the terminal device. In this way, the terminal device can extract the text content therein based on the sixth audio data or the seventh audio data, thereby generating a corresponding customer service record. For example, the customer service record may include two pieces of information: 1) the content and time of the service (ie, ** service at ** time) and 2) the second item "satisfied, very satisfied" said by the user.

[0096] For another example, if the method provided in the embodiment of the present application is applied to the scenario of a smart home, the first audio data sent by the terminal device to the audio data processing device in S101a can be a recording of the conversation between the AI ​​voice system of the terminal device and the user (for example, the conversation between the terminal device and the user included in the first audio data is: "Hi, smart air conditioner", "I'm here", "Turn on the cooling function"), wherein the "I'm here" part spoken by the AI ​​voice system is mixed with the "turn on the cooling function" part spoken by the user, that is, the first audio data includes audio data spoken by humans and the artificial intelligence voice system at the same time. Furthermore, after the audio data processing device obtains the human-spoken part of the first audio data (which can be the sixth audio data or the seventh audio data) through all or part of the content in the above S102-S106, it carries the sixth audio data or the seventh audio data in the feedback message and sends it to the terminal device. In this way, the terminal device can extract the text content therein based on the sixth audio data or the seventh audio data, thereby triggering the function of turning on the air conditioning cooling.

[0097] Based on the same inventive concept, as an implementation of the above method, the embodiment of the present application also provides an audio data processing device, which corresponds to the above method embodiment. For ease of reading, this embodiment will no longer describe the details of the above method embodiment one by one, but it should be clear that the audio data processing device in this embodiment can correspond to all or part of the steps performed by the audio data processing device in the above method embodiment. Figure 8 As shown, the audio data processing device 40 includes:

[0098] An acquisition unit 401 is configured to acquire first audio data, wherein the first audio data includes a recording of a conversation between a human and an artificial intelligence speech system;

[0099] Processing unit 402 is configured to separate second audio data and third audio data from the first audio data; the second audio data includes a portion of the first audio data in which both the human and the artificial intelligence speech system speak simultaneously, and the third audio data includes a portion of the first audio data in which only the human speaks;

[0100] The processing unit 402 is further configured to input the second audio data into a first model to obtain fourth audio data output by the first model; the first model is a deep neural network model configured to remove the portion of audio produced by the artificial intelligence speech system from audio data produced simultaneously by a human and the artificial intelligence speech system;

[0101] The processing unit 402 is further configured to input the fourth audio data into a second model to obtain fifth audio data output by the second model; the second model is a long short-term memory network model configured to remove the portion of audio produced by the artificial intelligence speech system from audio data produced simultaneously by a human and the artificial intelligence speech system;

[0102] The processing unit 402 is further configured to concatenate the fifth audio data and the third audio data to obtain sixth audio data.

[0103] In one implementation, the processing unit 402 is configured to separate the second audio data and the third audio data from the first audio data, including:

[0104] The processing unit 402 is configured to separate the first audio data into second audio data and third audio data according to one or more of short-time energy, spectrum distribution, and time-domain waveform corresponding to the first audio data.

[0105] In one implementation, the processing unit 402 is configured to separate the second audio data and the third audio data from the first audio data, including:

[0106] A processing unit 402 is configured to input the first audio data into a third model to obtain second audio data and third audio data output by the third model;

[0107] Among them, the third model is: a voice activity detection model used to divide the parts in which humans and artificial intelligence voice systems speak simultaneously and the parts in which humans speak alone from audio data including recordings of conversations between humans and artificial intelligence voice systems.

[0108] In one implementation, the processing unit 402 is further used to input the sixth audio data into a fourth model to obtain seventh audio data output by the fourth model; the fourth model is: a deep complex convolutional recurrent network model for eliminating noise in audio data.

[0109] In one implementation, the method is applied to a server. An acquisition unit 401 is configured to acquire first audio data, specifically comprising:

[0110] The acquiring unit 401 is configured to receive first audio data from a terminal device.

[0111] In one implementation, the first model is a deep neural network model obtained by supervised learning using a first training sample set; wherein the first training sample set includes: a plurality of first sample audio data including recordings of human speech, and a plurality of second sample audio data obtained by adding the sound of an artificial intelligence voice system to the plurality of first sample audio data.

[0112] In one implementation, the second model is a long short-term memory network model obtained by supervised learning using a second training sample set; wherein the first training sample set includes: multiple third sample audio data including recordings of human speech, and multiple fourth sample audio data; wherein the multiple fourth sample audio data are audio data output by the first model after multiple fifth sample audio data are input into the first model; the multiple fifth sample audio data are audio data obtained by adding the sound of an artificial intelligence voice system to the multiple third sample audio data.

[0113] The audio data processing device 40 provided in an embodiment of the present application can execute part or all of the steps executed by the server in the method provided in any of the above embodiments. Its implementation principle and technical effect are similar and will not be repeated here.

[0114] Based on the same inventive concept, the embodiment of the present application also provides an audio data processing device 50. Figure 9 A structural diagram of the audio data processing device provided in an embodiment of the present application is shown in FIG. Figure 9As shown, the audio data processing device provided in this embodiment includes: a memory 501 and a processor 502, wherein the memory 501 is used to store a computer program, and the processor 502 is used to execute part or all of the steps performed by the audio data processing device in the method provided in the above embodiment when executing the computer program.

[0115] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the computing device implements part or all of the steps performed by the audio data processing device in the audio data processing method provided in the above embodiment.

[0116] Based on the same inventive concept, an embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computing device implements part or all of the steps performed by the audio data processing device in the audio data processing method provided in the above embodiment, or part or all of the steps performed by the server.

[0117] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0118] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0119] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0120] Computer-readable media includes both permanent and non-permanent, removable and non-removable storage media. Storage media can implement any method or technology for storing information, which can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some or all of the technical features therein, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for processing audio data, characterized in that: The method comprises: Acquire first audio data; the first audio data includes a recording of a conversation between a human and an artificial intelligence voice system; Separating the first audio data into second audio data and third audio data; the second audio data includes the portion of the first audio data in which the human and the artificial intelligence speech system speak simultaneously, and the third audio data includes the portion of the first audio data in which the human speaks alone; Inputting the second audio data into a first model to obtain fourth audio data output by the first model; the first model is a deep neural network model for removing the portion of the audio produced by the artificial intelligence voice system from audio data produced simultaneously by a human and an artificial intelligence voice system; Inputting the fourth audio data into a second model to obtain fifth audio data output by the second model; the second model is a long short-term memory network model for removing the portion of the audio produced by the artificial intelligence speech system from the audio data produced simultaneously by a human and an artificial intelligence speech system; The fifth audio data and the third audio data are concatenated to obtain sixth audio data.

2. The method according to claim 1, characterized in that The dividing the second audio data and the third audio data from the first audio data includes: Second audio data and third audio data are divided from the first audio data according to one or more of short-time energy, spectrum distribution, and time-domain waveform corresponding to the first audio data.

3. The method according to claim 1, characterized in that The dividing the second audio data and the third audio data from the first audio data includes: Inputting the first audio data into a third model to obtain second audio data and third audio data output by the third model; Among them, the third model is: a voice activity detection model used to divide the parts in which humans and artificial intelligence voice systems speak simultaneously and the parts in which humans speak alone from audio data including recordings of conversations between humans and artificial intelligence voice systems.

4. The method according to claim 1, wherein The method further comprises: The sixth audio data is input into a fourth model to obtain seventh audio data output by the fourth model; the fourth model is: a deep complex convolutional recurrent network model for eliminating noise in audio data.

5. The method according to any one of claims 1 to 4, characterized in that The method is applied to a server; the obtaining of the first audio data specifically includes: First audio data is received from a terminal device.

6. The method according to any one of claims 1 to 4, characterized in that The first model is a deep neural network model obtained by supervised learning using a first training sample set; Among them, the first training sample set includes: multiple first sample audio data including recordings of human speech, and multiple second sample audio data obtained by adding the sound of the artificial intelligence voice system to the multiple first sample audio data.

7. The method according to any one of claims 1 to 4, characterized in that The second model is a long short-term memory network model obtained by supervised learning using the second training sample set; Among them, the first training sample set includes: multiple third sample audio data including recordings of human speech, and multiple fourth sample audio data; wherein, the multiple fourth sample audio data are audio data output by the first model after multiple fifth sample audio data are input into the first model; the multiple fifth sample audio data are audio data obtained by adding the sound of the artificial intelligence voice system to the multiple third sample audio data.

8. An audio data processing device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program and the processor is used to enable the audio data processing device to implement the method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the computer implements the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to implement the method according to any one of claims 1 to 7.

Images