Method executed by electronic equipment, electronic equipment and storage medium

Abstract

The invention provides a method executed by electronic equipment, the electronic equipment and a storage medium, and particularly, the method executed by the electronic equipment comprises the steps that registration audio information of a user is determined based on bone conduction signals; extracting a second audio signal of the user from the first audio signal based on registration audio information of the user; and processing the extracted second audio signal and / or the first audio signal which does not contain the second audio signal. The method performed by the electronic device may be performed using an artificial intelligence model.

Description

Technical Field

[0001] The present disclosure relates to the field of audio processing, and in particular to a method executed by an electronic device, the electronic device, and a storage medium. Background Art

[0002] Speech extraction, also known as specific person speech separation or speech filtering, is a technology that extracts a specific target speech from a mixed speech signal. This technology can be used in a variety of speech-related applications, such as voice calls, online conferences, voice commands, voice searches, etc., to eliminate background noise, especially human voice noise. In order to improve the extraction quality of a specific target speaker's speech, existing speech extraction technology obtains the registration information of the target speech in advance to extract the target speech. Existing speech extraction technology makes it difficult to achieve real-time speech extraction of any target speaker. Summary of the invention

[0003] The present disclosure provides a method executed by an electronic device, an electronic device, and a storage medium to solve part or all of the above problems.

[0004] According to a first aspect of an embodiment of the present disclosure, a method performed by an electronic device is provided, the method comprising: determining a user's registered audio information based on a bone conduction signal; extracting a second audio signal of the user from a first audio signal based on the user's registered audio information; and processing the extracted second audio signal and / or the first audio signal not containing the second audio signal.

[0005] Optionally, determining the user's registered audio information based on the bone conduction signal includes: evaluating the quality of the extracted second audio signal based on the content of the extracted second audio signal and the bone conduction BC signal; and determining the user's registered audio information based on the quality of the extracted second audio signal.

[0006] Optionally, evaluating the quality of the extracted second audio signal includes: classifying the extracted second audio signal and the BC signal based on speech features; determining a matching probability between a category of the extracted second audio signal and a category of the BC signal; and evaluating the quality of the extracted second audio signal based on the matching probability.

[0007] Optionally, based on speech features, the extracted second audio signal and BC signal are classified, including: using a pre-trained classifier to classify the extracted second audio signal and BC signal respectively, and wherein, determining the matching probability between the category of the extracted second audio signal and the category of the BC signal includes: searching a pre-trained matching probability table for the category of the extracted second audio signal and the category of the BC signal to determine the matching probability.

[0008] Optionally, based on the quality of the extracted second audio signal, the user's registered audio information is determined, including: based on the quality of the extracted second audio signal, determining that the user's registered audio information needs to be updated; determining characteristic changes in BC registration information; based on the characteristic changes in BC registration information, predicting the registration audio information of the air-conducted AC signal as the user's registered audio information.

[0009] Optionally, based on the quality of the extracted second audio signal, determining that the registered audio information of the user needs to be updated includes: if the quality of the extracted second audio signal does not meet a predetermined condition, determining that the registered audio information of the user needs to be updated; and if the quality of the extracted second audio signal meets a predetermined condition, determining that the registered audio information of the user does not need to be updated.

[0010] Optionally, determining characteristic changes of BC registration information includes: based on historical BC registration information and current BC registration information, using a first artificial intelligence (AI) model to determine characteristic changes of the BC registration information, and wherein, based on the characteristic changes of the BC registration information, predicting the registration audio information of the air-conducted AC signal as the registration audio information of the user, including: based on the characteristic changes of the BC registration information, using a second AI model to predict characteristic changes of the registration audio information of the AC signal, and based on the characteristic changes of the registration audio information of the AC signal and historical registration audio information of the AC signal, using a third AI model to obtain the registration audio information of the AC signal as the registration audio information of the user.

[0011] Optionally, the historical registration audio information of the AC signal includes registration audio information corresponding to the extracted second audio signal evaluated as having the highest quality, and the historical BC registration information corresponds to the historical registration audio information of the AC signal.

[0012] Optionally, extracting the second audio signal of the user from the first audio signal based on the user's registered audio information includes: acquiring a feature of the first audio signal; acquiring a mask about the user based on the user's registered audio information and the feature of the first audio signal; and extracting the second audio signal based on the mask and the feature of the first audio signal.

[0013] Optionally, based on the user's registered audio information and the characteristics of the first audio signal, a mask about the user is obtained, including: based on the user's registered audio information, the characteristics of the first audio signal and the characteristics of the bone conduction signal, using a fourth AI model to obtain a mask about the user.

[0014] Optionally, obtaining the characteristics of the first audio signal includes: extracting characteristics of the first audio signal to obtain first frequency domain characteristics; dividing the first frequency domain characteristics into frequency bands to obtain sub-frequency domain characteristics of each of a plurality of sub-bands; and performing feature encoding on the sub-frequency domain characteristics of the plurality of sub-bands respectively to obtain characteristics of the plurality of sub-bands of the first audio signal as the characteristics of the first audio signal.

[0015] Optionally, based on the mask and the features of the first audio signal, extracting the second audio signal includes: obtaining features of each subband of the second audio signal based on the mask in the mask corresponding to each of the multiple subbands of the first audio signal and the features of each subband of the first audio signal; performing feature decoding on the features of the multiple subbands of the second audio signal respectively to obtain second frequency domain features of the multiple subbands; performing frequency band merging on the second frequency domain features of the multiple subbands; and obtaining the second audio signal based on the merged second frequency domain features.

[0016] Optionally, processing the extracted second audio signal and / or the first audio signal excluding the second audio signal includes: amplifying the extracted second audio signal and / or the first audio signal excluding the second audio signal, or mixing the second audio signal with a third audio signal.

[0017] Optionally, amplifying the extracted second audio signal and the first audio signal excluding the second audio signal includes: amplifying the extracted second audio signal and the first audio signal excluding the second audio signal at different proportions.

[0018] According to a second aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a registration audio determination module, configured to determine the registration audio information of a user based on a bone conduction signal; an audio extraction module, configured to extract a second audio signal of the user from a first audio signal based on the registration audio information of the user; and an audio processing module, configured to process the extracted second audio signal and / or the first audio signal not containing the second audio signal.

[0019] Optionally, the registration audio determination module is configured to: evaluate the quality of the extracted second audio signal based on the content of the extracted second audio signal and the bone conduction BC signal; and determine the user's registration audio information based on the quality of the extracted second audio signal.

[0020] Optionally, the registration audio determination module is configured to: classify the extracted second audio signal and the BC signal based on speech features; determine the matching probability between the category of the extracted second audio signal and the category of the BC signal; and evaluate the quality of the extracted second audio signal based on the matching probability.

[0021] Optionally, the registered audio determination module is configured to: classify the extracted second audio signal and BC signal respectively using a pre-trained classifier, and: search a pre-trained matching probability table for the category of the extracted second audio signal and the category of the BC signal to determine the matching probability.

[0022] Optionally, the registration audio determination module is configured to: determine the need to update the user's registration audio information based on the quality of the extracted second audio signal; determine the characteristic changes of the BC registration information; and predict the registration audio information of the air-conducted AC signal as the user's registration audio information based on the changing characteristics of the BC registration information.

[0023] Optionally, the registered audio determination module is configured to: determine that the registered audio information of the user needs to be updated when the quality of the extracted second audio signal does not meet a predetermined condition; and determine that the registered audio information of the user does not need to be updated when the quality of the extracted second audio signal meets a predetermined condition.

[0024] Optionally, the registration audio determination module is configured to: based on historical BC registration information and current BC registration information, use a first artificial intelligence (AI) model to determine characteristic changes in the BC registration information; based on the characteristic changes in the BC registration information, use a second AI model to predict characteristic changes in the registration audio information of the AC signal; and based on the characteristic changes in the registration audio information of the AC signal and the historical registration audio information of the AC signal, use a third AI model to obtain the registration audio information of the AC signal as the registration audio information of the user.

[0025] Optionally, the historical registration audio information of the AC signal includes registration audio information corresponding to the extracted second audio signal evaluated as having the highest quality, and the historical BC registration information corresponds to the historical registration audio information of the AC signal.

[0026] Optionally, the audio extraction module is configured to: obtain features of the first audio signal; obtain a mask about the user based on the user's registered audio information and the features of the first audio signal; and extract the second audio signal based on the mask and the features of the first audio signal.

[0027] Optionally, the audio extraction module is configured to obtain a mask about the user using a fourth AI model based on the user's registered audio information, features of the first audio signal, and features of the bone conduction signal.

[0028] Optionally, the audio extraction module is configured to: perform feature extraction on the first audio signal to obtain a first frequency domain feature; divide the first frequency domain feature into frequency bands to obtain sub-frequency domain features of each of the multiple sub-bands; and perform feature encoding on the sub-frequency domain features of the multiple sub-bands respectively to obtain features of the multiple sub-bands of the first audio signal as features of the first audio signal.

[0029] Optionally, the audio extraction module is configured to: obtain the characteristics of each subband of the second audio signal based on the mask in the mask corresponding to each of the multiple subbands of the first audio signal and the characteristics of each subband of the first audio signal; perform feature decoding on the characteristics of the multiple subbands of the second audio signal respectively to obtain second frequency domain characteristics of the multiple subbands; perform frequency band merging on the second frequency domain characteristics of the multiple subbands; and obtain the second audio signal based on the merged second frequency domain characteristics.

[0030] Optionally, the audio processing module is configured to: amplify the extracted second audio signal and / or the first audio signal excluding the second audio signal, or mix the second audio signal with a third audio signal.

[0031] Optionally, the audio processing module is configured to: amplify the extracted second audio signal and the first audio signal excluding the second audio signal at different proportions.

[0032] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, the device comprising: a processor; a memory storing a computer program, and when the computer program is executed by the processor, the method performed by the electronic device as described above is implemented.

[0033] According to a fourth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium, on which a computer program is stored, and when the program is executed, the method performed by the electronic device as described above is implemented.

[0034] The technical solution provided according to the embodiments of the present disclosure brings at least the following beneficial effects: since the user's registered audio information is determined based on the bone conduction signal, there is no need to register the target object's voice in advance, which greatly facilitates the voice extraction of the target object.

[0035] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0037] Figure 1 Schematic diagram showing the structure of the speech filter network.

[0038] Figure 2 A schematic diagram of the main flow of a speech extraction network according to an exemplary embodiment of the present disclosure is shown.

[0039] Figure 3 A flowchart of a method executed by an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0040] Figure 4 The difference between a bone conduction signal and an air conduction signal according to an exemplary embodiment of the present disclosure is shown.

[0041] Figure 5 The figure shows an internal block diagram of an extracted speech quality assessment module according to an exemplary embodiment of the present disclosure.

[0042] Figure 6 A flowchart of extracting speech quality assessment according to an exemplary embodiment of the present disclosure is shown.

[0043] Figure 7 A schematic diagram showing offline training according to an exemplary embodiment of the present disclosure.

[0044] Figure 8 A schematic diagram of matching bone conduction signals and extracted audio signals according to an exemplary embodiment of the present disclosure is shown.

[0045] Fig. 9 The sound and spectrum of air conduction signals and bone conduction signals are shown.

[0046] Fig.10 An internal block diagram of a real-time imperceptible registration module according to an exemplary embodiment of the present disclosure is shown.

[0047] Fig.11 A schematic diagram showing the principle of generating and updating the registered audio information of a user according to an exemplary embodiment of the present disclosure.

[0048] Fig.12 A workflow diagram for updating a user's registered audio information according to an exemplary embodiment of the present disclosure is shown.

[0049] Fig.13 A diagram of an encoder module according to an example embodiment of the present disclosure is shown.

[0050] Fig.14 A network flow diagram of an encoder module according to an example embodiment of the present disclosure is shown.

[0051] Fig.15 A decoder module diagram according to an example embodiment of the present disclosure is shown.

[0052] Fig.16A decoder module network flow diagram according to an example embodiment of the present disclosure is shown.

[0053] Fig.17 A schematic block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0054] Fig.18 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0055] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0056] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0057] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0058] Extracting target speech from interfering sounds is a very important technology. It can be used in almost all speech-related applications, such as voice calls, automatic speech recognition (ASR), voice search, voice conferencing, etc. The present disclosure proposes that a voice filter can be used for speech extraction. The voice filter requires the user to perform voice registration in a quiet environment in advance to obtain the user's voice features, and then extract the user's voice from the mixed audio. Figure 1 A schematic diagram showing the network structure of the speech filter. Figure 1As shown in FIG. 1 , the speech filter includes four modules: (1) Encoding module: performs short-time Fourier transform (STFT, Short-time Fourier Transform) on the input audio signal (the noisy audio in the figure), and encodes a high-dimensional feature vector to represent the information of different dimensions of the speech (the input sound spectrum in the figure); (2) Speaker registration module: first encodes the registered speech of the target speaker (the reference audio in the figure) (the speaker encoder LSTM (Long Short-Term Memory) in the figure), and outputs a vector representing the speaker, called d-vector; (3) Speech extraction module: inputs the features of the mixed speech (the input sound spectrum in the figure) and the vector representing the speaker d-vector, where the d-vector is concatenated with the features of the mixed speech at the frame level after the Convolutional Neural Network (CNN) module, and then predicted by soft mask. Finally, the module outputs the target speaker mask Mask, which represents the percentage of information occupied by the target speaker in the mixed audio. (4) Decoding module: The target speaker mask Mask is used to operate the input sound spectrum and then the masked sound spectrum is output. After the masked sound spectrum is subjected to an inverse short-time Fourier transform (inverse STFT as shown in the figure), the audio signal of the target speaker is output (the enhanced audio as shown in the figure).

[0059] At present, the speech extraction technology based on the target speaker registration information has the following two main problems: 1. It requires users to register their voice in advance, and the registered voice needs to be collected in a relatively quiet environment. 2. It will cause incorrect extraction results when the user's voice features change. When the user's voice changes, the voice features are significantly different from the previously registered voice features, which may lead to incorrect extraction results.

[0060] The present disclosure provides a method and an electronic device performed by an electronic device. On the one hand, the quality of the extracted audio is evaluated by content matching of the extracted audio and the bone conduction signal, and the extracted audio and the bone conduction signal are classified by content at the frame level, and the quality score can be calculated by referring to a trained matching probability table (Maching Probablility Graphic, MPG). On the other hand, when the quality of the extracted audio is low, the change of the audio information is predicted by referring to the change of the bone conduction information to update the user's registered audio information in real time.

[0061] In the present disclosure, bone conduction (BC) signals are used, which can also be called bone conduction signals. In recent years, stereo (True Wireless Stereo, TWS) headphones have been equipped with voice pickup units (VPUs) that can obtain bone conduction (BC) signals. The advantage of bone conduction signals is that since the interference and noise of other speakers cannot vibrate the user's skull, only the user's own bone conduction signal is collected by the VPU, and it is aligned with the user's voice collected by the external microphone at the frame level.

[0062] According to a method and electronic device performed by an electronic device disclosed in the present invention, the quality of extracted audio is evaluated based on bone conduction signals, and when the evaluated extracted audio quality is low, the user's registered audio information is updated in real time through the bone conduction signal, thereby realizing adaptive real-time registration of the user's voice. On the one hand, there is no need for advance registration of the user's voice, which improves the user experience. On the other hand, when the user's voice characteristics change, the user's registered audio information can also be updated in real time, which improves the quality of voice extraction and the naturalness of the voice.

[0063] The following will refer to the attached Figures 2 to 18 A method performed by an electronic device and an electronic device according to the present disclosure are described in detail.

[0064] In the embodiments of the present disclosure, the target object may be, for example, a target speaker from whom audio is to be extracted or a user from whom audio is to be extracted (hereinafter referred to as a user).

[0065] Figure 2 The main flow diagram of the speech extraction network according to the exemplary embodiment of the present disclosure is shown. The main flow diagram includes the following 5 parts:

[0066] Extracted voice quality assessment module: evaluates the quality of the extracted audio with the help of bone conduction signals. The input may include: the previous frame bone conduction signal S BC (n-1) and the output speech S extracted from the previous frame out (n-1) (hereinafter also referred to as extracted speech (signal), extracted audio (signal)), the output may include: the previous frame S out The mass of (n-1), e.g., mass fraction.

[0067] Real-time non-perceptual registration module: real-time registration / update registration audio information. Input may include: current frame bone conduction signal S BC (n), previous frame S out The quality of (n-1) and the previous frame S out (n-1). Figure 2 The input of the real-time non-perceptual registration module in the previous frame Sout (n-1), but this module can only use the extracted audio S in the initial state used by the user out The information is directly used as the user's registered audio information, and the subsequent extraction of audio quality (for example, the previous frame S out The output of the real-time imperceptible registration module may include: the user's registered audio information C reg (also called, currently registered voice information C reg ).

[0068] Speech extraction module: Outputs the mask of the target object (e.g., target speaker, user) (i.e., the mask about the user) for extracting the target object speech from the mixed speech, where the mask of the target object indicates the information percentage of the target object in the mixed audio. Inputs may include: user's registered audio information C reg , features of the current frame mixed speech and features of the current frame bone conduction signal, and the output may include: a mask of the target object.

[0069] Encoder module: Perform short-time Fourier transform on the input audio signal and encode a high-dimensional feature vector to represent information of different dimensions of speech. The input may include: the current frame mixed speech and the current frame bone conduction signal S BC (n), and the output may include: features of the current frame mixed speech and features of the current frame bone conduction signal.

[0070] Decoder module: decodes the mixed speech using the mask of the target object and outputs the speech of the target object (i.e., the extracted output speech), where the operation of the module with "*" can be considered as part of the operation of the decoder module. The input may include: the features of the mixed speech of the current frame and the mask of the target object, and the output may include: the extracted output speech S out (n).

[0071] The above main network process can be applied not only to speech extraction, but also to speech enhancement and speech separation without changing its basic architecture.

[0072] It should be understood that the main flowchart diagram here is only an example, and some modules may be omitted or other modules may be added, and the present disclosure is not limited to this.

[0073] Figure 3 A flowchart of a method executed by an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0074] Reference Figure 3 , in step S310, the user's registered audio information is determined based on the bone conduction signal.

[0075] According to an embodiment of the present disclosure, the quality of the extracted second audio signal may be evaluated based on the content of the extracted second audio signal and the bone conduction BC signal; and the user's registered audio information may be determined based on the quality of the extracted second audio signal.

[0076] For example, refer to Figure 2 , if the audio signal S is extracted out If the quality of (n-1) (i.e., the extracted second audio signal) is high, there is no need to update the user's registered audio information C reg But if S out (n-1) If the quality is low, it means that the user's user voice may change, and the user's registered audio information cannot correctly guide the voice extraction module to work. Therefore, the quality of the extracted audio signal can be evaluated first.

[0077] According to an embodiment of the present disclosure, the extracted second audio signal and the BC signal may be classified based on speech features; a matching probability between the category of the extracted second audio signal and the category of the BC signal may be determined; and the quality of the extracted second audio signal may be evaluated based on the matching probability.

[0078] In an embodiment of the present disclosure, a predetermined number of extracted audio signals (i.e., extracted second audio signals) including the most recent extracted audio signal and corresponding bone conduction signals may be collected, each extracted audio signal and each bone conduction signal may be classified respectively to obtain a predetermined number of air conduction-bone conduction category pairs, matching may be performed for each air conduction-bone conduction category pair to obtain a predetermined number of matching probabilities, and the quality of the extracted audio signal may be evaluated based on the predetermined number of matching probabilities.

[0079] In an embodiment of the present disclosure, the bone conduction (BC) signal of the target object can be used to evaluate the quality of the extracted audio. The bone conduction signal can be picked up by the VPU on the TWS. The disadvantage of the bone conduction signal is that the user's voice is transmitted in the skull. Unlike the air conduction (AC) signal (i.e., the signal in which the sound is transmitted in the air, also referred to as the "air conduction signal" below), the bone conduction signal has a narrowband characteristic (only having frequency components less than 1kHz). Figure 4 The difference between a bone conduction (BC) signal and an air conduction (AC) signal according to an exemplary embodiment of the present disclosure is shown. Compared with a full-band air conduction signal, a bone conduction signal lacks a large amount of frequency information.

[0080] The bone conduction signal has no interference and is aligned with the speech signal of the target object (e.g., target speaker, user). Therefore, it can be used to evaluate the extracted audio signal by signal matching, but the bone conduction signal only has frequency information below 1kHz, making it unable to evaluate the high-frequency quality of the extracted audio signal.

[0081] In an embodiment of the present disclosure, the quality is evaluated by content matching of the extracted audio signal and the bone conduction signal instead of using frequency-related signal matching. The content matching here can be frame-level content matching. Content matching can be achieved by classifying the extracted audio signal and the bone conduction signal into phoneme-like categories based on speech features. Phonemes are the smallest speech units in a language that can distinguish meanings, such as [ai] and [ The phoneme-like category refers to a classification of speech features that is similar to the standard phoneme classification but not one-to-one. We do not need to decode specific phoneme information, so no additional steps are required to obtain specific phoneme information, and the classification results of similar phonemes are sufficient.

[0082] According to an embodiment of the present disclosure, a pre-trained classifier may be used to classify the extracted second audio signal and BC signal respectively, and a pre-trained matching probability table may be searched for the category of the extracted second audio signal and the category of the BC signal to determine the matching probability.

[0083] the following Figures 5 to 8 The operation can correspond to Figure 2 Extract the relevant operations of the speech quality assessment module.

[0084] Figure 5 FIG. 1 shows an internal block diagram of a voice quality assessment module according to an exemplary embodiment of the present disclosure. Figure 5 In the extracted speech quality evaluation module, a predetermined number (e.g., N frames, where N is a positive integer) of extracted audio signals (i.e., the extracted second audio signals) may be collected through a cache, or a predetermined number (e.g., N frames) of bone conduction signals of the target object may be collected through a cache. The predetermined number of extracted audio signals may include the most recent extracted audio signals, for example, Figure 2 S out (n-1), accordingly, the predetermined number of bone conduction signals may include the most recent bone conduction signal, for example, Figure 2 S BC (n-1). After the extracted audio signal and bone conduction signal of each frame are input into the same classifier for classification, a total of a predetermined number (e.g., N) of air conduction-bone conduction category pairs are obtained, and then the matching probability table is searched for the air conduction-bone conduction category pairs of each frame to obtain a total of a predetermined number of matching probabilities, and a quality score (e.g., the mean of the matching probabilities) is calculated based on the matching probability and output.

[0085] It should be understood that the internal block diagram of the extracted speech quality assessment module here is only an example, and some modules may be omitted or other modules may be added, which is not limited in the present disclosure.

[0086] Figure 6A flowchart of extracting speech quality assessment according to an exemplary embodiment of the present disclosure is shown.

[0087] 1. Classify bone conduction signals and extract audio signals:

[0088] The classifier is obtained by training speech features offline. When performing online speech quality assessment, the input bone conduction signal and the extracted audio signal are classified using the offline trained classifier. The categories of the bone conduction signal and the extracted audio signal are obtained based on the classification results and the distance between the class centers of the existing categories. The process can be frame by frame, and the output is the air conduction-bone conduction category pair for each frame. Figure 6 , the input speech content (e.g., "Okay") is divided into three categories according to the extracted audio signal and bone conduction signal, and each category corresponds to a phoneme-like content information. For example, corresponding to the three frames of extracted audio signal and bone conduction signal, three category pairs are output, namely (A1, B1), (A2, B3) and (A5, B1).

[0089] 2. Evaluate the matching probability of the extracted audio signal and bone conduction signal through the probability matching table:

[0090] The probability matching table is obtained through offline training. The classifier classifies the bone conduction signal and the extracted audio signal to obtain a category pair. All category pairs obtained based on the training data constitute a matching probability table. The matching probability table represents the correspondence between the classification of the extracted audio signal and the classification of the bone conduction signal (for example, similarity). For example, the matching probabilities obtained after searching the matching probability table for the above three category pairs are 1, 0.1, and 0.9, respectively. Since the bone conduction signal lacks vowels, similar pronunciations will be classified into one category, so one bone conduction signal category may correspond to multiple extracted audio signal categories.

[0091] 3. Evaluate the extracted audio quality based on the matching probability:

[0092] The average of the matching probabilities (similarity) of all frame-extracted audio signals and bone conduction signals can be used as the quality evaluation result (i.e., quality score) of the extracted audio. As shown in the following formula,

[0093]

[0094] In the above formula, s i represents the matching probability of the extracted audio signal and the bone conduction signal of the i-th frame, and N represents the total number of frames. For example, the quality score obtained based on the above three matching probabilities of 1, 0.1 and 0.9 is 0.67.

[0095] It should be understood that the use of the mean value of the matching probability as the quality assessment result here is only an example, and the sum value of all frames, etc. may also be used, and the present disclosure is not limited thereto.

[0096] Figure 7 A schematic diagram of offline training according to an exemplary embodiment of the present disclosure is shown. Figure 7 In the operation of extracting speech quality assessment, the classifier can be trained offline by clustering speech features offline, and the matching probability table (MSG) is a probability matching table obtained by offline training. The classifier here can be, for example, but not limited to, a neural network that performs a classification task, such as a convolutional neural network + fully connected (CNN + FC).

[0097] Figure 8 FIG. 1 shows a schematic diagram of the matching principle of the bone conduction signal and the extracted audio signal according to an exemplary embodiment of the present disclosure. Figure 8 , Ai represents the extracted audio signal (ie, extracted speech signal) S out category, and Bi represents the bone conduction signal S BC It can be seen that class B1 matches class A1 and A2 with probabilities of 0.8 and 0.85. A1 and A2 may be pronounced similarly, so they both point to the same class B1. The signal in class A3 is assumed to be in class A2, but it has high-frequency interference from other speakers, so the classifier classifies the signal as class A3, and the probability that B1 matches A3 is only 0.2, which means that the signal in B1 does not match the signal in A3.

[0098] The above-mentioned extracted speech quality evaluation method can solve the problem of frequency limitation of bone conduction signals, and evaluate the extracted audio signals through bone conduction signals. Content matching works properly not only in daily communication scenarios, but also in scenarios with highly consistent content (such as chorus and reading aloud). This is because based on frame-level content matching, the probability of exactly the same content is low.

[0099] According to an embodiment of the present disclosure, determining the user's registered audio information based on the quality of the extracted second audio signal may include: determining that the user's registered audio information needs to be updated based on the quality of the extracted second audio signal; determining the characteristic change of the BC registration information; and predicting the registration audio information of the air-conducted AC signal as the user's registered audio information based on the characteristic change of the BC registration information.

[0100] According to an embodiment of the present disclosure, when the quality of the extracted second audio signal does not meet the predetermined condition, it can be determined that the user's registered audio information needs to be updated; and when the quality of the extracted second audio signal meets the predetermined condition, it can be determined that the user's registered audio information does not need to be updated.

[0101] In an embodiment of the present disclosure, the user's registered audio information can be obtained based on the predetermined duration of the target object's (i.e., user's) voice extracted from the first audio signal (e.g., mixed speech). Specifically, the information of the user's predetermined duration of voice extracted from the mixed speech can be used as the user's registered audio information in the initial state. In the initial state used by the user (e.g., Figure 2 In the initial state of the voice extraction network used for the first time or again or the initial state used after changing the user), the information of the user's voice of a predetermined length (for example, 1 frame) can be extracted from the mixed voice as the user's registered audio information, and the registered audio information of the user in the initial state can be used to start extracting the user's voice in the mixed voice. At this time, the quality of the extracted audio is evaluated, and the quality of the extracted audio may not be good. The quality of the subsequent extracted audio can be improved by updating the user's registered audio information. Therefore, on the one hand, in the initial state used by the user, the information of the extracted audio can be directly used as the user's registered audio information; on the other hand, when the quality of the subsequent extracted audio is low, the information of the extracted audio can be continuously updated as the user's registered audio information in the initial state to obtain the updated user's registered audio information. Here, examples of the predetermined duration include but are not limited to the duration of 1 frame. In the embodiments of the present disclosure, the information of the extracted audio can be used as the real-time user's registered audio information without the user registering the voice in advance, eliminating the inconvenience caused by the advance registration to the user and improving the user experience.

[0102] Reference Figure 2 , if the audio signal S is extracted out If the quality of (n-1) meets the predetermined condition, it can be determined that there is no need to update the user's registered audio information C reg The previous user's registered audio information will continue to be used, but if S out (n-1) If the quality does not meet the predetermined condition, it means that the user's voice may change, and the user's registered audio information cannot correctly guide the voice extraction module to work, and it can be determined that the user's registered audio information needs to be updated. The predetermined condition here can be, for example, but not limited to, the quality score is greater than the predetermined score.

[0103] According to an embodiment of the present disclosure, the characteristic change of the BC registration information can be determined by using a first artificial intelligence AI model based on the historical BC registration information and the current BC registration information. The characteristic change of the registered audio information of the AC signal can be predicted by using a second AI model based on the characteristic change of the BC registration information, and the registered audio information of the AC signal can be obtained as the user's registered audio information by using a third AI model based on the characteristic change of the registered audio information of the AC signal and the historical registered audio information of the AC signal.

[0104] Fig. 9 Shows the sound and spectrum of air conduction and bone conduction signals. Fig. 9 When the user's voice changes, the changes in the bone conduction signal (BC signal) and the changes in the air conduction signal from the auditory sense (i.e., air conduction signal, AC signal) are related and regular. Moreover, the bone conduction signal is reliable for the current speaker characteristics because it is only related to the user. Based on learning the change characteristics of the user's bone conduction signal (i.e., the change of characteristics), predicting and generating the change characteristics (i.e., the change of characteristics) of the user's air conduction signal (i.e., audio signal), the user's registered audio information is updated in real time and adaptively.

[0105] the following Figures 10 to 12 The operation can correspond to Figure 2 Related operations of the real-time non-aware registration module.

[0106] Fig.10 An internal block diagram of a real-time imperceptible registration module according to an exemplary embodiment of the present disclosure is shown.

[0107] Reference Fig.10 , the internal input of the real-time imperceptible registration module when performing the update operation may include: historical BC registration information (also referred to as historical BC information), current BC registration information (also referred to as current BC information) and historical registration audio information of AC signals (hereinafter referred to as historical registration audio information), and the output includes: the updated user's registration audio information, wherein the historical BC registration information and the historical registration audio information can be considered as information collected inside the implementation imperceptible registration module. Fig.10 , the real-time imperceptible registration module includes three artificial intelligence models (AI). Among them, the input of the first artificial intelligence model may include: historical BC registration information and current BC registration information, and the output may include: change characteristics of BC registration information (that is, the characteristic changes of BC registration information). The input of the second artificial intelligence model may include: change characteristics of BC registration information, and the output may include: change characteristics of the registered audio information of the AC signal (hereinafter also referred to as change characteristics of the registered audio information, that is, the characteristic changes of the registered audio information of the AC signal). The input of the third artificial intelligence model may include: change characteristics of the registered audio information and historical registered audio information, and the output may include the registered audio information of the AC signal as the (updated) user's registered audio information. The user's registered audio information may be information that characterizes the user's registered audio signal (that is, the registered voice signal). In addition, the change characteristics of the registered audio information and the historical registered audio information can be spliced ​​and input into the third artificial intelligence model.

[0108] The BC registration information here (including historical BC registration information and current BC registration information) can be considered as the bone conduction BC signal (corresponding to the historical BC signal and the current BC signal (for example, S BC (n))), and the BC registration information may be information characterizing the BC signal. The historical registered audio information here may be considered to be information obtained after feature processing (e.g., feature extraction) is performed on the basis of the historical registered audio signal, and the historical registered audio information may be information characterizing the historical registered audio signal. In addition, the historical BC registration information may correspond to the historical registration audio information, and specifically, the time period of the historical BC signal selection segment may correspond to the historical registration audio signal. Therefore, one or more feature processing modules may be added before the first artificial intelligence model to convert the bone conduction signal into the corresponding BC registration information and / or convert the audio signal into the corresponding audio information, or one or more feature processing modules may be part of the above-mentioned artificial intelligence model, so that the input of the artificial intelligence model may be the corresponding bone conduction signal (e.g., S BC (n)) or audio signal.

[0109] It should be understood that the internal block diagram of the real-time non-perceptual registration module here is only an example, and some modules may be omitted or other modules may be added, which is not limited in the present disclosure.

[0110] The AI ​​model can be obtained by training. Here, "obtained by training" refers to training a basic AI model with multiple training data through a training algorithm to obtain a predefined operating rule or AI model, and the operating rule or AI model is configured to perform the required features (or purposes). The first artificial intelligence model, the second artificial intelligence model, and the third artificial intelligence model here can be, for example, a neural network model (attention model, Attention), but the present disclosure is not limited to this, and the artificial intelligence model can also include a convolutional neural network (CNN), a deep neural network (Deep Nueral Network, DNN), a recurrent neural network (Recurrent Nueral Network, RNN), a restricted Boltzmann machine (Restricted Boltzmann Machine, RBM), a deep belief network (Deep Belief Network, DBN), a bidirectional recursive deep neural network (Bidirectional Recurrent Deep Neural Network, BRDNN), a generative adversarial network (Generative Adversarial Network, GAN) and a neural network model of a deep Q network, or other artificial intelligence models.

[0111] According to an embodiment of the present disclosure, the historical registration audio information of the AC signal includes registration audio information corresponding to the extracted second audio signal evaluated as having the highest quality, and the historical BC registration information corresponds to the historical registration audio information of the AC signal.

[0112] The following uses Figure 11-12 right Fig.10 Provide further explanation.

[0113] Fig.11 A schematic diagram showing the principle of generating and updating the registered audio information of a user according to an exemplary embodiment of the present disclosure is shown. Fig.11 , get historical BC registration information To current BC registration information The transformation function f(·) is used to predict the registered audio information C reg The change function g(·) is based on the historical registered audio information C reg (1),C reg (2),…,C reg (M) and the variation function g(·) to obtain the registered audio information C of the AC signal reg (n) (i.e., the current C reg (n)) as the user's registered audio information. Here is the historical registered audio information The extracted audio evaluated as having the highest quality may be based on (used by) registered audio information of M (where M is a positive integer) users, and the historical BC registration information corresponds to the historical registration audio information.

[0114] Fig.12 FIG. 1 shows a flowchart of updating a user's registered audio information according to an exemplary embodiment of the present disclosure. Fig.12 ,The operation of updating the user’s registered audio information can be divided into two ,modules, namely: the bone conduction information change estimation module and the ,registered audio change prediction module.

[0115] 1. Bone conduction information change estimation module: learn the change characteristics of BC registration information from historical BC registration information and current BC registration information. When the user's voice changes, the change of bone conduction signal and air conduction signal are related and regular both in subjective hearing and objective laws. Use an artificial intelligence model (e.g., the first AI model) to learn the change characteristics of BC registration information from historical and current BC registration information. The input of this module is historical BC registration information. and current BC registration information And the output is the change characteristics from the historical BC registration information to the current BC registration information.

[0116] 2. The registered audio change prediction module includes two sub-modules, namely the registered audio information change prediction sub-module (for example, the second AI model) and the registered audio information estimation sub-module (for example, the third AI model).

[0117] The change characteristics of the BC registration information are input to the registration audio information change prediction submodule, and the registration audio information change prediction submodule predicts and outputs the change characteristics of the registration audio information (or called the change characteristics of the registration voice information).

[0118] Because the bone conduction signal is very reliable for the current speaker's characteristics and is only related to the user, it can reflect the low-frequency change characteristics of the registered audio information of the AC signal. When the user's voice changes, the change of the bone conduction signal is related to the change of the air conduction signal (audio signal) in terms of both subjective hearing and objective laws, and there is a regularity to follow. Therefore, the change of the air conduction signal can be deduced from the change of the bone conduction signal.

[0119] Change characteristics of registered audio information and historical registered audio information (or historical registered voice information) C reg (1),C reg (2),…,C reg (M) is input to the registration audio information estimation submodule, and the registration audio information estimation submodule outputs the registration audio information C of the AC signal reg (n) (i.e., the current registered voice information) serves as the user's registered audio information.

[0120] Historical registration audio information here C reg (1),C reg (2),…,C reg (M) and historical BC registration information It can be the previously collected registration audio information (i.e., the registration audio information of the AC signal) of M (M is a positive integer) users and their corresponding BC registration information, or it can be the registration audio information of M users based on which the extracted audio (i.e., the extracted second audio signal) with the highest quality is selected according to the evaluation result of the extracted voice quality evaluation module and their corresponding BC registration information. Here, the time period of the historical BC signal selection segment can correspond to the historical registration audio signal. High-quality historical registration audio information can reflect the stable characteristics of the user, and combining it with the change characteristics of the registration audio information predicted by the change characteristics of the BC registration information can generate the registration audio information of the changed AC signal.

[0121] It should be understood that the workflow diagram for updating a user's registered audio information here is only an example, and some modules may be omitted or other modules may be added, which is not limited in the present disclosure.

[0122] In an embodiment of the present disclosure, when the user's voice changes, the user's registered audio information can be updated to obtain the correct extracted voice. The current technology based on the target speaker's registration information often does not update the user's registered audio information after registering a voice in advance. When the user's voice changes, there is a large difference between the changed user's voice features and the pre-registered audio features, which will cause errors in the current extraction results, so the user may face the operation of re-registering. However, the registration environment is demanding, and changes in the user's voice are relatively common in life scenarios. Frequent registration will bring great inconvenience to the user. The present disclosure uses bone conduction signals to update the user's registered audio information in real time when the user's voice changes, without the need for the user to re-register, thereby improving the user experience.

[0123] return Figure 3 In step S320, based on the registered audio information of the user, a second audio signal of the user is extracted from the first audio signal.

[0124] According to an embodiment of the present disclosure, features of a first audio signal may be acquired; a mask about the user may be acquired based on the user's registered audio information and the features of the first audio signal; and a second audio signal may be extracted based on the mask and the features of the first audio signal.

[0125] According to an embodiment of the present disclosure, a mask about the user may be acquired based on the user's registered audio information, features of the first audio signal, and features of the bone conduction signal using a fourth AI model.

[0126] Return to reference Figure 2 The voice extraction module can be based on the user's registered audio information and the encoding (for example, Figure 2 The speech extraction module can be implemented by an artificial intelligence model (for example, a fourth artificial intelligence model), the input of which is the user's registered audio information, the features of the mixed voice and the features of the bone conduction signal, and outputs a mask about the user. The artificial intelligence model can be, for example, a neural network model (attention model, Attention), but the present disclosure is not limited to this. The artificial intelligence model can also be a neural network model including a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a generative adversarial network (GAN) and a deep Q network, or other artificial intelligence models.

[0127] According to an embodiment of the present disclosure, obtaining features of a first audio signal may include: extracting features from the first audio signal to obtain first frequency domain features; dividing the first frequency domain features into frequency bands to obtain sub-frequency domain features of each of a plurality of sub-bands; and respectively feature encoding the sub-frequency domain features of the plurality of sub-bands to obtain features of the plurality of sub-bands of the first audio signal as features of the first audio signal.

[0128] In order to model the audio signal and use the network to learn the intrinsic relationship of the signal, it is necessary to first extract the signal features and obtain high-dimensional information. This module encodes the input audio signal and obtains a high-dimensional feature vector.

[0129] Fig.13 A diagram of an encoder module according to an example embodiment of the present disclosure is shown. Fig.14 A network flow diagram of an encoder module according to an example embodiment of the present disclosure is shown.

[0130] The audio signal described below has a sampling rate of 16k and an input audio signal length of 4s. In actual use, you can input a signal of any length. Fig.13 and Fig.14 , the encoder module uses a sub-band approach for parallel processing, including the following three operations:

[0131] 1. Feature extraction:

[0132] Extract features from the input time domain signal (e.g., mixed speech or bone conduction signal) to obtain feature vectors of another dimension (i.e., features in the frequency domain), which is beneficial for the model to model and learn the input signal. This module can be implemented using Short-Time Fourier Transform (STFT) or other feature extraction methods, such as using an artificial intelligence model (e.g., convolutional neural network CNN) for feature extraction.

[0133] In this embodiment, feature extraction uses STFT, and the input signal s1 is framed, windowed, and short-time Fourier transformed to obtain the features in the frequency domain (i.e., the first frequency domain features). A signal of n seconds is sampled at a sampling rate of 16k, and there are sampling point data L=n*16000. STFT with the number of points s_n is performed, that is, the number of sampling points per frame is s_n, and the overlapping area between frames is s_n / 2 (50% overlap); after STFT, the number of frames:

[0134] k = L / (s_n / 2)-1;

[0135] The frequency points per frame are:

[0136] f = s_n / 2;

[0137] Take out the real part and imaginary part of the frequency domain respectively, and the output eigenvector f k The dimension is [k,2*f].

[0138] For calculating the STFT of 512 points of a 4s signal, the number of frequency points per frame can be obtained as s_n / 2=512 / 2=256, and the feature vector (i.e., the first frequency domain feature) f k The dimension is [249, 256], that is, there are 249 frames, each frame has 256 frequency points, and each frequency point can be represented by a real part and an imaginary part, where k = {0, 1, 2, ..., 248} represents the frame number.

[0139] 2. Frequency band division (division):

[0140] The encoder module can directly use the encoder to encode to obtain a higher-dimensional feature vector after performing the feature extraction in the previous step, or it can divide the features and encode the sub-features using their own encoders to reduce the complexity of the model and improve the processing speed of the model.

[0141] The embodiment of the present disclosure adopts the encoding method of sub-band features, and cuts the 16k frequency band into N sub-bands. Generally, the more sub-bands are divided, the finer the feature processing is, but more sub-encoders will be introduced, which will increase the model complexity. Considering the performance and model complexity, 4-6 encoders are the best.

[0142] In the embodiment of the present disclosure, the frequency domain feature (for example, the first frequency domain feature) is divided into frequency bands to obtain frequency domain features of multiple sub-bands (for example, sub-frequency domain features of multiple sub-bands). For example, a 4-sub-band division method is used to divide the feature data in the frequency domain obtained in the previous step into 4 sub-bands f1 for each frame of 256 frequency points. k , f2 k , f3 k , f4 k The frequency points included in each sub-band are {1~32}, {33~64}, {65~128}, {129~256}, and the corresponding frequencies are 0~2k, 2k~4k, 4k-8k, 8k~16k.

[0143] 3. Sub-coding:

[0144] If the frequency band division operation is not performed, only one encoder can be used to encode the full-band features to obtain a higher-dimensional feature vector. Since the embodiment of the present disclosure adopts a sub-band processing method, the full-band features are divided into multiple sub-band features, and each sub-band feature needs to be encoded using a different sub-band encoder (for example, a 2-D CNN operation is performed on the sub-band feature), thereby achieving parallel encoding and reducing complexity. If the number of sub-band divisions is N, there are N sub-band encoders corresponding to each sub-band feature. In the embodiment of the present disclosure, there are 4 sub-encoders (sub-band encoders), each of which will process the corresponding sub-band features and encode them into a higher-dimensional feature vector.

[0145] Calculate the subband eigenvector x1 k :Pair band f1 k The dimension [249,64] is expanded to [1,1,249,64], and a 2D convolution operation is performed. The output channel is 256, the convolution kernel is 5*5, and the step size is 1*1, and the output vector x1 is obtained. k [1,256,249,64].

[0146] Calculate the subband eigenvector x2 k :Pair band f2 k The dimension [249,64] is expanded to [1,1,249,64], and a 2D convolution operation is performed. The output channel is 256, the convolution kernel is 5*5, and the step size is 1*1, and the output vector x2 is obtained. k [1,256,249,64]

[0147] Calculate the subband eigenvector x3 k :Pair with f3 k The dimension [249,128] is expanded to [1,1,249,128], and a 2D convolution operation is performed. The output channel is 256, the convolution kernel is 5*6, and the step size is 1*2, and the output vector x3 is obtained. k [1,256,249,64]

[0148] Calculate subband eigenvector x4 k : Pair with f4 k The dimension [249,256] is expanded to [1,1,249,256], and a 2D convolution operation is performed. The output channel is 256, the convolution kernel is 5*6, and the step size is 1*4, and the output vector x4 is obtained. k [1,256,249,64]

[0149] After being processed by the encoder, the characteristic vector xi of each subband k The dimensions are all [1, 256, 249, 64].

[0150] Both the mixed speech (ie, the first audio signal) and the bone conduction signal can be used as inputs of the encoder module, thereby outputting features of multiple sub-bands of the mixed speech and features of multiple sub-bands of the bone conduction signal.

[0151] According to an embodiment of the present disclosure, based on the mask in the mask corresponding to each of the multiple subbands of the first audio signal and the characteristics of each subband of the first audio signal, the characteristics of each subband of the second audio signal are obtained; the characteristics of the multiple subbands of the second audio signal are feature decoded respectively to obtain second frequency domain characteristics of the multiple subbands; the second frequency domain characteristics of the multiple subbands are frequency band merged; and the second audio signal is obtained based on the merged second frequency domain characteristics.

[0152] In the decoding operation, the mask Mask about the user obtained by the speech extraction module and the features of the mixed speech output by the encoder are multiplied by a point operation, and the features are further decoded to restore the time domain signal of the target speech.

[0153] Fig.15 A decoder module diagram according to an example embodiment of the present disclosure is shown. Fig.16 FIG. 1 shows a decoder module network flow chart according to an exemplary embodiment of the present disclosure. Fig.15 and Fig.16 , the decoder module includes the following three operations:

[0154] Sub-decoding: Decode the sub-band mask mi predicted by the speech extraction module k The subband feature xi output by the encoder k A dot multiplication operation is performed on the corresponding elements (to obtain the features of multiple sub-bands of the second audio signal), and then a linear fully connected layer (Full Connection) is used for feature decoding (the inverse process of feature encoding, for example, the inverse process of the above-mentioned 2-D CNN operation). Other methods can also be used for feature conversion, such as but not limited to using an artificial intelligence model (for example, a convolutional neural network CNN), etc., to obtain the sub-band frequency domain features yi of the second audio signal. k (ie, the second frequency domain feature of the subband), where i represents the i-th subband and k represents the k-th frame.

[0155] Feature merging: The obtained sub-band frequency domain features yi k Merge to get the frequency domain feature y k (i.e., the second frequency domain feature) to facilitate the subsequent signal recovery process.

[0156] Time domain signal recovery: Short-time inverse Fourier transform can be used, or other feature transformation methods can be used, such as but not limited to using artificial intelligence models (e.g., convolutional neural network CNN) to recover speech signals. In the embodiment of the present disclosure, the encoder module uses short-time Fourier transform for feature extraction, so here the features are subjected to short-time inverse Fourier transform (e.g., 512-point short-time inverse Fourier transform) to obtain the extracted time domain signal of the target object as the extracted target audio (i.e., the second audio signal).

[0157] return Figure 3 In step S330, the extracted second audio signal and / or the first audio signal not including the second audio signal may be processed.

[0158] According to an embodiment of the present disclosure, the extracted second audio signal and / or the first audio signal excluding the second audio signal may be amplified, or the second audio signal may be mixed with the third audio signal.

[0159] According to an embodiment of the present disclosure, the extracted second audio signal and the first audio signal excluding the second audio signal may be amplified at different ratios.

[0160] In an embodiment of the present disclosure, the extracted audio signal and / or the mixed speech excluding the extracted audio signal may be amplified, and the extracted audio signal and the mixed speech excluding the extracted audio signal may be distinguishably amplified (e.g., at different ratios). In addition, the extracted audio signal may be mixed with other audio signals (e.g., music) to meet different needs of users. This will be described in conjunction with the following chat scenario and ear return scenario.

[0161] The method disclosed in the present invention can be applied to the scenario where the user is chatting with others, and ambient sound amplification can be used. Ambient sound amplification refers to amplifying all sounds that can be captured so that they can be clearly heard by the user, but this indiscriminate amplification will make the user's voice appear too loud and deafening. The method disclosed in the present invention is used to extract the user's voice from the ambient sound, and distinguish between the user's own voice and the ambient sound and amplify the user's own voice and the ambient sound (for example, the user's own voice is amplified twice, and the ambient sound is amplified five times) so that the user can get a better auditory experience. The steps in this scenario may include: 1) Turn on the switch to extract your own voice in the ambient sound amplification mode. 2) The earphones can pick up your own voice and ambient sound, and extract your own voice from the ambient sound. 3) The user can hear distant sounds, and his own voice will not be amplified too much.

[0162] The method disclosed in the present invention can be applied to the ear-return scenario. When the user sings, the user needs to hear his own voice to determine whether the rhythm and intonation of the song are correct. When the user sings on the stage, because the speaker is facing the audience and the noise on the stage is very loud, he needs an in-ear monitor to hear the rhythm of his singing. In addition, the method proposed in the present invention can be applied to headphones, for example. When the user sings using a karaoke application at home or in some small public places, he needs to use headphones to listen to the accompaniment. Ambient noise, reverberation, etc. will cause the user to be unable to hear his own song clearly. The method disclosed in the present invention is used to extract the user's voice from the ambient sound. The steps in this scenario may include: 1) the user turns on the button for extracting his own voice in the ambient sound mode; 2) starts extracting the user's own voice from the noisy environment; 3) mixes his own voice and the music rhythm and plays it in the headphones.

[0163] The method disclosed herein can be applied to some scenarios where the user does not need to speak, such as listening to a report, and the user can turn off the extraction function button to reduce the power consumption of the extraction device (e.g., headphones). The steps in this scenario may include: 1) the user chooses to turn off the button for extracting his own voice; 2) then in the subsequent headphone signal reception, the user can hear the voice of the distant speaker clearly. This scenario is mainly for scenarios where the user does not need to speak, and the voice extraction function can be selected to be turned on or off by the user interface provided by the extraction device, and power consumption can be saved when the user chooses to turn it off.

[0164] According to the above-mentioned method performed by the electronic device, the quality of the extracted audio is evaluated based on the bone conduction signal, and when the evaluated extracted audio quality is low, the user's registered audio information is updated in real time through the bone conduction signal. On the one hand, there is no need for advance registration of the user's voice. In the existing voice extraction technology, because at least one sentence needs to be input to decouple the spoken content from the voice signal to eliminate the impact on the content and obtain the speaker's global voice feature expression, advance registration is required. And because of environmental changes such as the generation of environmental noise, registration failure may occur, and the registered voice needs to be collected in a relatively quiet environment, which has relatively strict requirements on the collection environment. The method disclosed in the present invention does not require advance registration of the user's voice, which greatly facilitates users and improves user experience. On the other hand, the method of the present invention realizes adaptive real-time registration of the user's voice, and even in some cases, the user's voice characteristics have changed (including the user may have a hoarse voice due to a cold or sore throat; the user's vocal cords may experience edema in many cases, such as passive changes such as changes in weather / location, the user drinking or physical fatigue, and also including active changes when the user sings, he will change the pitch and timbre of the voice, such as the pitch becomes sharp and the user's voice is more likely to sound very different), the method of the present invention can also achieve real-time update of the registered voice, thereby improving the quality of voice extraction and the naturalness of the voice.

[0165] Fig.17 A schematic block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0166] Reference Fig.17 , the electronic device 1700 of the present disclosure may include a registered audio determination module 1710, an audio extraction module 1720 and an audio processing module 1730. The registered audio determination module 1710 may determine the registered audio information of the user based on the bone conduction signal, the audio extraction module 1720 may extract the second audio signal of the user from the first audio signal based on the registered audio information of the user; the audio processing module 1730 may process the extracted second audio signal and / or the first audio signal not containing the second audio signal.

[0167] According to an embodiment of the present disclosure, the registration audio determination module 1710 may evaluate the quality of the extracted second audio signal based on the content of the extracted second audio signal and the bone conduction BC signal; and determine the user's registration audio information based on the quality of the extracted second audio signal.

[0168] According to an embodiment of the present disclosure, the registration audio determination module 1710 may classify the extracted second audio signal and the BC signal based on speech features; determine the matching probability between the category of the extracted second audio signal and the category of the BC signal; and evaluate the quality of the extracted second audio signal based on the matching probability.

[0169] According to an embodiment of the present disclosure, the registration audio determination module 1710 may use a pre-trained classifier to classify the extracted second audio signal and the BC signal respectively, and: search the pre-trained matching probability table for the category of the extracted second audio signal and the category of the BC signal to determine the matching probability.

[0170] According to an embodiment of the present disclosure, the registration audio determination module 1710 can determine that the user's registration audio information needs to be updated based on the quality of the extracted second audio signal; determine the characteristic change of the BC registration information; and based on the change characteristics of the BC registration information, predict the registration audio information of the air-conducted AC signal as the user's registration audio information.

[0171] According to an embodiment of the present disclosure, the registration audio determination module 1710 may determine that the user's registration audio information needs to be updated when the quality of the extracted second audio signal does not meet the predetermined conditions; and determine that the user's registration audio information does not need to be updated when the quality of the extracted second audio signal meets the predetermined conditions.

[0172] According to an embodiment of the present disclosure, the registration audio determination module 1710 can determine the characteristic changes of the BC registration information based on the historical BC registration information and the current BC registration information using a first artificial intelligence AI model, predict the characteristic changes of the registration audio information of the AC signal based on the characteristic changes of the BC registration information using a second AI model, and obtain the registration audio information of the AC signal as the user's registration audio information based on the characteristic changes of the registration audio information of the AC signal and the historical registration audio information of the AC signal using a third AI model.

[0173] According to an embodiment of the present disclosure, the historical registration audio information of the AC signal includes registration audio information corresponding to the extracted second audio signal evaluated as having the highest quality, and the historical BC registration information corresponds to the historical registration audio information of the AC signal.

[0174] According to an embodiment of the present disclosure, the audio extraction module 1720 may obtain features of a first audio signal; obtain a mask about the user based on the user's registered audio information and the features of the first audio signal; and extract a second audio signal based on the mask and the features of the first audio signal.

[0175] According to an embodiment of the present disclosure, the audio extraction module 1720 may obtain a mask about the user by using the fourth AI model based on the user's registered audio information, the characteristics of the first audio signal, and the characteristics of the bone conduction signal.

[0176] According to an embodiment of the present disclosure, the audio extraction module 1720 can perform feature extraction on the first audio signal to obtain a first frequency domain feature; divide the first frequency domain feature into frequency bands to obtain sub-frequency domain features of each of the multiple sub-bands; and perform feature encoding on the sub-frequency domain features of the multiple sub-bands respectively to obtain features of the multiple sub-bands of the first audio signal as features of the first audio signal.

[0177] According to an embodiment of the present disclosure, the audio extraction module 1720 can obtain the characteristics of each subband of the second audio signal based on the mask in the mask corresponding to each subband of the multiple subbands of the first audio signal and the characteristics of each subband of the first audio signal; perform feature decoding on the characteristics of the multiple subbands of the second audio signal respectively to obtain second frequency domain characteristics of the multiple subbands; merge the second frequency domain characteristics of the multiple subbands; and obtain the second audio signal based on the merged second frequency domain characteristics.

[0178] According to an embodiment of the present disclosure, the audio processing module 1730 may amplify the extracted second audio signal and / or the first audio signal excluding the second audio signal, or mix the second audio signal with the third audio signal.

[0179] According to an embodiment of the present disclosure, the audio processing module 1730 may amplify the extracted second audio signal and the first audio signal excluding the second audio signal at different ratios.

[0180] According to the above-mentioned device for voice extraction, the quality of the extracted audio is evaluated based on the bone conduction signal, and when the evaluated extracted audio quality is low, the user's registered audio information is updated in real time through the bone conduction signal, thereby realizing adaptive real-time registration of the user's voice. On the one hand, there is no need for advance registration of the user's voice, which improves the user experience. On the other hand, when the user's voice characteristics change, the real-time update of the registered voice can also be realized, which improves the quality of voice extraction and the naturalness of the voice.

[0181] The method executed by the electronic device and the basic information of the electronic device disclosed in the present invention include: the purpose is to extract the user's own voice from the mixed voice by automatic voice registration, without the need to include the target registration voice in advance, applicable scenarios include voice calls and chats, and special hardware requirements may include: TWS with VPU (or related hardware units that can pick up bone conduction signals).

[0182] The measured results of the target object of this disclosure are shown in Table 1 below:

[0183] Table 1

[0184]

[0185] In order to illustrate the performance of the disclosed method in maintaining speech extraction when the speech changes. We mixed three types of noise, namely, noisy speech noise (multiple people talking), car noise and music, with the WSJ0 data to simulate the actual complex environment, with signal-to-noise ratios (SNR) of 0, 5, 10 and 15 dB respectively. The evaluation metric uses the scale-invariant signal-to-distortion ratio (SISDR), which is a common metric for evaluating the performance of extraction methods. It is a ratio expressed in dB, and the higher the ratio, the better the performance.

[0186] The results show that under the average SISDR, the average SISDR of the prior art when using a 0.5s registration tone reaches 7.68dB, which is 34.24% higher than the average result of 10.31dB of the present invention. This shows that the method of the present invention performs better than the prior art. And even when the ambient noise is high, the method of the present invention can still extract the speech of the target object well.

[0187] The method disclosed in this disclosure can be applied to target voice extraction in voice calls or recorded audio of mobile phones or headphones. Using the method disclosed in this disclosure, real-time imperceptible registration of the target object and real-time update of the registered voice can be achieved, and the sound of the target object can be extracted based on the registered voice, while significantly improving the language quality and naturalness of the call. In addition, the example network disclosed in this disclosure can be used not only in voice extraction tasks, but also in voice enhancement and voice separation tasks. Overall solution and Figure 2 That is, without modifying the model, only the input training data and training objectives need to be changed to achieve the tasks of speech enhancement and speech separation using the present disclosure.

[0188] Fig.18 FIG. 1 is a block diagram of an electronic device according to an example embodiment. Fig.18 The electronic device 1800 may include at least one memory 1810 and at least one processor 1820, wherein the at least one memory stores computer executable instructions, and when the computer executable instructions are executed by the at least one processor, the at least one processor 1820 executes the method performed by the electronic device according to an embodiment of the present disclosure.

[0189] At least one of the above-mentioned multiple modules can be implemented by an AI model. Functions associated with AI can be executed by a non-volatile memory, a volatile memory, and a processor.

[0190] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as a central processing unit (CPU), an application processor (AP), etc., a processor used only for graphics (such as a graphics processing unit (GPU), a visual processing unit (VPU) and / or an AI-specific processor (such as a neural processing unit (NPU)).

[0191] One or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models can be provided by training or learning. Here, providing by learning means that by applying a learning algorithm to multiple learning data, a predefined operating rule or AI model with desired characteristics is formed. Learning can be performed in the device itself that executes AI according to an embodiment, and / or can be implemented by a separate server / device / system.

[0192] A learning algorithm is a method of using multiple learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0193] According to the present invention, in the image processing method performed by the electronic device, an output image after processing the target area can be obtained by using the input image as input data of the artificial intelligence model.

[0194] The artificial intelligence model can be obtained through training. Here, "obtained through training" means training a basic artificial intelligence model with multiple training data through a training algorithm to obtain a predefined operating rule or artificial intelligence model, which is configured to perform the desired features (or purposes).

[0195] As an example, the artificial intelligence model may include multiple neural network layers. Each of the multiple neural network layers includes multiple weight values, and the neural network calculation is performed by calculating between the calculation result of the previous layer and the multiple weight values. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recursive neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recursive deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks. As an example, the electronic device may be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above-mentioned instructions (or instruction sets) individually or jointly. The electronic device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with an interface locally or remotely (e.g., via wireless transmission).

[0196] In electronic devices, the processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0197] The processor can execute instructions or codes stored in the memory, wherein the memory can also store data. Instructions and data can also be sent and received through the network via the network interface device, wherein the network interface device can adopt any known transmission protocol.

[0198] The memory may be integrated with the processor, for example, RAM or flash memory is arranged within an integrated circuit microprocessor or the like. In addition, the memory may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory and the processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the memory.

[0199] In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0200] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute a method performed by an electronic device according to an exemplary embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The instructions or computer programs in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0201] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method performed by an electronic device, comprising: Determining the user's registered audio information based on the bone conduction signal; Extracting a second audio signal of the user from the first audio signal based on the registered audio information of the user; The extracted second audio signal and / or the first audio signal excluding the second audio signal is processed.

2. The method of claim 1, wherein: Determine the user's registered audio information based on the bone conduction signal, including: evaluating the quality of the extracted second audio signal based on the content of the extracted second audio signal and the bone conduction BC signal; Based on the quality of the extracted second audio signal, the registered audio information of the user is determined.

3. The method of claim 2, wherein: The quality of the extracted second audio signal is evaluated, including: Based on the speech features, classifying the extracted second audio signal and the BC signal; determining a matching probability between the category of the extracted second audio signal and the category of the BC signal; The quality of the extracted second audio signal is evaluated based on the matching probability.

4. The method of claim 3, wherein: Based on the speech features, the extracted second audio signal and the BC signal are classified, including: The extracted second audio signal and the BC signal are respectively classified using a pre-trained classifier, and The step of determining the matching probability between the category of the extracted second audio signal and the category of the BC signal includes: A pre-trained matching probability table is searched for the category of the extracted second audio signal and the category of the BC signal to determine the matching probability.

5. The method according to claim 2 or 3, wherein: Determining the user's registered audio information based on the quality of the extracted second audio signal includes: Based on the quality of the extracted second audio signal, determining that the registered audio information of the user needs to be updated; Determine changes in the characteristics of BC registration information; Based on the characteristic changes of the BC registration information, the registration audio information of the air-conducted AC signal is predicted as the registration audio information of the user.

6. The method of claim 5, wherein: Determining, based on the quality of the extracted second audio signal, that the registered audio information of the user needs to be updated includes: If the quality of the extracted second audio signal does not meet a predetermined condition, determining that the registered audio information of the user needs to be updated; and In a case where the quality of the extracted second audio signal satisfies a predetermined condition, it is determined that there is no need to update the registered audio information of the user.

7. The method of claim 5, wherein: Determine changes in the characteristics of BC registration information, including: Based on the historical BC registration information and the current BC registration information, the first artificial intelligence AI model is used to determine the characteristic changes of the BC registration information, and The method of predicting the registration audio information of the air-conducted AC signal as the registration audio information of the user based on the characteristic change of the BC registration information includes: Based on the characteristic changes of the BC registration information, the second AI model is used to predict the characteristic changes of the registration audio information of the AC signal, and Based on the characteristic changes of the registered audio information of the AC signal and the historical registered audio information of the AC signal, the third AI model is used to obtain the registered audio information of the AC signal as the registered audio information of the user.

8. The method of claim 7, wherein: The historical registration audio information of the AC signal includes registration audio information corresponding to the extracted second audio signal evaluated to be of the highest quality, and the historical BC registration information corresponds to the historical registration audio information of the AC signal.

9. The method of claim 1, wherein: Extracting a second audio signal of the user from the first audio signal based on the registered audio information of the user includes: Acquiring a characteristic of a first audio signal; Based on the registered audio information of the user and the characteristics of the first audio signal, obtaining a mask about the user; Based on the mask and the feature of the first audio signal, a second audio signal is extracted.

10. The method of claim 9, wherein: Acquiring a mask about the user based on the registered audio information of the user and the feature of the first audio signal, including: Based on the user's registered audio information, the characteristics of the first audio signal, and the characteristics of the bone conduction signal, a mask about the user is obtained using the fourth AI model.

11. The method of claim 9, wherein: Acquiring a feature of the first audio signal includes: Performing feature extraction on the first audio signal to obtain a first frequency domain feature; Dividing the first frequency domain feature into frequency bands to obtain sub-frequency domain features of each of the multiple sub-bands; and Feature encoding is performed on the sub-frequency domain features of the multiple sub-bands respectively, and the features of the multiple sub-bands of the first audio signal are obtained as the features of the first audio signal.

12. The method of claim 11, wherein: Extracting a second audio signal based on the mask and a feature of the first audio signal includes: acquiring a feature of each subband of the second audio signal based on a mask in the mask corresponding to each of the plurality of subbands of the first audio signal and a feature of each subband of the first audio signal; Performing feature decoding on features of multiple sub-bands of the second audio signal respectively to obtain second frequency domain features of the multiple sub-bands; Performing frequency band merging on the second frequency domain features of the plurality of sub-bands; and A second audio signal is obtained based on the combined second frequency domain features.

13. The method of claim 1, wherein: Processing the extracted second audio signal and / or the first audio signal not including the second audio signal includes: amplifying the extracted second audio signal and / or the first audio signal not including the second audio signal, or The second audio signal is mixed with the third audio signal.

14. The method of claim 13, wherein: Amplifying the extracted second audio signal and the first audio signal excluding the second audio signal includes: The extracted second audio signal and the first audio signal not including the second audio signal are amplified at different ratios.

15. An electronic device, comprising: A registration audio determination module, configured to determine the user's registration audio information based on the bone conduction signal; an audio extraction module, configured to extract a second audio signal of the user from the first audio signal based on the registered audio information of the user; The audio processing module is configured to process the extracted second audio signal and / or the first audio signal not including the second audio signal.

16. An electronic device, comprising: processor; A memory storing a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

17. A computer-readable storage medium, wherein: A computer program is stored thereon, and when the program is executed, the method according to any one of claims 1 to 14 is implemented.

Images