Hearing aid tone customizing method, mobile terminal and storage medium

Abstract

The invention discloses a hearing aid tone customizing method, a mobile terminal and a storage medium, and the method comprises the steps: receiving a first audio sample comprising acoustic features, the acoustic features comprising audio features and text features; extracting the acoustic features corresponding to the first audio sample, and inputting the acoustic features into a Transform GPT language model to obtain a first acoustic code sequence corresponding to the first audio sample; and based on a received target audio sample and the first sound code sequence, synthesizing a first fitting audio corresponding to the target audio sample, and transmitting the first fitting audio to a hearing aid for playing. The method can meet the requirement of a user for customizing the audio and improve the user experience.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a hearing aid tone customization method, a mobile terminal and a storage medium. Background Art

[0002] Hearing loss can seriously affect the physical and mental health of hearing-impaired patients. Wearing hearing aids is currently the most effective way for hearing-impaired patients to improve their hearing.

[0003] Currently, hearing aid fitting is mainly achieved by fitters constantly playing audio to users and constantly debugging hearing aids based on user feedback. Due to the single fitting audio, the actual hearing aid experience of users is not good. In addition, since some hearing aid users have a relatively fixed communication group, there is a demand for playing customized audio through hearing aids. Obviously, traditional hearing aids can no longer meet the personalized needs of users, and the user experience is poor.

[0004] Application Contents

[0005] The present application provides a hearing aid tone customization method, a mobile terminal, and a storage medium to meet the user's needs for customized audio and improve the user experience.

[0006] In a first aspect, the present application provides a hearing aid timbre customization method, comprising:

[0007] Receiving a first audio sample including acoustic features, wherein the acoustic features include audio features and text features;

[0008] Extracting the acoustic feature corresponding to the first audio sample, and inputting the acoustic feature into a Transformer GPT language model to obtain a first voice code sequence corresponding to the first audio sample;

[0009] Based on the received target audio sample and the first voice code sequence, a first fitting audio corresponding to the target audio sample is synthesized, and the first fitting audio is transmitted to the hearing aid for playback.

[0010] By extracting the acoustic features of the first audio sample, the embodiment of the present application can completely retain the rich acoustic information in the first audio sample, facilitate the subsequent cloning of the text, timbre, semantics and intonation in the first audio sample, and ensure the quality of the subsequent synthesized audio; by inputting the acoustic features into the Transformer GPT language model to obtain the first voice code sequence corresponding to the first audio sample, the first voice code sequence corresponding to all the information in the acoustic features can be effectively captured, and the subsequent synthesis of the first fitting audio is convenient; based on the received target audio sample and the first voice code sequence, the first fitting audio corresponding to the target audio sample is synthesized, and the first fitting audio is transmitted to the hearing aid for playback, so that the timbre can be customized, and the target audio sample can be converted into the timbre of a person familiar to the user, meeting the user's demand for customized audio and improving the user experience.

[0011] Furthermore, before receiving the first audio sample including the acoustic feature, the method further includes:

[0012] Determine whether a default timbre operation input by the user is received;

[0013] If the default timbre operation is received, a second audio sample detected in the hearing aid is obtained, and based on the second audio sample and the default vocode sequence, a second fitting audio corresponding to the second audio sample is generated, and the second fitting audio is transmitted to the hearing aid for playback;

[0014] If the default timbre operation is not received, the first audio sample including the acoustic feature is received.

[0015] Further, the extracting the acoustic feature corresponding to the first audio sample is specifically:

[0016] extracting a feature vector of the first audio sample through an audio codec model;

[0017] The feature vector is quantized step by step to obtain the acoustic feature corresponding to the first audio sample.

[0018] In this way, by extracting the acoustic features of the first audio sample through the audio codec model, rich acoustic information such as speaker identity, emotion, environmental noise, etc. in the first audio sample can be completely retained, which facilitates the subsequent cloning of text, timbre, semantics and intonation in the first audio sample, and ensures the quality of subsequent synthesized audio.

[0019] Furthermore, the step of inputting the acoustic features into the Transformer GPT language model to obtain a first sound coding sequence corresponding to the first audio sample is specifically:

[0020] Converting the text features in the acoustic features into a first-level voice code sequence through an autoregressive encoder;

[0021] Converting the audio features in the acoustic features into a second-level vocoded sequence by a non-autoregressive encoder;

[0022] The first-level vocoding sequence and the second-level vocoding sequence are merged to obtain a first vocoding sequence corresponding to the first audio sample.

[0023] In this way, by inputting the acoustic features into the Transformer GPT language model to obtain the first voice code sequence corresponding to the first audio sample, the first voice code sequence corresponding to all the information in the acoustic features can be effectively captured, which facilitates the synthesis of subsequent fitting audio, meets the user's needs for customized audio, and improves the user experience.

[0024] Furthermore, the synthesizing a first fitting audio corresponding to the target audio sample based on the received target audio sample and the first voice coding sequence is specifically:

[0025] converting the vocoded sequence into a speech waveform by a decoder of an audio codec model;

[0026] Based on the target audio sample and the speech waveform, a first fitting audio corresponding to the target audio sample is synthesized.

[0027] In this way, based on the received target audio sample and the first voice code sequence, the first fitting audio corresponding to the target audio sample is synthesized, speech synthesis can be performed directly based on the discrete voice code sequence, and the target audio sample can be converted into the timbre of a person familiar to the user, thereby meeting the user's demand for customized audio and improving the user experience.

[0028] Furthermore, after transmitting the first fitting audio to the hearing aid for playing, the method further includes:

[0029] Receive a save tone operation selected by the user;

[0030] Acquire the first voice code sequence corresponding to the first audio sample based on the timbre saving operation, and acquire a user-defined name for the first voice code sequence;

[0031] The first vocoding sequence and the name are saved.

[0032] In this way, by obtaining the name of the first voice coding sequence corresponding to the first audio sample customized by the user and saving the first voice coding sequence and the name, the user can save a satisfactory first voice coding sequence, which is convenient for using the voice coding sequence to generate other debugging feedback audio in the future, thereby meeting the user's customized audio needs and improving the user experience.

[0033] In a second aspect, the present application provides a mobile terminal, including: a receiving module, an extracting module and a synthesizing module;

[0034] The receiving module is used to receive a first audio sample including an acoustic feature, wherein the acoustic feature includes an audio feature and a text feature;

[0035] The extraction module is used to extract the acoustic features corresponding to the first audio sample, and input the acoustic features into the Transformer GPT language model to obtain a first voice code sequence corresponding to the first audio sample;

[0036] The synthesis module is used to synthesize a first fitting audio corresponding to the target audio sample based on the received target audio sample and the first voice code sequence, and transmit the first fitting audio to the hearing aid for playback.

[0037] By extracting the acoustic features of the first audio sample, the embodiment of the present application can completely retain the rich acoustic information in the first audio sample, facilitate the subsequent cloning of the text, timbre, semantics and intonation in the first audio sample, and ensure the quality of the subsequent synthesized audio; by inputting the acoustic features into the Transformer GPT language model to obtain the first voice code sequence corresponding to the first audio sample, the first voice code sequence corresponding to all the information in the acoustic features can be effectively captured, and the subsequent synthesis of the first fitting audio is convenient; based on the received target audio sample and the first voice code sequence, the first fitting audio corresponding to the target audio sample is synthesized, and the first fitting audio is transmitted to the hearing aid for playback, so that the timbre can be customized, and the target audio sample can be converted into the timbre of a person familiar to the user, meeting the user's demand for customized audio and improving the user experience.

[0038] Furthermore, the mobile terminal further includes: a selection module;

[0039] The receiving module is used to receive a first audio sample including an acoustic feature, wherein the acoustic feature includes an audio feature and a text feature;

[0040] The extraction module is used to extract the acoustic features corresponding to the first audio sample, and input the acoustic features into the Transformer GPT language model to obtain a first voice code sequence corresponding to the first audio sample;

[0041] The synthesis module is used to synthesize a first fitting audio corresponding to the target audio sample based on the received target audio sample and the first voice code sequence, and transmit the first fitting audio to the hearing aid for playback.

[0042] Further, the extraction module includes: an extraction unit;

[0043] The extraction unit is used to extract the acoustic feature corresponding to the first audio sample, including an extraction subunit and a quantization subunit;

[0044] The extraction subunit is used to extract the feature vector of the first audio sample through an audio codec model;

[0045] The quantization subunit is used to quantize the feature vector step by step to obtain the acoustic feature corresponding to the first audio sample.

[0046] Furthermore, the extraction module further comprises: an input unit;

[0047] The input unit is used to input the acoustic feature into the Transformer GPT language model to obtain a first voice code sequence corresponding to the first audio sample, including a first conversion subunit, a second conversion subunit and a synthesis subunit;

[0048] The first conversion subunit is used to convert the text features in the acoustic features into a first-level voice code sequence through an autoregressive encoder;

[0049] The second conversion subunit is used to convert the audio feature in the acoustic feature into a second-level voice code sequence through a non-autoregressive encoder;

[0050] The synthesis subunit is used to merge the first-level vocoding sequence and the second-level vocoding sequence to obtain a first vocoding sequence corresponding to the first audio sample.

[0051] Further, the synthesis module includes: a conversion unit and a synthesis unit;

[0052] The conversion unit is used to convert the vocoded sequence into a speech waveform through a decoder of an audio codec model;

[0053] The synthesis unit is used to synthesize a first fitting audio corresponding to the target audio sample based on the target audio sample and the speech waveform.

[0054] Furthermore, the mobile terminal also includes a storage module;

[0055] The storage module includes: a receiving unit, a naming unit and a storage unit;

[0056] The receiving unit is used to receive a save timbre operation selected by a user;

[0057] The naming unit is used to obtain the first voice code sequence corresponding to the first audio sample based on the timbre saving operation, and obtain a user-defined name for the first voice code sequence;

[0058] The storage unit is used to store the first voice code sequence and the name.

[0059] In a third aspect, the present application provides a computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the hearing aid tone customization method as described in the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flowchart of an embodiment of a hearing aid timbre customization method provided by the present application;

[0061] Figure 2 It is a structural diagram of an embodiment of a mobile terminal provided by the present application. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0063] It should be understood that the step numbers used in this article are only for the convenience of description and are not intended to limit the order in which the steps are executed.

[0064] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0065] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0066] The term "and / or" means and includes any and all possible combinations of one or more of the associated listed items.

[0067] Hearing loss seriously affects the physical and mental health of hearing-impaired patients, and wearing hearing aids is an effective way to improve hearing. Currently, hearing aids are mainly adjusted by fitters to play audio and based on user feedback, but the actual experience is not good. Some hearing aid users have the need to customize audio, but traditional hearing aids are difficult to meet users' personalized needs, resulting in poor user experience.

[0068] Embodiment 1

[0069] Please refer to Figure 1 , Figure 1 It is a flowchart of an embodiment of a hearing aid timbre customization method provided by the present application, including steps S1 to S3;

[0070] Step S1, receiving a first audio sample including acoustic features, wherein the acoustic features include audio features and text features;

[0071] It is understandable that the first audio sample may be recorded directly through the mobile terminal, or may be recorded in advance by the user through existing technology and uploaded to the mobile terminal.

[0072] It is understandable that the hearing aid needs to be connected to the hearing aid before the hearing aid can play the audio sample selected by the user, wherein the mobile terminal can establish a connection with the hearing aid through wireless communication technologies such as wifi and Bluetooth. After the mobile terminal is connected to the hearing aid, the user can choose whether to use the default timbre in the mobile terminal. If the mobile terminal receives the user's selection of the default timbre operation, the second audio sample detected in the hearing aid is obtained, and based on the second audio sample and the default voice code sequence, the second fitting audio corresponding to the second audio sample is generated, and the second fitting audio is transmitted to the hearing aid for playback. If the default timbre operation is not received, the first audio sample including the acoustic feature is received; wherein the default timbre operation can be implemented through a button. Exemplarily, when the user clicks the default timbre operation in the mobile terminal, the real-time voice fitting process is entered, and the mobile terminal will combine the second audio sample received from the hearing aid with the default voice code sequence, quickly generate the second fitting audio corresponding to the default timbre, and play it through the hearing aid.

[0073] It should be noted that the hearing aid tone customization method can be installed in scenarios such as applications, web pages and platforms, and the method can be implemented by importing the applications, web pages and platforms equipped with the hearing aid tone customization method into a mobile terminal, wherein the mobile terminal can be a mobile phone, a tablet, a computer, etc., and the present invention does not impose any restrictions on this.

[0074] It should be noted that there may be multiple default voice code sequences, and multiple default voice code sequences will be displayed on the adjustment interface of the mobile terminal. The user can hear audio corresponding to different timbres in the hearing aid by changing the default voice code sequence on the adjustment interface.

[0075] It should be noted that after the second fitting audio is generated, the mobile terminal will also obtain the parameters corresponding to the second fitting audio and display them on the adjustment interface of the mobile terminal. The user adjusts the parameters by adjusting the slider, and can adjust the frequency signal and sound intensity of the second fitting audio to adjust the response of the user's auditory system to different audio signals and improve the auditory experience of people with hearing loss.

[0076] In this way, based on the user's selection of the default tone operation, it can help the user confirm whether customized audio settings are needed, meet the user's needs for customized audio and improve the user experience.

[0077] Step S2: extracting the acoustic features corresponding to the first audio sample, and inputting the acoustic features into a Transformer GPT language model to obtain a first voice code sequence corresponding to the first audio sample;

[0078] First, a feature vector of the first audio sample is extracted through an audio codec model; and the feature vector is quantized step by step to obtain the acoustic feature corresponding to the first audio sample. Exemplarily, the high-dimensional first audio sample is compressed into a low-dimensional feature vector through the encoder in the audio codec model of val ecodec, and then the feature vector is quantized step by step by combining val codec with RVQ to completely retain all information of the first audio sample. The present application is not limited to the audio codec model of val codec.

[0079] It should be noted that the value codec is based on a neural codec model, which can convert speech signals into discrete coding representations, retain all the information in the audio, and is used to achieve high-quality audio synthesis. The value codec mainly includes an encoder and a decoder.

[0080] RVQ stands for Residual Vector Quantization, which is a quantization technology used in image and video compression. It can use the residual of the vector for further quantization to convert the vector into a low-dimensional codeword to reduce the amount of data and facilitate storage and transmission.

[0081] In this way, by extracting the acoustic features of the first audio sample through the audio codec model, rich acoustic information such as speaker identity, emotion, environmental noise, etc. in the first audio sample can be completely retained, which facilitates the subsequent cloning of text, timbre, semantics and intonation in the first audio sample, and ensures the quality of subsequent synthesized audio.

[0082] Secondly, the acoustic features are input into the Transformer GPT language model, and the text features in the acoustic features are converted into a first-level vocoder sequence through an autoregressive encoder; the audio features in the acoustic features are converted into a second-level vocoder sequence through a non-autoregressive encoder; the first-level vocoder sequence and the second-level vocoder sequence are fused to obtain a first vocoder sequence corresponding to the first audio sample. It can be understood that the acoustic features corresponding to the first audio sample are input into the autoregressive encoder AR to convert the text features in the acoustic features into a first-level vocoder sequence according to conditional contexts such as a phoneme sequence of the text and an existing audio vocoder sequence, and the acoustic features corresponding to the first audio sample are input into the non-autoregressive encoder NAR to convert the audio features in the acoustic features into a second-level vocoder sequence; wherein the second-level vocoder sequence includes a generated codebook and a vocoder sequence of the speaker's voice, etc.

[0083] It should be noted that the Tranformer GPT language model is a powerful sequence generation model based on the Transformer architecture, consisting of multiple layers of encoders and decoders. Each layer contains a multi-head attention mechanism and a feedforward neural network to capture the long-term dependencies of the input sequence and generate the target sequence. Among them, the Tranformer GPT language model includes an autoregressive encoder AR and a non-autoregressive encoder NAR.

[0084] It should be noted that the training process of the Tranformer GPT language model is specifically as follows: it includes two steps: training data preparation and model training. In the training data preparation, firstly, several audio data sets of predetermined length are collected, wherein the audio data sets include Chinese and English audio. The audio data sets are preprocessed to reduce noise and segment the original audio signals into small audio segments of less than 1 minute, and the speech in the truncated small audio segments is guaranteed to be complete. Then, the ASR model is used to identify the unlabeled data to generate speech transcription text, and the text is normalized, cleaned and standardized, and the noise characters and meaningless punctuation are removed. The text is further divided into words and phonemes for model processing. Finally, the audio samples are encoded and quantized using the pre-trained audio codec model to generate the corresponding discrete sequence matrix. In the model training, the constructed phoneme prompts and acoustic prompts are input into the Transformer decoder model for training to generate the acoustic token of the target audio sample. The generator uses a 12-layer Transformer architecture, each Transformer contains 16 attention heads, the embedding dimension of each attention head is 1024, the intermediate dimension of the feedforward layer is 4096, and the final output layer dimension is 1024. The discriminator uses the WaveGAN model to guide the training of the generator by learning the direct difference between the real speech and the generated speech. The Tranformer GPT language model is trained on 8 NVIDIA TESLAV10040GB GPUs, each GPU processing 8000 acoustic tokens. The AdamW optimizer is used for parameter update. The learning rate is gradually increased to 5×10^-3 in the first 32,000 steps, and then gradually linearly decayed. The training is repeated until 800,000 steps of training are completed. For each training sample, the cross entropy loss between the acoustic token sequence output by the model and the target acoustic token sequence is calculated through the generator, and the gradient is back-propagated. The Tranformer GPT language model is obtained by alternating training of the generator and the discriminator.

[0085] In this way, by inputting the acoustic features into the Transformer GPT language model to obtain the first voice code sequence corresponding to the first audio sample, the first voice code sequence corresponding to all the information in the acoustic features can be effectively captured, which facilitates the synthesis of subsequent fitting audio, meets the user's needs for customized audio, and improves the user experience.

[0086] Step S3: synthesize a first fitting audio corresponding to the target audio sample based on the received target audio sample and the first voice code sequence, and transmit the first fitting audio to the hearing aid for playback.

[0087] Specifically, the vocoded sequence is converted into a speech waveform by a decoder of the audio codec model; and based on the target audio sample and the speech waveform, a first fitting audio corresponding to the target audio sample is synthesized. It can be understood that the vocoded sequence generated by the val le codec can be converted into a speech waveform by a decoder of the val le codec, and based on the speech waveform and the target audio sample, a first fitting audio corresponding to the target audio sample is synthesized.

[0088] It should be noted that the target audio samples are audio samples collected by the hearing aid in real time.

[0089] In this way, based on the received target audio sample and the first voice code sequence, the first fitting audio corresponding to the target audio sample is synthesized, speech synthesis can be performed directly based on the discrete voice code sequence, and the target audio sample can be converted into the timbre of a person familiar to the user, thereby meeting the user's demand for customized audio and improving the user experience.

[0090] When the user hears a satisfactory audio sample in the hearing aid, the user can select the save timbre operation in the mobile terminal. The mobile terminal will obtain the first voice code sequence corresponding to the first audio sample based on the save timbre operation. After receiving the user's custom name for the first voice code sequence, the mobile terminal will save the first voice code sequence and the user-defined name. Among them, the audio saving operation can be implemented by clicking a button of the mobile terminal. Exemplarily, when the user hears a satisfactory audio sample, he will click the save button of the mobile terminal. Then the user can name the voice code sequence corresponding to the audio sample as "friend" or "family", etc. The mobile terminal will save the voice code sequence corresponding to the audio sample and the user-defined name. The user can directly find the saved voice code sequence of "friend" or "family" in the mobile terminal next time. By clicking, the voice code sequence of "friend" or "family" can be combined with the target audio sample obtained by the hearing aid to synthesize the first fitting audio corresponding to the target audio sample, and play the target audio sample containing the timbre of "friend" or "family" through the hearing aid.

[0091] In this way, by obtaining the name of the first voice coding sequence corresponding to the first audio sample customized by the user and saving the first voice coding sequence and the name, the user can save a satisfactory first voice coding sequence, which is convenient for using the voice coding sequence to generate other debugging feedback audio in the future, thereby meeting the user's customized audio needs and improving the user experience.

[0092] By extracting the acoustic features of the first audio sample, the embodiment of the present application can completely retain the rich acoustic information in the first audio sample, facilitate the subsequent cloning of the text, timbre, semantics and intonation in the first audio sample, and ensure the quality of the subsequent synthesized audio; by inputting the acoustic features into the Transformer GPT language model to obtain the first voice code sequence corresponding to the first audio sample, the first voice code sequence corresponding to all the information in the acoustic features can be effectively captured, and the subsequent synthesis of the first fitting audio is convenient; based on the received target audio sample and the first voice code sequence, the first fitting audio corresponding to the target audio sample is synthesized, and the first fitting audio is transmitted to the hearing aid for playback, so that the timbre can be customized, and the target audio sample can be converted into the timbre of a person familiar to the user, meeting the user's demand for customized audio and improving the user experience.

[0093] Embodiment 2

[0094] Please refer to Figure 2 , Figure 2 It is a structural diagram of an embodiment of a mobile terminal provided by the present application, comprising a receiving module 01, an extracting module 02 and a synthesizing module 03;

[0095] The receiving module 01 is used to receive a first audio sample including an acoustic feature, wherein the acoustic feature includes an audio feature and a text feature;

[0096] It is understandable that the first audio sample may be recorded directly through the mobile terminal, or may be recorded in advance by the user through existing technology and uploaded to the mobile terminal.

[0097] The mobile terminal also includes: a selection module; the selection module includes: a judgment unit, a first processing unit and a second processing unit; the judgment unit is used to judge whether a default timbre operation input by a user is received; the first processing unit is used to obtain a second audio sample detected in the hearing aid if the default timbre operation is received, and generate a second fitting audio corresponding to the second audio sample based on the second audio sample and a default voice code sequence, and then transmit the second fitting audio to the hearing aid for playback; the second processing unit is used to receive the first audio sample including acoustic features if the default timbre operation is not received.

[0098] It is understandable that the hearing aid needs to be connected to the hearing aid before the hearing aid can play the audio sample selected by the user, wherein the mobile terminal can establish a connection with the hearing aid through wireless communication technologies such as wifi and Bluetooth. After the mobile terminal is connected to the hearing aid, the user can choose whether to use the default timbre in the mobile terminal. If the mobile terminal receives the user's selection of the default timbre operation, the second audio sample detected in the hearing aid is obtained, and based on the second audio sample and the default voice code sequence, the second fitting audio corresponding to the second audio sample is generated, and the second fitting audio is transmitted to the hearing aid for playback. If the default timbre operation is not received, the first audio sample including the acoustic feature is received; wherein the default timbre operation can be implemented through a button. Exemplarily, when the user clicks the default timbre operation in the mobile terminal, the real-time voice fitting process is entered, and the mobile terminal will combine the second audio sample received from the hearing aid with the default voice code sequence, quickly generate the second fitting audio corresponding to the default timbre, and play it through the hearing aid.

[0099] It should be noted that the hearing aid tone customization method can be installed in scenarios such as applications, web pages and platforms, and the method can be implemented by importing the applications, web pages and platforms equipped with the hearing aid tone customization method into a mobile terminal, wherein the mobile terminal can be a mobile phone, a tablet, a computer, etc., and the present invention does not impose any restrictions on this.

[0100] It should be noted that there may be multiple default voice code sequences, and multiple default voice code sequences will be displayed on the adjustment interface of the mobile terminal. The user can hear audio corresponding to different timbres in the hearing aid by changing the default voice code sequence on the adjustment interface.

[0101] It should be noted that after the second fitting audio is generated, the mobile terminal will also obtain the parameters corresponding to the second fitting audio and display them on the adjustment interface of the mobile terminal. The user adjusts the parameters by adjusting the slider, and can adjust the frequency signal and sound intensity of the second fitting audio to adjust the response of the user's auditory system to different audio signals and improve the auditory experience of people with hearing loss.

[0102] In this way, based on the user's selection of the default tone operation, it can help the user confirm whether customized audio settings are needed, meet the user's needs for customized audio and improve the user experience.

[0103] The extraction module 02 is used to extract the acoustic features corresponding to the first audio sample, and input the acoustic features into the Transformer GPT language model to obtain a first voice code sequence corresponding to the first audio sample;

[0104] The extraction module 02 includes: an extraction unit; the extraction unit is used to extract the acoustic features corresponding to the first audio sample, including an extraction subunit and a quantization subunit; the extraction subunit is used to extract the feature vector of the first audio sample through an audio codec model; the quantization subunit is used to quantize the feature vector step by step to obtain the acoustic features corresponding to the first audio sample. Exemplarily, the high-dimensional first audio sample is compressed into a low-dimensional feature vector by an encoder in the valle codec audio codec model, and then the feature vector is quantized step by step by combining valle codec with RVQ to completely retain all the information of the first audio sample. The present application is not limited to the valle codec audio codec model.

[0105] It should be noted that the Valle codec is based on a neural codec model, which can convert speech signals into discrete coding representations and retain all the information in the audio to achieve high-quality audio synthesis. The Valle codec mainly includes an encoder and a decoder.

[0106] RVQ stands for Residual Vector Quantization, which is a quantization technology used in image and video compression. It can use the residual of the vector for further quantization to convert the vector into a low-dimensional codeword to reduce the amount of data and facilitate storage and transmission.

[0107] In this way, by extracting the acoustic features of the first audio sample through the audio codec model, rich acoustic information such as speaker identity, emotion, environmental noise, etc. in the first audio sample can be completely retained, which facilitates the subsequent cloning of text, timbre, semantics and intonation in the first audio sample, and ensures the quality of subsequent synthesized audio.

[0108] The extraction module 02 further includes: an input unit; the input unit is used to input the acoustic feature into the Transformer GPT language model to obtain a first voice code sequence corresponding to the first audio sample, including a first conversion subunit, a second conversion subunit and a synthesis subunit; the first conversion subunit is used to convert the text feature in the acoustic feature into a first-level voice code sequence through an autoregressive encoder; the second conversion subunit is used to convert the audio feature in the acoustic feature into a second-level voice code sequence through a non-autoregressive encoder; the synthesis subunit is used to fuse the first-level voice code sequence and the second-level voice code sequence to obtain a first voice code sequence corresponding to the first audio sample.

[0109] It can be understood that the acoustic features corresponding to the first audio sample are input into the autoregressive encoder AR to convert the text features in the acoustic features into a first-level vocoder sequence according to conditional context such as a phoneme sequence of a text and an existing audio vocoder sequence, and the acoustic features corresponding to the first audio sample are input into the non-autoregressive encoder NAR to convert the audio features in the acoustic features into a second-level vocoder sequence; wherein the second-level vocoder sequence includes a generated codebook and a vocoder sequence of a speaker's voice, etc.

[0110] It should be noted that the Tranformer GPT language model is a powerful sequence generation model based on the Transformer architecture, consisting of multiple layers of encoders and decoders. Each layer contains a multi-head attention mechanism and a feedforward neural network to capture the long-term dependencies of the input sequence and generate the target sequence. Among them, the Tranformer GPT language model includes an autoregressive encoder AR and a non-autoregressive encoder NAR.

[0111] It should be noted that the training process of the Tranformer GPT language model is specifically as follows: it includes two steps: training data preparation and model training. In the training data preparation, firstly, several audio data sets of predetermined length are collected, wherein the audio data sets include Chinese and English audio. The audio data sets are preprocessed to reduce noise and segment the original audio signals into small audio segments of less than 1 minute, and the speech in the truncated small audio segments is guaranteed to be complete. Then, the ASR model is used to identify the unlabeled data to generate speech transcription text, and the text is normalized, cleaned and standardized, and the noise characters and meaningless punctuation are removed. The text is further divided into words and phonemes for model processing. Finally, the audio samples are encoded and quantized using the pre-trained audio codec model to generate the corresponding discrete sequence matrix. In the model training, the constructed phoneme prompts and acoustic prompts are input into the Transformer decoder model for training to generate the acoustic token of the target audio sample. The generator uses a 12-layer Transformer architecture, each Transformer contains 16 attention heads, the embedding dimension of each attention head is 1024, the intermediate dimension of the feedforward layer is 4096, and the final output layer dimension is 1024. The discriminator uses the WaveGAN model to guide the training of the generator by learning the direct difference between the real speech and the generated speech. The Tranformer GPT language model is trained on 8 NVIDIA TESLAV10040GB GPUs, each GPU processing 8000 acoustic tokens. The AdamW optimizer is used for parameter update. The learning rate is gradually increased to 5×10^-3 in the first 32,000 steps, and then gradually linearly decayed. The training is repeated until 800,000 steps of training are completed. For each training sample, the cross entropy loss between the acoustic token sequence output by the model and the target acoustic token sequence is calculated through the generator, and the gradient is back-propagated. The Tranformer GPT language model is obtained by alternating training of the generator and the discriminator.

[0112] In this way, by inputting the acoustic features into the Transformer GPT language model to obtain the first voice code sequence corresponding to the first audio sample, the first voice code sequence corresponding to all the information in the acoustic features can be effectively captured, which facilitates the synthesis of subsequent fitting audio, meets the user's needs for customized audio, and improves the user experience.

[0113] The synthesis module 03 is used to synthesize a first fitting audio corresponding to the target audio sample based on the received target audio sample and the first voice code sequence, and transmit the first fitting audio to the hearing aid for playback.

[0114] The synthesis module 03 includes: a conversion unit and a synthesis unit; the conversion unit is used to convert the voice code sequence into a speech waveform through a decoder of an audio codec model; the synthesis unit is used to synthesize a first fitting audio corresponding to the target audio sample based on the target audio sample and the speech waveform.

[0115] It can be understood that the decoder of the valve codec can convert the vocoding sequence generated by the valve codec into a speech waveform, and synthesize the first fitting audio corresponding to the target audio sample based on the speech waveform and the target audio sample.

[0116] It should be noted that the target audio samples are audio samples collected by the hearing aid in real time.

[0117] In this way, based on the received target audio sample and the first voice code sequence, the first fitting audio corresponding to the target audio sample is synthesized, speech synthesis can be performed directly based on the discrete voice code sequence, and the target audio sample can be converted into the timbre of a person familiar to the user, thereby meeting the user's demand for customized audio and improving the user experience.

[0118] The mobile terminal further includes a saving module; the saving module includes: a receiving unit, a naming unit and a saving unit; the receiving unit is used to receive a saving timbre operation selected by a user; the naming unit is used to obtain the first voice code sequence corresponding to the first audio sample based on the saving timbre operation, and obtain a user-defined name for the first voice code sequence; the saving unit is used to save the first voice code sequence and the name.

[0119] It is understandable that when the user hears a satisfactory audio sample in the hearing aid, the user can select the save timbre operation in the mobile terminal, and the mobile terminal will obtain the first voice code sequence corresponding to the first audio sample based on the save timbre operation. After receiving the user's custom name for the first voice code sequence, the mobile terminal will save the first voice code sequence and the user-defined name. Among them, the audio saving operation can be implemented by clicking a button of the mobile terminal. Exemplarily, when the user hears a satisfactory audio sample, he will click the save button of the mobile terminal, and then the user can name the voice code sequence corresponding to the audio sample as "friend" or "family", etc., and the mobile terminal will save the voice code sequence corresponding to the audio sample and the user-defined name. The user can directly find the saved voice code sequence of "friend" or "family" in the mobile terminal next time, and by clicking, the voice code sequence of "friend" or "family" can be combined with the target audio sample obtained by the hearing aid to synthesize the first fitting audio corresponding to the target audio sample, and play the target audio sample containing the timbre of "friend" or "family" through the hearing aid.

[0120] In this way, by obtaining the name of the first voice coding sequence corresponding to the first audio sample customized by the user and saving the first voice coding sequence and the name, the user can save a satisfactory first voice coding sequence, which is convenient for using the voice coding sequence to generate other debugging feedback audio in the future, thereby meeting the user's customized audio needs and improving the user experience.

[0121] By extracting the acoustic features of the first audio sample, the embodiment of the present application can completely retain the rich acoustic information in the first audio sample, facilitate the subsequent cloning of the text, timbre, semantics and intonation in the first audio sample, and ensure the quality of the subsequent synthesized audio; by inputting the acoustic features into the Transformer GPT language model to obtain the first voice code sequence corresponding to the first audio sample, the first voice code sequence corresponding to all the information in the acoustic features can be effectively captured, and the subsequent synthesis of the first fitting audio is convenient; based on the received target audio sample and the first voice code sequence, the first fitting audio corresponding to the target audio sample is synthesized, and the first fitting audio is transmitted to the hearing aid for playback, so that the timbre can be customized, and the target audio sample can be converted into the timbre of a person familiar to the user, meeting the user's demand for customized audio and improving the user experience.

[0122] Embodiment 3

[0123] The embodiment of the present application further provides a computer-readable storage medium, which stores instructions executable by a processor. The instructions executable by the processor are used to implement the above-mentioned hearing aid timbre customization method when executed by the processor.

[0124] The contents of the above-mentioned hearing aid tone customization method embodiment are all applicable to the present storage medium embodiment. The functions specifically implemented by the present storage medium embodiment are the same as those of the above-mentioned hearing aid tone customization method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned hearing aid tone customization method embodiment. Therefore, this embodiment can also provide a more personalized and comfortable hearing experience and improve fitting efficiency.

[0125] Those skilled in the art can understand that all or part of the processes in the above embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above embodiments. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0126] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A method for customizing the timbre of a hearing aid. It is characterized in that include: Receiving a first audio sample including acoustic features, wherein the acoustic features include audio features and text features; Extracting the acoustic feature corresponding to the first audio sample, and inputting the acoustic feature into a TransformerGPT language model to obtain a first voice code sequence corresponding to the first audio sample; Based on the received target audio sample and the first voice code sequence, a first fitting audio corresponding to the target audio sample is synthesized, and the first fitting audio is transmitted to the hearing aid for playback.

2. The hearing aid timbre customization method according to claim 1, It is characterized in that Before receiving the first audio sample including the acoustic feature, the method further includes: Determine whether a default timbre operation input by the user is received; If the default timbre operation is received, a second audio sample detected in the hearing aid is obtained, and based on the second audio sample and the default vocode sequence, a second fitting audio corresponding to the second audio sample is generated, and the second fitting audio is transmitted to the hearing aid for playback; If the default timbre operation is not received, the first audio sample including the acoustic feature is received.

3. The hearing aid timbre customization method according to claim 1, It is characterized in that The extracting the acoustic feature corresponding to the first audio sample is specifically: extracting a feature vector of the first audio sample through an audio codec model; The feature vector is quantized step by step to obtain the acoustic feature corresponding to the first audio sample.

4. The hearing aid timbre customization method according to claim 1, It is characterized in that The step of inputting the acoustic features into the Transformer GPT language model to obtain a first sound coding sequence corresponding to the first audio sample is specifically: Converting the text features in the acoustic features into a first-level voice code sequence through an autoregressive encoder; Converting the audio features in the acoustic features into a second-level vocoded sequence by a non-autoregressive encoder; The first-level vocoding sequence and the second-level vocoding sequence are merged to obtain a first vocoding sequence corresponding to the first audio sample.

5. The hearing aid timbre customization method according to claim 1, It is characterized in that The synthesizing the first fitting audio corresponding to the target audio sample based on the received target audio sample and the first voice coding sequence is specifically: converting the vocoded sequence into a speech waveform by a decoder of an audio codec model; Based on the target audio sample and the speech waveform, a first fitting audio corresponding to the target audio sample is synthesized.

6. The hearing aid timbre customization method according to claim 1, It is characterized in that After transmitting the first fitting audio to the hearing aid for playing, the method further includes: Receive a save tone operation selected by the user; Acquire the first voice code sequence corresponding to the first audio sample based on the timbre saving operation, and acquire a user-defined name for the first voice code sequence; The first vocoding sequence and the name are saved.

7. A mobile terminal, It is characterized in that include: Receiving module, extracting module and synthesizing module; The receiving module is used to receive a first audio sample including an acoustic feature, wherein the acoustic feature includes an audio feature and a text feature; The extraction module is used to extract the acoustic features corresponding to the first audio sample, and input the acoustic features into the Transformer GPT language model to obtain a first voice code sequence corresponding to the first audio sample; The synthesis module is used to synthesize a first fitting audio corresponding to the target audio sample based on the received target audio sample and the first voice code sequence, and transmit the first fitting audio to the hearing aid for playback.

8. The mobile terminal according to claim 7, It is characterized in that Also includes: Select the module; The selection module includes: a judgment unit, a first processing unit and a second processing unit; The determination unit is used to determine whether a default timbre operation input by a user is received; The first processing unit is configured to obtain a second audio sample detected in the hearing aid if the default timbre operation is received, generate a second fitting audio corresponding to the second audio sample based on the second audio sample and a default vocode sequence, and then transmit the second fitting audio to the hearing aid for playback; The second processing unit is configured to receive the first audio sample including the acoustic feature if the default timbre operation is not received.

9. The mobile terminal according to claim 7, It is characterized in that The extraction module comprises: an extraction unit; The extraction unit is used to extract the acoustic feature corresponding to the first audio sample, including an extraction subunit and a quantization subunit; The extraction subunit is used to extract the feature vector of the first audio sample through an audio codec model; The quantization subunit is used to quantize the feature vector step by step to obtain the acoustic feature corresponding to the first audio sample.

10. The mobile terminal according to claim 7, It is characterized in that The extraction module further includes: an input unit; The input unit is used to input the acoustic feature into the Transformer GPT language model to obtain a first voice code sequence corresponding to the first audio sample, including a first conversion subunit, a second conversion subunit and a synthesis subunit; The first conversion subunit is used to convert the text features in the acoustic features into a first-level voice code sequence through an autoregressive encoder; The second conversion subunit is used to convert the audio feature in the acoustic feature into a second-level voice code sequence through a non-autoregressive encoder; The synthesis subunit is used to merge the first-level vocoding sequence and the second-level vocoding sequence to obtain a first vocoding sequence corresponding to the first audio sample.

11. The mobile terminal according to claim 7, It is characterized in that The synthesis module comprises: a conversion unit and a synthesis unit; The conversion unit is used to convert the vocoded sequence into a speech waveform through a decoder of an audio codec model; The synthesis unit is used to synthesize a first fitting audio corresponding to the target audio sample based on the target audio sample and the speech waveform.

12. The mobile terminal according to claim 7, It is characterized in that Also includes a save module; The storage module includes: a receiving unit, a naming unit and a storage unit; The receiving unit is used to receive a save timbre operation selected by a user; The naming unit is used to obtain the first voice code sequence corresponding to the first audio sample based on the timbre saving operation, and obtain a user-defined name for the first voice code sequence; The storage unit is used to store the first voice code sequence and the name.

13. A computer-readable storage medium, It is characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the hearing aid timbre customization method according to any one of claims 1 to 6.

Images