Hearing aid and hearing aid system with multi-language automatic conversion function

Abstract

The present invention provides a hearing aid and a hearing aid system with a multi-language automatic conversion function, which belongs to the technical field of hearing aids. The hearing aid integrates a multi-language recognition and conversion module, which can recognize and convert the language types in the voice signal in real time. At the same time, the hearing aid also has a language conversion module under a transparent dialogue link and a language translation module under a telephone link, as well as an intelligent control system to optimize the accuracy and efficiency of language conversion. Through the knowledge distillation method, a translation model that can be deployed on the hearing aid is distilled from the existing end-to-end voice translation model, thereby realizing the multi-language automatic conversion function of the hearing aid. The present invention adopts the above-mentioned hearing aid and hearing aid system with a multi-language automatic conversion function to solve the limitations of existing hearing aids in cross-language communication, so that users can easily cope with the communication challenges in different language environments.

Description

Technical Field

[0001] The present invention relates to the technical field of hearing aids, and in particular to a hearing aid and a hearing aid system with a multi-language automatic conversion function. Background Art

[0002] With the acceleration of globalization, cross-language communication has become an indispensable part of daily life. However, for people with hearing loss, they face more severe challenges. Traditional hearing aids mainly focus on sound amplification to help hearing-impaired people better receive and understand sound information in the surrounding environment. However, these hearing aids have obvious limitations in cross-language communication and cannot meet the communication needs of hearing-impaired people in different language environments.

[0003] Specifically, when hearing-impaired people are in a non-native language environment, they may have difficulty understanding what people around them are saying, leading to communication difficulties. In addition, even if they can understand the other person's language, they may not be able to respond effectively due to the limitations of their own language ability. This limitation not only limits the scope of social activities for hearing-impaired people, but may also have a negative impact on their mental health.

[0004] Therefore, it is particularly important to develop a hearing aid with automatic multi-language conversion function. This hearing aid can recognize the language type in the received voice signal in real time and automatically convert it into a specific language type preset by the user, thereby helping the hearing-impaired to easily cope with the communication challenges in different language environments. By realizing this function, the hearing aid will no longer be just a tool for sound amplification, but an important assistant for the hearing-impaired to communicate effectively in different language environments. Summary of the invention

[0005] The purpose of the present invention is to provide a hearing aid and a hearing aid system with a multi-language automatic conversion function, which solves the limitations of existing hearing aids in cross-language communication and enables users to easily cope with the communication challenges in different language environments.

[0006] To achieve the above object, the present invention provides a hearing aid with a multi-language automatic conversion function, comprising:

[0007] The multi-language recognition and conversion module is integrated in the hearing aid and is used to recognize the language type in the received voice signal in real time and automatically convert it into the specific language type preset by the user;

[0008] The language conversion module under the transparent dialogue link receives voice signals from the surrounding environment through a microphone, performs real-time language conversion using a multi-language recognition and conversion module, and transmits the converted voice signals to the user through headphones;

[0009] The language translation module under the telephone link amplifies the voice of the other party during telephone communication, and translates the user's voice signal into the other party's language in real time, and then sends it to the other party through the telephone link, while receiving and converting the other party's language;

[0010] The intelligent control system is used to control the working status of the multi-language recognition and conversion module, and to process the instructions issued by the user through the control button or the remote terminal APP, and can optimize the accuracy and efficiency of language conversion according to the user's calling habits and language preferences.

[0011] Preferably, the multi-language recognition and conversion module distills a translation model that can be deployed on a hearing aid from an existing end-to-end speech-to-speech translation model through a knowledge distillation method.

[0012] Preferably, the method of knowledge distillation includes:

[0013] S1. Obtain a public speech translation dataset and preprocess it to obtain preprocessed data, and divide the preprocessed data into a training set, a test set, and a validation set;

[0014] S2, select a teacher model and use the training set to train it to obtain a trained teacher model;

[0015] S3, build and train the student model;

[0016] S4, evaluate and optimize the trained student model;

[0017] S5. Deploy the optimized student model to the hearing aid and optimize the inference speed.

[0018] Preferably, in step S1, the preprocessing is to perform short-time Fourier transform on the audio signal in the public data set to extract spectral features.

[0019] Preferably, in step S3, constructing a student model and training it includes the following steps:

[0020] S31. Design student model based on Transformer architecture;

[0021] S32, setting the temperature coefficient for calculating the soft label;

[0022] S33. Perform knowledge distillation, including calculating KL divergence and hard label loss.

[0023] Preferably, in step S33, the calculation steps of KL divergence are as follows:

[0024] Compute soft labels for the teacher model:

[0025] ;

[0026] in, represents the soft label of the teacher model; represents the temperature coefficient; Indicates that the teacher model The output vector of samples has C categories. An index variable representing the category; Represents the teacher model The sample in The original predicted value of each category;

[0027] Compute soft labels for the student model:

[0028] ;

[0029] in, represents the soft label of the student model; Indicates that the student model is The output vector of samples; Represents the student model pair The sample in The original predicted value of each category;

[0030] Calculate the KL divergence for each sample:

[0031] ;

[0032] in, Indicates that the teacher model Samples, j Soft label probability of each category; Indicates that the student model is Samples, j Soft label probability of each category;

[0033] Calculate the average KL divergence:

[0034] ;

[0035] in, represents the number of samples;

[0036] The calculation steps of hard label loss are as follows:

[0037] The student model is compared to The output vector of samples Converted to probability distribution through softmax function:

[0038] ;

[0039] in, Indicates that the student model is The predicted probability distribution of samples;

[0040] Calculate the cross entropy loss for each sample:

[0041] ;

[0042] in, represents the true label; Indicates The true category label vector of the samples j categories; Indicates that the student model is The sample prediction is j The probability of each category;

[0043] Calculate the average cross entropy loss over all samples;

[0044] .

[0045] Preferably, in step S4, the accuracy and BLEU score are used for evaluation, and the temperature parameter is used for optimization;

[0046] The calculation formula of accuracy is as follows:

[0047] ;

[0048] in, Represents the model for i The predicted category of samples; Indicates The true category of samples;

[0049] The calculation formula of BLEU score is as follows:

[0050] ;

[0051] in, BP represents the length penalty term, which is used to penalize candidate translations that are too short; express n-gram of The maximum value of yes n-gram Precision, indicating the accuracy of candidate translations n-gram With reference translation n-gram The matching ratio; represents the weight, = .

[0052] Preferably, step S5, deploying the optimized student model to the hearing aid and optimizing the inference speed,

[0053] ;

[0054] in, represents the predicted output; Represents a given input x When the output is y The posterior probability of Indicates that the The largest value.

[0055] The present invention also provides a hearing aid system with a multi-language automatic conversion function, comprising at least one hearing aid as described above, and a matching remote terminal APP for remotely controlling and setting the functions of the hearing aid.

[0056] Preferably, the remote terminal APP provides a user interface for displaying the current status of the hearing aid and the language conversion record.

[0057] Therefore, the present invention adopts the above-mentioned hearing aid and hearing aid system with multi-language automatic conversion function, and the beneficial technical effects are as follows:

[0058] Compared with traditional hearing aids, the hearing aid and system with multi-language automatic conversion function of the present invention show significant advantages and beneficial technical effects in cross-language communication.

[0059] The present invention realizes real-time recognition and automatic conversion of the language types in the received voice signal by integrating a multi-language recognition and conversion module. This function greatly broadens the communication scope of the hearing-impaired, enabling them to communicate effectively in different language environments.

[0060] The present invention introduces a language conversion module under the transparent dialogue link in the hearing aid, which receives the voice signal in the surrounding environment through the microphone, performs real-time language conversion, and transmits the converted voice signal to the user through the earphone. This function not only improves the understanding ability of the hearing-impaired in face-to-face communication, but also enables them to better integrate into the social environment and enhance interaction with others.

[0061] The present invention also provides a language translation module under the telephone link, which realizes real-time translation of the languages ​​of both parties during telephone communication. This function not only solves the language barrier of hearing-impaired people in telephone communication, but also improves their communication efficiency with international friends and broadens their communication channels.

[0062] The present invention uses an intelligent control system to learn and optimize the user's conversation habits and language preferences, thereby improving the accuracy and efficiency of language conversion. This function enables the hearing aid to better meet the needs of users and provide more personalized services.

[0063] The present invention adopts a knowledge distillation method to distill a translation model suitable for deployment on a hearing aid from an existing model, thereby improving the reasoning speed.

[0064] In summary, the hearing aid and system with multi-language automatic conversion function of the present invention not only solves the limitations of hearing-impaired people in cross-language communication, but also improves their social skills and quality of life, and has important social significance and practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a structural schematic diagram of a hearing aid system with multi-language automatic conversion function according to the present invention;

[0066] Figure 2 Flow chart of knowledge distillation in the present invention. DETAILED DESCRIPTION

[0067] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0068] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0069] Embodiment 1

[0070] like Figure 1 , which is a schematic diagram of the structure of a hearing aid system with a multi-language automatic conversion function according to the present invention, including a hearing aid and a remote terminal APP (which can be a mobile phone, tablet, etc.).

[0071] Hearing aids include:

[0072] The multi-language recognition and conversion module is integrated in the hearing aid and is used to recognize the language type in the received voice signal in real time and automatically convert it into the specific language type preset by the user;

[0073] The language conversion module under the transparent dialogue link receives voice signals from the surrounding environment through a microphone, performs real-time language conversion using a multi-language recognition and conversion module, and transmits the converted voice signals to the user through headphones;

[0074] The language translation module under the telephone link amplifies the voice of the other party during telephone communication, and translates the user's voice signal into the other party's language in real time, and then sends it to the other party through the telephone link, while receiving and converting the other party's language;

[0075] The intelligent control system is used to control the working status of the multi-language recognition and conversion module, and to process the instructions issued by the user through the control button or the remote terminal APP, and can optimize the accuracy and efficiency of language conversion according to the user's calling habits and language preferences.

[0076] The multi-language recognition and conversion module uses the knowledge distillation method to distill a translation model that can be deployed on hearing aids from the existing end-to-end speech-to-speech translation model.

[0077] like Figure 2 As shown, the methods of knowledge distillation include:

[0078] S1. Obtain a public speech translation dataset and preprocess it to obtain preprocessed data, and divide the preprocessed data into a training set, a test set, and a validation set;

[0079] The dataset can be the dataset provided by IWSLT: International Workshop on Spoken Language Translation or TED Talks: TED speech dataset, which is widely used in speech translation tasks.

[0080] After the data set is selected, the data is preprocessed, that is, speech feature extraction, usually using short-time Fourier transform (STFT):

[0081] ;

[0082] in, Indicates that short-time Fourier transform (STFT) is performed on the audio signal to extract its spectral features.

[0083] Assume the audio signal is , the window function is , then the STFT can be expressed as:

[0084] ;

[0085] in, Indicates the time position and frequency The complex spectrum value at ; represents the original audio signal; Indicates the time position Window functions on ; represents the basis functions of Fourier transform; is an imaginary unit; It's time.

[0086] S2. Select a teacher model and use the training set for training to obtain a trained teacher model. The teacher model can be a model such as NAST-S2X.

[0087] S3. Build and train the student model, including the following steps:

[0088] S31. Design student model based on Transformer architecture;

[0089] S32, setting the temperature coefficient for calculating the soft label;

[0090] S33. Perform knowledge distillation, including calculating KL divergence and hard label loss.

[0091] The calculation steps of KL divergence are as follows:

[0092] Compute soft labels for the teacher model:

[0093] ;

[0094] in, represents the soft label of the teacher model; represents the temperature coefficient; Indicates that the teacher model The output vector of samples has C categories. An index variable representing the category; Represents the teacher model The sample in The original predicted value of each category;

[0095] Compute soft labels for the student model:

[0096] ;

[0097] in, represents the soft label of the student model; Indicates that the student model is The output vector of samples; Represents the student model The sample in The original predicted value of each category;

[0098] Calculate the KL divergence for each sample:

[0099] ;

[0100] in, Indicates that the teacher model Samples, j Soft label probability of each category; Indicates that the student model is Samples, j Soft label probability of each category;

[0101] Calculate the average KL divergence:

[0102] ;

[0103] in, represents the number of samples;

[0104] The calculation steps of hard label loss are as follows:

[0105] The student model is compared to The output vector of samples Converted to probability distribution through softmax function:

[0106] ;

[0107] in, Indicates that the student model is The predicted probability distribution of samples;

[0108] Calculate the cross entropy loss for each sample:

[0109] ;

[0110] in, represents the true label; Indicates The true category label vector of the samples j categories; Indicates that the student model is The sample prediction is j The probability of each category;

[0111] Calculate the average cross entropy loss over all samples;

[0112] .

[0113] S4, evaluate and optimize the trained student model;

[0114] Evaluated by accuracy and BLEU score, optimized by temperature parameters;

[0115] The calculation formula of accuracy is as follows:

[0116] ;

[0117] in, Represents the model for The predicted category of samples; Indicates The true category of samples;

[0118] The calculation formula of BLEU score is as follows:

[0119] ;

[0120] in, BP represents the length penalty term, which is used to penalize candidate translations that are too short; express n-gram in The maximum value of yes n-gram Precision, indicating the accuracy of candidate translations n-gram With reference translation n-gram The matching ratio; represents the weight, = .

[0121] S5. Deploy the optimized student model to the hearing aid and optimize the inference speed.

[0122] ;

[0123] in, represents the predicted output; Represents a given input x The output is y The posterior probability of Indicates that the The largest value.

[0124] The present invention also provides a hearing aid system with a multi-language automatic conversion function, comprising at least one hearing aid as described above, and a matching remote terminal APP for remotely controlling and setting the functions of the hearing aid.

[0125] The remote terminal APP provides a user interface for displaying the current status of the hearing aid and language conversion records.

[0126] It is worth noting that the contents not elaborated in detail in the present invention are all prior art and are well known to those skilled in the art.

[0127] Therefore, the present invention adopts the above-mentioned hearing aid and hearing aid system with multi-language automatic conversion function to solve the limitations of existing hearing aids in cross-language communication, allowing users to easily cope with communication challenges in different language environments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A hearing aid with multi-language automatic conversion function, characterized in that: include: The multi-language recognition and conversion module is integrated in the hearing aid and is used to recognize the language type in the received voice signal in real time and automatically convert it into the specific language type preset by the user; The language conversion module under the transparent dialogue link receives voice signals from the surrounding environment through a microphone, performs real-time language conversion using a multi-language recognition and conversion module, and transmits the converted voice signals to the user through headphones; The language translation module under the telephone link amplifies the voice of the other party during telephone communication, and translates the user's voice signal into the other party's language in real time, and then sends it to the other party through the telephone link, while receiving and converting the other party's language; An intelligent control system for the working status of the multi-language recognition and conversion module, and for processing commands issued by the user through the control button or the remote terminal APP, and for optimizing the accuracy and efficiency of language conversion according to the user's calling habits and language preferences; The multi-language recognition and conversion module uses knowledge distillation to distill a translation model that can be deployed on hearing aids from the existing end-to-end speech-to-speech translation model. Methods for knowledge distillation include: S1. Obtain a public speech translation dataset and preprocess it to obtain preprocessed data, and divide the preprocessed data into a training set, a test set, and a validation set; S2, select a teacher model and use the training set to train it to obtain a trained teacher model; S3, build and train the student model; S4, evaluate and optimize the trained student model; S5. Deploy the optimized student model to the hearing aid and optimize the inference speed.

2. A hearing aid with multi-language automatic conversion function according to claim 1, characterized in that: In step S1, the preprocessing is to perform short-time Fourier transform on the audio signal in the public data set to extract spectrum features.

3. A hearing aid with multi-language automatic conversion function according to claim 2, characterized in that: In step S3, the student model is constructed and trained, including the following steps: S31. Design student model based on Transformer architecture; S32, setting the temperature coefficient for calculating the soft label; S33. Perform knowledge distillation, including calculating KL divergence and hard label loss.

4. A hearing aid with multi-language automatic conversion function according to claim 3, characterized in that: In step S33, the calculation steps of KL divergence are as follows: Compute soft labels for the teacher model: ； in, represents the soft label of the teacher model; represents the temperature coefficient; Indicates that the teacher model The output vector of samples has C categories. An index variable representing the category; Indicates that the teacher model The sample in The original predicted value of each category; Compute soft labels for the student model: ； in, represents the soft label of the student model; Indicates that the student model is The output vector of samples; Indicates that the student model is The sample in The original predicted value of each category; Calculate the KL divergence for each sample: ； in, Indicates that the teacher model Samples, Soft label probability of each category; Indicates that the student model is Samples, Soft label probability of each category; Calculate the average KL divergence: ； in, represents the number of samples; The calculation steps of hard label loss are as follows: The student model is compared to The output vector of samples Converted to probability distribution through softmax function: =softmax( ); in, Indicates that the student model is The predicted probability distribution of samples; Calculate the cross entropy loss for each sample: ； in, represents the true label; Indicates The true category label vector of the samples categories; Indicates that the student model is The sample prediction is The probability of each category; Calculate the average cross entropy loss over all samples; 。 5. A hearing aid with multi-language automatic conversion function according to claim 4, characterized in that: In step S4, the accuracy and BLEU scores are used for evaluation and the temperature parameters are used for optimization; The calculation formula of accuracy is as follows: ； in, Represents the model for The predicted category of samples; Indicates The true category of samples; The calculation formula of BLEU score is as follows: ； in, BP represents the length penalty term, which is used to penalize candidate translations that are too short; express n-gram of The maximum value of yes n-gram Precision, indicating the accuracy of candidate translations n-gram With reference translation n-gram The matching ratio; represents the weight, = .

6. A hearing aid system with multi-language automatic conversion function, characterized in that: It comprises at least one hearing aid as described in any one of claims 1 to 5, and a remote terminal APP matched therewith, for remotely controlling and setting the functions of the hearing aid.

7. The hearing aid system with multi-language automatic conversion function according to claim 6, characterized in that: The remote terminal APP provides a user interface for displaying the current status of the hearing aid and the language conversion record.

Images