A multi-lingual artificial intelligence hearing aid fitting model construction and fitting method

Abstract

This invention discloses a multilingual AI-based hearing aid fitting model construction and fitting method. The model construction method includes: pre-training a hearing aid fitting model using a full set of multilingual samples; treating hearing aid fitting in different language families as different meta-learning tasks, and using samples from each language family to perform meta-learning training on the pre-trained model to obtain a parent model with new language family learning capabilities; embedding LoRA adapters into the feature fusion module and each expert network of the parent model; and performing the following operations for each language family to be fitted: fine-tuning each LoRA adapter and gating network using samples from the current language family; and using the fine-tuned parameters and other parts of the parent model except for the gating network to jointly constitute a hearing aid fitting model suitable for the current language family. This embodiment can consider the influence of language family in hearing aid fitting, improving fitting accuracy.

Description

Technical Field

[0001] The embodiments of the present invention relate to the fields of artificial intelligence and hearing aid technology, and in particular to a method for constructing and fitting a multilingual artificial intelligence hearing aid fitting model. Background Technology

[0002] Hearing aid fitting refers to adjusting parameters such as the inflection point, gain, and compression ratio of the hearing aid in different frequency bands according to the user's hearing loss, in order to adjust the frequency response curve for different input sound intensities. At the same time, it is supplemented by adjusting advanced functional parameters such as noise reduction and feedback suppression to meet the user's personalized hearing needs.

[0003] Traditional hearing aid fitting relies primarily on the personal experience and clinical accumulation of audiologists. However, with the application of artificial intelligence in hearing intervention and rehabilitation, methods have emerged that use machine learning techniques to train neural network models to predict a user's hearing aid parameter set. However, existing machine learning techniques typically use a general model and only input the user's gender, age, type and degree of hearing loss for model training and prediction, resulting in the following technical limitations: Insufficient adaptability to different dialect groups: my country is a multi-dialect country, including Mandarin, Wu, Cantonese, Min, Xiang, Gan, Hakka, and other dialect regions. Different dialects have multi-dimensional acoustic differences, but current fitting methods treat users from different dialect groups as the same group, ignoring the personalized needs of users with the same degree of hearing loss for hearing aid fitting parameters due to differences in dialect groups, thus limiting the accuracy of fitting.

[0004] The distribution of fitting data across different dialect groups is uneven: In multi-dialect application scenarios, data is abundant for mainstream languages ​​(such as Mandarin and Northern Mandarin), while data for less commonly spoken languages ​​(such as minority languages) is scarce. Traditional deep learning methods struggle to quickly adapt to less commonly spoken languages, leading to poor feedback from hearing aid users.

[0005] Insufficient adaptability to new language families: After initial training, the inherent potential of existing models to adapt to new language families is essentially locked. Each time a new language family is encountered, it is almost necessary to start from scratch, including rebuilding a dataset containing data from all language families and retraining the model, resulting in high R&D costs. Summary of the Invention

[0006] This invention provides a method for constructing and fitting a multilingual artificial intelligence hearing aid fitting model, in order to solve at least one of the above-mentioned technical problems.

[0007] In a first aspect, embodiments of the present invention provide a method for constructing a multilingual artificial intelligence hearing aid fitting model, comprising: Obtain the training set and the hearing aid fitting model to be trained; wherein, the input of each sample in the training set includes the user's basic information, language family, and acoustic features of the language family, and the output of each sample includes the user's hearing aid fitting parameters and frequency response curve; the hearing aid fitting model includes a feature interaction module, a feature fusion module, a gating network, multiple expert networks, and an output module; The hearing aid fitting model was pre-trained using multilingual samples to learn the common audiological compensation rules for each language family. Hearing aid fitting in different language families is used as a different meta-learning task. The pre-trained model is trained using samples from each language family to obtain a parent model with the ability to learn new language families. LoRA adapters are embedded in the feature fusion module of the parent model and each expert network, respectively. For each language family to be fitted, the following operations are performed: the LoRA adapters and gating networks are fine-tuned using samples from the current language family to adapt to the characteristics of the current language family and learn to assign expert networks differently for different language families; the fine-tuned parameters and other parts of the parent model except for the gating network together constitute a hearing aid fitting model suitable for the current language family; wherein, each language family to be fitted can include language families outside the training set.

[0008] Secondly, embodiments of the present invention provide a multilingual artificial intelligence hearing aid fitting method, including: Obtain multiple inputs from new users, including: basic information, language family, and acoustic features of the language family; Load a hearing aid fitting model applicable to the language family. The hearing aid fitting model includes a feature interaction module, a feature fusion module and its LoRA adapter, a gating network, multiple expert networks and their respective LoRA adapters, and an output module. Each input is preprocessed into a feature vector, and then input into the hearing aid fitting model applicable to the language family to predict the hearing aid fitting parameters and frequency response curve of the new user.

[0009] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the multilingual artificial intelligence hearing aid fitting method described in any embodiment.

[0010] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multilingual artificial intelligence hearing aid fitting method described in any embodiment.

[0011] In summary, this invention provides a method for constructing a multilingual AI-powered hearing aid fitting model. By integrating hybrid expert models, language-aware feature engineering, multi-task learning, LoRA fine-tuning, and meta-learning techniques, it addresses the problems of insufficient language adaptability, uneven data distribution, limited model generalization ability, and insufficient fitting prediction accuracy in existing technologies, achieving high-precision and personalized hearing aid parameter prediction. Compared with existing technologies, this implementation achieves the following beneficial effects: 1. Achieves accurate language family adaptation: By introducing acoustic features and language family embedding, and combining the structural advantages of the MOE model, the model can deeply understand and adapt to the acoustic characteristics of different language families, greatly improving the accuracy of hearing aid parameter prediction.

[0012] 2. Efficient data utilization: Through meta-learning, a master model with the ability to quickly adapt to new language families is obtained. Based on the master model, partial feature fusion modules and expert networks are used to fine-tune the model for specific language families, enabling the model to perform well even on language families with limited data, effectively solving the problems of data imbalance and cold start.

[0013] 3. Improves model robustness: The introduction of multi-task learning (hearing aid fitting parameter learning task + frequency response curve learning task) as a regularization method makes the feature representation learned by the model more generalizable, avoids overfitting, and enhances the robustness of the model.

[0014] 4. A novel model training mechanism is provided, balancing model performance and development efficiency: First, pre-training is performed using multilingual samples, enabling the model to learn general audiological compensation rules and reflect the general rules of language family features in hearing aid fitting, providing a robust foundation for meta-learning generalization. Then, to accommodate new and rare language families lacking sample accumulation, different language families are treated as different meta-learning tasks. A master model is constructed based on the pre-training results using meta-learning methods, enabling the master model to learn quickly for all language families. Finally, the master model is fine-tuned using a small number of samples from the language family to be fitted in practical applications, quickly obtaining a hearing aid fitting model for a specific language family, improving fitting accuracy. During the model fine-tuning stage, considering the significant time required for full fine-tuning, this embodiment introduces LoRA for lightweight fine-tuning in the feature fusion module and expert network. However, due to the limited effectiveness of LoRA adjustment, this embodiment performs full fine-tuning of the MoE gating network, providing ample customization space for different language families, while also enabling the router to assign experts to different language families to balance the utilization of the expert network. The entire model training mechanism described above works together to improve the matching accuracy of multilingual models, enabling the models to fully consider language family characteristics and be applicable to small sample language families and new language families.

[0015] 5. Provides an end-to-end personalized solution: Offers a complete, automated solution from data preparation to final prediction, enabling users to receive truly personalized hearing aid fitting services.

[0016] 6. Provides an efficient parameter deployment solution: In actual products, there is no need to store multiple huge model files. Only one master model and multiple KB-level LoRA configuration files and gated network configuration files need to be stored, which greatly reduces storage pressure. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a method for constructing a multilingual artificial intelligence hearing aid fitting model according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a multilingual artificial intelligence hearing aid fitting model provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the hearing aid fitting parameters output by a multilingual artificial intelligence hearing aid fitting model provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the frequency response curve of the output of a multilingual artificial intelligence hearing aid fitting model provided in an embodiment of the present invention; Figure 5 This is a flowchart of a method for constructing a multilingual artificial intelligence hearing aid fitting model provided in an embodiment of the present invention; Figure 6 This is a flowchart of a multilingual artificial intelligence hearing aid fitting method provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0020] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0021] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0022] Figure 1 This is a flowchart illustrating a method for constructing a multilingual AI-powered hearing aid fitting model, as provided in an embodiment of the present invention. This method is particularly suitable for multilingual application scenarios and is executed by an electronic device. Figure 1 As shown, the method specifically includes: S110. Obtain the training set; wherein, the input of each sample in the training set includes the user's basic information, language family, and acoustic characteristics of the language family, and the output of each sample includes the user's hearing aid fitting parameters and frequency response curve.

[0023] This step prepares the training dataset for training the AI ​​model for hearing aid fitting. Optionally, it can collect information such as the user's gender, age, hearing aid wearing history, hearing impairment status, language family labels, as well as the final hearing aid fitting parameters and the user's frequency response curve. The hearing aid fitting parameters may include loud gain, soft gain, compression inflection point, maximum output limit, etc., and the frequency response curve is the hearing aid output loudness curve corresponding to different frequencies (0-8000Hz).

[0024] Furthermore, the collected user data covers multiple language families, including different dialect families. For example, my country is a country with multiple dialect families, including Mandarin, Wu, Cantonese, Min, Xiang, Gan, and Hakka. It can also include different language families, such as Chinese, English, and Korean. Different language families exhibit multi-dimensional differences in acoustics, including tonal fundamental frequency characteristics, segmental phonological timing, formant structure, prosodic rhythm, and vocal quality. This embodiment extracts representative acoustic statistical features (such as the number of vowels, average fundamental frequency, and the average and variance of formant F1 / F2) for each language family to characterize its features. Optionally, multiple representative sound signals from the same language family can be collected, and the fundamental frequency and formants of each sound signal can be extracted. The fundamental frequency of the multiple sound signals can be averaged, and the mean and variance of the formants F1 / F2 of the multiple sound signals can be calculated. The number of vowels, the average fundamental frequency, and the mean and variance of the formants F1 / F2 are used as the acoustic features of the same language family.

[0025] After data collection is complete, all data for each user will be used as a sample. The user's basic information (gender, age, hearing aid wearing history, hearing impairment), language family label, and acoustic characteristics of that language family will be used as model input in subsequent training. The user's hearing aid fitting parameters and frequency response curve will be used as model output in subsequent training.

[0026] S120. Obtain the hearing aid fitting model to be trained.

[0027] This embodiment will utilize artificial intelligence methods to achieve automatic hearing aid fitting and construct a system such as... Figure 2 The model structure shown includes a feature interaction module, a feature fusion module, a gating network, multiple expert networks, and an output module. The following example, using the processing flow of a single sample, illustrates the function of each module.

[0028] In one specific implementation, when processing a sample, the inputs in the sample are first preprocessed into feature vectors. Combined with... Figure 2 User basic information includes age, gender, and audiogram. Age and gender features can be embedded to obtain feature vectors, while the audiogram can be generated as a feature vector using an LSTM network or MLP. All feature vectors from the user information together constitute the user features. Simultaneously, the user's language label is transformed into a low-dimensional dense vector (language embedding) through an embedding layer. User features, language embeddings, and language acoustic features are all feature vectors.

[0029] Then, all feature vectors are input to the feature interaction module for feature interaction. Optionally, the feature vectors can be concatenated and used as input to the feature interaction module, which can employ an MLP-Mixer structure to reduce interference between local features. In contrast, other feature extraction structures, such as CNNs, focus on extracting local features of the data. For example, in image processing, they pay more attention to local information between pixels (such as texture information, spatial information, etc.). The MLP-Mixer structure weakens the positional correlation between local data, avoiding misidentification of locally close data as texture or spatial features, thus preventing unnecessary interference. This is particularly suitable for situations where the local positions between input features are not particularly meaningful in this application.

[0030] The feature interaction module is used to implement preliminary feature interaction; in this embodiment, the output of this module is referred to as the interactive feature. The interactive feature is input to the feature fusion module for further deep feature fusion. Optionally, the feature fusion module can adopt a Transformer structure to enhance the interaction between global features. Here, the Transformer is compared with the commonly used CNN structure. The CNN structure focuses on the fusion of local near-field features, while the Transformer structure realizes the interactive computation between global features, which can more fully explore the complex relationships between the input features of this application.

[0031] The output of the feature fusion module can be called the fused feature, which is simultaneously input to each expert network and the gating network. Each expert network further processes the fused feature to obtain new fused features. The gating network then outputs the weights of each expert network based on the fused feature. Optionally, both the gating network and each expert network can adopt an MLP structure. The output of the gating network can be expressed as: (1) in, Indicates input The module's feature concatenation vector (numerical features, categorical features, acoustic features, language embedding). and These represent the weights and bias parameters of the gating network, respectively. , These represent the weight allocations of the N expert networks.

[0032] Based on the weights of each expert network, the outputs of each expert network can be analyzed. Perform weighted fusion. Features after weighted fusion. It has stronger generalization ability and can be represented as follows: (2) Finally, the weighted fused features are input into the output module to predict the hearing aid fitting parameters and frequency response curve of the current sample. The final output of the model is as follows: Figure 3 and Figure 4 As shown. Optionally, the output module can adopt two MLP structures. One MLP is used to output hearing aid fitting parameters, and the number of output nodes of the MLP is equal to the number of fitting parameters. The other MLP is used to output frequency response curves, and the number of output nodes of the MLP is equal to the number of horizontal axis values ​​in the frequency response curves. The value of each node is equal to the curve value corresponding to each horizontal axis value.

[0033] S130. The hearing aid fitting model is pre-trained using multilingual samples to learn the common audiological compensation rules for each language family.

[0034] This step trains the hybrid expert model using all language samples in the training set (also known as full multilingual samples). The aim is to enable the model to learn general audiological compensation rules and obtain feature representations with strong generalization ability. For ease of distinction and description, this embodiment refers to this stage of training as pre-training. Optionally, during pre-training, the difference between the true and predicted values ​​of hearing aid fitting parameters and frequency response curves can be minimized, and the model parameters can be updated using equalization among expert networks. Equalization refers to avoiding assigning the same expert network to different language families. In one specific implementation, the following loss function can be used: (3) in, Represents the total loss function. This represents the MSE loss between the model-predicted hearing aid fitting parameters and the actual hearing aid fitting parameters. This represents the MSE loss between the model-predicted frequency response curve and the actual frequency response curve. This indicates load balancing auxiliary losses. These represent the weight hyperparameters of each loss function, used to control the importance of each loss term.

[0035] (4) in, Indicates the first The frequency with which an expert network is selected in a batch can be considered as the frequency of selection. We can assume that the top K expert networks with the largest weights in each prediction are the selected expert networks, thus obtaining the i-th... The frequency with which an expert network is selected in the same batch of samples; Indicates the first The batch average of the gating probabilities of the expert network, optionally for the . A network of experts, each sample in the same batch Each corresponds to a gating weight. For each sample in the same batch Summing, we can obtain the first... The cumulative sum corresponding to each expert network (in, For sample index, (for batch size), using Dividing by the batch size can achieve normalization, resulting in... ,Right now . This is used to avoid assigning different language families to the same expert network and to maintain a balanced utilization of each expert network.

[0036] S140. Treat hearing aid fitting in different language families as different meta-learning tasks, and use samples from each language family to perform meta-learning training on the pre-trained model to obtain a mother model with the ability to learn new language families.

[0037] This step completes the construction of the meta-learning-based master model. Building upon the pre-trained model, it endows the model with the ability to "learn," enabling it to acquire a set of initial parameters that are highly sensitive to new tasks, thus solving the cold start problem. Optionally, this process may include the following steps: S10. Set up two model parameter sets (referred to as the first model parameter set and the second model parameter set respectively) to store the model parameters in the subsequent two nested loops; and use the parameters of the pre-trained model as the initial values ​​of the first model parameter set.

[0038] S20. Start the outer loop, and perform the following operation in each iteration: S201. Assign the values ​​from the first model parameter set to the second model parameter set; S202. Start the inner loop. In each iteration, load the values ​​from the second model parameter set into the hearing aid fitting model, select an unselected language family, update the parameters of the hearing aid fitting model using samples from that language family, and then overwrite the original values ​​in the second model parameter set with the updated parameters. Then return to S202 to enter the next cycle of the inner loop. This process repeats continuously, updating the values ​​in the second model parameter set until all language families have been traversed.

[0039] S203. Load the latest values ​​from the second model parameter set into the hearing aid fitting model, and update the values ​​in the first model parameter set using the updated model loss function value. Then return to S201 and enter the next cycle of the outer loop. Repeat this process until the set termination condition is met (such as reaching the set number of cycle cycles, or the first model parameter set remaining unchanged).

[0040] This approach allows for the discovery of initial model parameters for subsequent training. These initial parameters are not optimal for a specific task, but rather possess a "potential"—based on this initial point, the model can quickly achieve excellent performance with only a small number of gradient update steps on a very small amount of new task data. The model trained through meta-learning can rapidly adapt to the characteristics of new language families, whether they are already present in the training set or not.

[0041] In one specific implementation, hearing aid fitting in different language families can be treated as different tasks in meta-learning, and samples from different language family tasks can be divided into the same dataset. .in, This indicates different language family tasks. This indicates the number of language family tasks in the training set. Each language family task's dataset includes a support set and a query set, i.e. Among them, support set Query set , and These represent the number of samples in the support set and the query set, respectively. These represent the input and output of the sample, respectively.

[0042] Then, two nested loops can be set up, with two parameter sets to store the model parameters for each loop. In the outer loop, the parameters of the pre-trained model are first used as the model's initialization parameters, and then the inner loop is started. In the inner loop, for each language family task, the model is updated using a few steps or a single step of fast gradient updates based on the support set: (5) in, This represents the learning rate of the inner loop. These represent the model's initialization parameters. It is the loss of the support set. This represents the updated model parameters.

[0043] After the inner loop finishes, it returns to the outer loop. In the outer loop, it optimizes the initial parameters of the model across tasks, minimizes the loss on the query set, and updates the initial weights of the model so that the adapted parameters perform well on the query set. (6) (7) After the outer loop completes, the final meta-learning mother model is obtained. This model is not optimal for a specific language family, but it has strong plasticity and can provide good initialization parameters for subsequent training.

[0044] S150. LoRA adapters are embedded in the feature fusion module of the parent model and each expert network, respectively. The following operations are performed for each language family to be fitted: the LoRA adapters and gating networks are fine-tuned using samples from the current language family to adapt to the characteristics of the current language family and learn to assign expert networks differently for different language families. The fine-tuned parameters and other parts of the parent model except the gating network together constitute a hearing aid fitting model suitable for the current language family. The language family to be fitted can include language families outside the training set.

[0045] This step performs LoRA fine-tuning on the model based on the meta-learning master model, generating customized parameters for specific language families (especially those with scarce data) based on the capabilities of the master model. Optionally, since LoRA fine-tuning is based on low-rank factorization matrices, this embodiment selects to embed (or inject) LoRA adapters into the linear layers of the feature fusion module of the master model and the linear layers of each expert network.

[0046] In one specific implementation, most parameters of the parent model are first frozen (including most layers of the MLP-mixer and Transformer), and then low-rank decomposition matrices are injected into the QKV projection layer (linear layer) of the Transformer and the linear layer of the MoE expert network. (8) in, This represents the pre-trained weight matrix of the linear layer. This represents the fine-tuning matrix to be injected. and They represent The low-rank decomposition matrix, , , It is a low-rank dimension.

[0047] LoRA fine-tuning is a lightweight approach that saves computational and time costs, but its adjustment effect is limited. To provide sufficient customization space for different language families while balancing the utilization of the expert network, this embodiment, in addition to the aforementioned LoRA fine-tuning, also performs full fine-tuning of the MoE's gated network, as the router must learn to assign experts to different language families. After fine-tuning for each language family is completed, a small set of LoRA weight files and gated network weight files are stored for each language family. In subsequent use, the corresponding weight files are loaded according to different language families to form hearing aid fitting models for different language families.

[0048] It's worth noting that the master model trained through meta-learning also possesses a strong learning ability for language families not covered in the initial training set. Therefore, the fine-tuning stage can be performed on language families already covered in the training set, or on language families not covered in the training set, both of which can yield well-customized models with a small number of samples. The entire method described above can also be combined with... Figure 4 I understand.

[0049] Furthermore, key code snippets from an application example are provided below to illustrate each step more fully. This example divides the entire method into four stages: Phase 1: Core model architecture definition (corresponding to S120); Phase 2: Multi-task pre-training (corresponding to S130, where multi-task refers to two tasks: predicting hearing aid fitting parameters and predicting frequency response curves) Phase 3: Meta-learning (corresponding to S140) Phase 4: LoRA fine-tuning (corresponding to S150) # Phase 1: Core Model Architecture Definition # ========================================== classHearingAidMoE(nn.Module): def__init__(self, nLanguage): self.mlp_mixer=MlpMixerBlock() # Freeze and exclude from LoRA self.Transformer = TransformerEncoder() # Inject LoRA (Q, V matrix) self.language_emb=nn.Embedding(nLanguage) self.audio_feat_emb=AudioFeatEmb() self.user_feat_emb=UserFeatEmb() self.router=MLP_Gate() # Full fine-tuning self.experts=nn.ModuleList([MLP_Expert()for_inrange(8)]) # Inject LoRA # LoRA Configuration: Enabled only in Step 4; Steps 2 and 3 are standard fully connected. self.heads={ 'main': ParameterPredictor(), 'aux': CurvePredictor() } defforward(self, x): # 1. Feature Preprocessing x=FeatPreProcess( self.language_emb, self.audio_feat_emb, self.user_feat_emb, x ) feat=self.mlp_mixer(x) # 2. Feature Interaction feat=self.mlp_mixer(x) # 3. Feature Fusion (Transformer) feat_trans=self.Transformer(feat) # 4. MoE Routing and Calculation router_logits=self.router(feat_trans) weights=softmax(router_logits) # Top-k route weights expert_outputs=sum([w*expert(feat_trans) forw, expertinzip(weights, self.experts)]) # 5. Multi-task output params=self.heads['main'](expert_outputs) curve=self.heads['aux'](expert_outputs) returnparams, curve # ========================================== # Phase 2: Multi-task Pre-training # ========================================== defstep2_pretraining(model, all_data_loader): optimizer=Adam(model.parameters(), lr=1e-4) forbatchinall_data_loader: # Input: Audio, user information, language ID # Tags: Real experimental parameters (y_param), Ideal frequency response curve (y_curve) pred_param, pred_curve=model(batch.x) # Multi-task loss function loss_main=MSE(pred_param, batch.y_param) loss_aux=MSE(pred_curve, batch.y_curve) total_loss = loss_main + 0.5 * loss_aux # Auxiliary task weight 0.5 total_loss.backward() optimizer.step() returnmodel # Get the pre-trained base model # ========================================== # Phase 3: Meta-Learning (MAML) # ========================================== defstep3_meta_learning(pretrained_model, task_distribution): # Initialize the metamodel parameter theta meta_model=copy.deepcopy(pretrained_model) meta_optimizer=Adam(meta_model.parameters(), lr=1e-3) foriterationinrange(MAML_STEPS): meta_loss=0 # Sample a batch of tasks (different dialects), for example, Task_i = Cantonese, Task_j = Wu dialect tasks=task_distribution.sample_tasks(batch_size=4) fortaskintasks: # 1. Inner Loop (Support Set): Simulates rapid adaptation theta_prime=meta_model.clone() support_loss=compute_loss(theta_prime, task.support_set) # Calculate the gradient and temporarily update the parameters (Fast weights) grads=torch.autograd.grad(support_loss, theta_prime.parameters()) theta_prime=theta_prime-inner_lr*grads # 2. Outer Loop (Query Set): Evaluate the effects of adaptation. query_loss=compute_loss(theta_prime, task.query_set) meta_loss += query_loss # 3. Meta-update: Update the initial parameter theta to facilitate faster adaptation later. meta_loss / = len(tasks) meta_loss.backward() meta_optimizer.step() returnmeta_model # Obtain the "meta-learning mother model" # ========================================== # Phase 4: LoRA Fine-tuning (LoRA Adaptation) # ========================================== defstep4_LoRA_tuning(meta_model, target_dialect_data): # 1. Architectural Adjustment # Freeze all parameters forparaminmeta_model.parameters(): param.requires_grad=False # Injecting the LoRA adapter (Transformer Attn + Expert Linear) meta_model=inject_LoRA_layers(meta_model, rank=8) # Unfreeze the gated network (Router) forparaminmeta_model.router.parameters(): param.requires_grad=True # 2. Fine-tuning for specific language families # At this point, only the LoRA matrix (A, B) and Router parameters are trained; the backbone parameters remain in the state of Step 3. optimizer=Adam(filter(lambdap: p.requires_grad, meta_model.parameters()), lr=1e-3) forbatchintarget_dialect_data: preds, _=meta_model(batch.x) loss = MSE(preds, batch.y) loss.backward() optimizer.step() # When saving, only save the LoRA weights and Router weights. returnLoRA_weights, router_weights In summary, this embodiment provides a method for constructing a multilingual AI-powered hearing aid fitting model. By integrating hybrid expert models, language-aware feature engineering, multi-task learning, LoRA fine-tuning, and meta-learning techniques, it addresses the problems of insufficient language adaptability, uneven data distribution, limited model generalization ability, and insufficient fitting prediction accuracy in existing technologies, achieving high-precision and personalized hearing aid parameter prediction. Compared with existing technologies, this embodiment can achieve the following beneficial effects: 1. Achieves accurate language family adaptation: By introducing acoustic features and language family embedding, and combining the structural advantages of the MOE model, the model can deeply understand and adapt to the acoustic characteristics of different language families, greatly improving the accuracy of hearing aid parameter prediction.

[0050] 2. Efficient data utilization: A parent model with new language family capabilities is obtained through meta-learning. Based on the parent model, feature fusion modules and expert networks are used to fine-tune the model for specific language families, enabling the model to perform well even on language families with limited data, effectively solving the problems of data imbalance and cold start.

[0051] 3. Improves model robustness: Introducing multi-task learning (hearing aid fitting parameter learning task + frequency response curve learning task) as a regularization method. The frequency response curve and hearing aid fitting parameters have a strong correlation. By allowing the model to learn both hearing aid fitting parameters and frequency response curves simultaneously, the model can leverage the commonalities and complementarities between the frequency response curve and hearing aid fitting parameters to improve overall performance. This makes the feature representations learned by the model more generalizable, avoids overfitting, and enhances the robustness of the model.

[0052] 4. A novel model training mechanism is provided, balancing model performance and development efficiency: First, pre-training is performed using multilingual samples, enabling the model to learn general audiological compensation rules and reflect the general rules of language family features in hearing aid fitting, providing a robust foundation for meta-learning generalization. Then, to accommodate new and rare language families lacking sample accumulation, different language families are treated as different meta-learning tasks. A master model is constructed based on the pre-training results using meta-learning methods, enabling the master model to learn quickly for all language families. Finally, the master model is fine-tuned using a small number of samples from the language family to be fitted in practical applications, quickly obtaining a hearing aid fitting model for a specific language family, improving fitting accuracy. During the model fine-tuning stage, considering the significant time required for full fine-tuning, this embodiment introduces LoRA for lightweight fine-tuning in the feature fusion module and expert network. However, due to the limited effectiveness of LoRA adjustment, this embodiment performs full fine-tuning of the MoE gating network, providing ample customization space for different language families, while also enabling the router to assign experts to different language families to balance the utilization of the expert network. The entire model training mechanism described above works together to improve the matching accuracy of multilingual models, enabling the models to fully consider language family characteristics and be applicable to small sample language families and new language families.

[0053] 5. Provides an end-to-end personalized solution: Offers a complete, automated solution from data preparation to final prediction, enabling users to receive truly personalized hearing aid fitting services.

[0054] 6. Provides an efficient parameter deployment solution: In actual products, there is no need to store multiple huge model files. Only one master model and multiple KB-level LoRA configuration files and gated network configuration files need to be stored, which greatly reduces storage pressure.

[0055] Figure 6 This is a flowchart of a multilingual artificial intelligence hearing aid fitting method provided by an embodiment of the present invention, which specifically includes the following steps: S210. Obtain multiple inputs from the new user, wherein the multiple inputs include: basic information, language family, and acoustic features of the language family.

[0056] S220. Load the hearing aid fitting model applicable to the language family. The hearing aid fitting model includes a feature interaction module, a feature fusion module and its LoRA adapter, a gating network, multiple expert networks and their respective LoRA adapters, and an output module.

[0057] The hearing aid fitting model here is the hearing aid fitting model obtained by the construction method in any of the above embodiments, and it is loaded with LoRA weights and Router weights corresponding to the language family of the new user.

[0058] S230. Each input is preprocessed into a feature vector, and then input into the hearing aid fitting model applicable to the language family to predict the hearing aid fitting parameters and frequency response curve of the new user.

[0059] Each input is processed sequentially through the modules in the model according to the steps in S120, and finally the hearing aid fitting parameters and frequency response curve of the new user are obtained. In the processing of the feature fusion module and the expert network, the module processing results after LoRA fine-tuning are used.

[0060] This embodiment is based on the same inventive concept as the above model construction embodiment. Any limitations in the above model construction embodiment are applicable to this embodiment and can achieve the same technical effect.

[0061] It should be noted that all user data involved in this application is information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0062] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 7 As shown, the device includes a processor 60, a memory 61, an input device 62, and an output device 63; the number of processors 60 in the device can be one or more. Figure 7 Taking a processor 60 as an example; the processor 60, memory 61, input device 62, and output device 63 in the device can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0063] The memory 61, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the multilingual artificial intelligence hearing aid fitting method in this embodiment of the invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 61, thereby realizing the aforementioned multilingual artificial intelligence hearing aid fitting method.

[0064] The memory 61 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 61 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include memory remotely located relative to the processor 60, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0065] Input device 62 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 63 may include display devices such as a display screen.

[0066] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multilingual artificial intelligence hearing aid fitting method of any embodiment.

[0067] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0068] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0069] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0070] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as C or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a multilingual artificial intelligence hearing aid fitting model, characterized in that, include: Obtain the training set and the hearing aid fitting model to be trained; wherein, the input of each sample in the training set includes the user's basic information, language family, and acoustic features of the language family, and the output of each sample includes the user's hearing aid fitting parameters and frequency response curve; the hearing aid fitting model includes a feature interaction module, a feature fusion module, a gating network, multiple expert networks, and an output module; The hearing aid fitting model was pre-trained using multilingual samples to learn the common audiological compensation rules for each language family. Hearing aid fitting in different language families is used as a different meta-learning task. The pre-trained model is trained using samples from each language family to obtain a parent model with the ability to learn new language families. LoRA adapters are embedded in the feature fusion module of the parent model and each expert network, respectively. For each language family to be fitted, the following operations are performed: the LoRA adapters and gating networks are fine-tuned using samples from the current language family to adapt to the characteristics of the current language family and learn to assign expert networks differently for different language families; the fine-tuned parameters and other parts of the parent model except for the gating network together constitute a hearing aid fitting model suitable for the current language family; wherein, each language family to be fitted can include language families outside the training set.

2. The method according to claim 1, characterized in that, The acquisition of the training set includes: Collect multiple sound signals from the same language family; Extract the fundamental frequency and formant of each audio signal segment separately; The fundamental frequency mean, formant mean, and variance of the multiple sound signals, along with the number of vowels in the same language family, are collectively used as the acoustic features of the same language family.

3. The method according to claim 1, characterized in that, The pre-training of the hearing aid fitting model using multilingual samples includes: Each input of any sample is preprocessed into a feature vector, and all feature vectors are input together into the feature interaction module for feature interaction. The interactive features are input into the feature fusion module for deep feature fusion. The fused features are simultaneously input into each expert network and the gating network, and the weights of each expert network are output by the gating network. The outputs of each expert network are weighted and fused according to their respective weights. The weighted and fused features are input into the output module to predict the hearing aid fitting parameters and frequency response curve of any sample.

4. The method according to claim 1, characterized in that, The feature interaction module adopts an MLP-Mixer structure to reduce interference between local features; The feature fusion module adopts a Transformer structure to enhance the interaction between global features; The gated network and each expert network adopt an MLP structure.

5. The method according to claim 1, characterized in that, Also includes: During training, the difference between the true and predicted values ​​of hearing aid fitting parameters and frequency response curves is minimized, and each expert network uses equalization to update the model parameters.

6. The method according to claim 1, characterized in that, The meta-learning training of the pre-trained model using samples from various language families includes: S10. Set the first model parameter set and the second model parameter set, and use the parameters of the pre-trained model as the initial values ​​of the first model parameter set; S20. Start the first loop, and perform the following operation in each iteration: S201. Assign the values ​​from the first model parameter set to the second model parameter set; S202. Start the second loop. In each loop, load the values ​​in the second model parameter set into the hearing aid fitting model, select a language family that has not been selected, update the parameters of the hearing aid fitting model with the samples of the language family, and overwrite the original values ​​in the second model parameter set with the updated parameters. Then return to S202 and enter the next cycle of the second loop until all language families have been traversed. S203. Load the latest value in the second model parameter set into the hearing aid fitting model, and update the value in the first model parameter set using the updated model loss function value. Then return to S201 and enter the next cycle of the first loop until the set termination condition is met.

7. The method according to claim 1, characterized in that, The LoRA adapter is embedded in the feature fusion module of the parent model and each expert network, including: In the linear layer of the feature fusion module of the parent model, a LoRA adapter is embedded; LoRA adapters are embedded in the linear layers of each expert network in the parent model.

8. A multilingual artificial intelligence hearing aid fitting method, characterized in that, include: Obtain multiple inputs from new users, including: basic information, language family, and acoustic features of the language family; Load a hearing aid fitting model applicable to the language family. The hearing aid fitting model includes a feature interaction module, a feature fusion module and its LoRA adapter, a gating network, multiple expert networks and their respective LoRA adapters, and an output module. Each input is preprocessed into a feature vector, and then input into the hearing aid fitting model applicable to the language family to predict the hearing aid fitting parameters and frequency response curve of the new user.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the multilingual AI hearing aid fitting model construction method according to any one of claims 1-7, or the multilingual AI hearing aid fitting method according to claim 8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the multilingual AI hearing aid fitting model construction method according to any one of claims 1-7, or the multilingual AI hearing aid fitting method according to claim 8.

Images