Intelligent fitting method of multi-modal hearing aid based on ear canal spatial structure

Abstract

This invention discloses a multimodal intelligent hearing aid fitting method based on ear canal spatial structure, comprising: acquiring the user's multimodal hearing information, wherein one modality is the spatial structure modality of the ear canal and the hearing aid shell; using a spatial structure encoder to extract potential features of the ear canal spatial structure and the hearing aid spatial structure respectively; and identifying key hearing information under the spatial structure modality based on each potential feature, the key hearing information including the direction matching information between the hearing aid and the ear canal, and / or the hearing aid vent information; using a feature extraction module for other modalities to extract potential features and key hearing information of other modalities; and combining the potential features and key hearing information of each modality to determine the fitting parameters of the hearing aid. This embodiment can improve the fit of hearing aids.

Description

Technical Field

[0001] The embodiments of the present invention relate to the fields of artificial intelligence and hearing aid fitting technology, and in particular to a multimodal intelligent hearing aid fitting method based on the ear canal spatial structure. Background Technology

[0002] In the field of hearing aid fitting, traditional methods typically involve inputting hearing test information and personal details (such as gender, age, and medical history) into a fitting formula (e.g., the DSL or NAL2 formula). The formula then outputs a set of hearing aid parameters based on calculation rules derived from clinical data statistics. Artificial intelligence-based hearing aid fitting methods, developed on this basis, largely follow this input-output parameter model. However, this input data cannot fully reflect the user's personalized hearing information, therefore the fitting parameters of the hearing aid cannot be well adapted to the user's hearing condition. Summary of the Invention

[0003] This invention provides a multimodal intelligent hearing aid fitting method based on the ear canal spatial structure to solve the above-mentioned technical problems.

[0004] In a first aspect, embodiments of the present invention provide a multimodal intelligent hearing aid fitting method based on the ear canal spatial structure, comprising: Acquire the user's multimodal hearing information, one of which is the spatial structure mode of the ear canal and hearing aid shell; Using a spatial structure encoder, potential features of the ear canal spatial structure and the hearing aid spatial structure are extracted respectively; and based on each potential feature, key hearing information under the spatial structure mode is identified. The key hearing information includes the direction matching information between the hearing aid and the ear canal, and / or the hearing aid vent information. By utilizing feature extraction modules for other modalities, latent features and key hearing information from other modalities can be extracted; By combining the potential characteristics of each modality and key hearing information, the fitting parameters of the hearing aid are determined.

[0005] In a second aspect, embodiments of the present invention provide an electronic device, the electronic device comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal intelligent hearing aid fitting method based on the ear canal spatial structure as described in any embodiment.

[0006] Thirdly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal intelligent hearing aid fitting method based on the ear canal spatial structure described in any embodiment.

[0007] In summary, the embodiments of the present invention provide a multimodal intelligent hearing aid fitting method based on the spatial structure of the ear canal. By utilizing multimodal data such as audiology, medical imaging, three-dimensional ear canal and electroencephalogram (EEG) signals, it captures the user's personalized hearing characteristics, constructs a richer, more multidimensional and more personalized deep learning network, realizes a multimodal intelligent hearing aid fitting method, and reduces manual intervention, providing important support for improving the efficiency and fit level of existing fitting methods.

[0008] In particular, considering that users with the same hearing test data (such as the test threshold) may have vastly different frequency responses and advanced function requirements due to differences in ear canal morphology (such as short and straight ear canals or collapsed ear canals), for example, if the ear canal is very short and straight, it is relatively easier to produce whistling, so the feedback suppression level needs to be increased, this embodiment introduces the individual differences in the user's ear canal structure (such as curvature and volume) into hearing aid fitting. Through pre-trained spatial encoders and spatial decoders, potential features that can represent three-dimensional spatial information are extracted, and multiple tasks such as hearing aid region segmentation, ear canal direction recognition, and sound hole direction recognition are established using a multi-task mechanism. Based on the task execution results, the direction matching information of the hearing aid ear canal and sound hole, as well as the vent information, are further determined, which provides an important foundation for improving the fit of hearing aid fitting.

[0009] Furthermore, this embodiment extracts two types of features for each modality of hearing information: one is deep latent features (such as feature vectors or feature matrices) extracted through a neural network model, and the other is key hearing information further identified from the hearing data itself or the deep latent features. The deep latent features represent the original hearing information through finite-dimensional data, providing a comprehensive information representation. The key hearing information, however, is a crucial component of both the original hearing information and the deep latent features for hearing aid fitting. This embodiment extracts the key hearing information for each modality separately for subsequent calculations of hearing aid parameters to prevent subsequent algorithms from insufficiently learning this key hearing information when learning the deep latent features, thus affecting hearing aid fit. Attached Figure Description

[0010] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0011] Figure 1 This is a flowchart of a multimodal intelligent hearing aid fitting method based on the ear canal spatial structure provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of a hearing test report provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of an ear CT / MRI image provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a three-dimensional spatial structure of the ear canal provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of an electroencephalogram (EEG) signal provided in an embodiment of the present invention; Figure 6 This is a flowchart of another intelligent multimodal hearing aid fitting method based on the ear canal spatial structure provided by an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0013] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0014] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0015] Figure 1 This is a flowchart illustrating a multimodal intelligent hearing aid fitting method based on the ear canal spatial structure, as provided in an embodiment of the present invention. The method is executed by an electronic device, such as... Figure 1 As shown, the specific steps include the following: S110. Acquire the user's multimodal hearing information, one of which is the spatial structure mode of the ear canal and the hearing aid shell.

[0016] This embodiment uses the user's multimodal hearing information to fully reflect the user's personalized hearing information, so as to better adapt to the user's hearing condition.

[0017] Optionally, the hearing information includes at least one of the following modalities: (1) Text modality (e.g., hearing test report) Figure 2 The text in the document, referred to as the hearing test text, includes test data such as hearing thresholds (air conduction, bone conduction, and discomfort thresholds), broadband acoustic impedance, acoustic reflection threshold, audible contrast threshold, maximum speech recognition rate, otoacoustic emissions, Quick Sin test, and tinnitus matching test results.

[0018] (2) Medical imaging modalities (e.g., ear CT / MRI images, Figure 3 ); (3) Spatial structural modality (e.g., the three-dimensional spatial structure of the ear canal, the three-dimensional spatial structure of the hearing aid shell, Figure 4 Optionally, the 3D spatial structure of the ear canal and the hearing aid shell can be obtained through laser scanning. In practical applications, it has been found that users with the same hearing threshold may have vastly different frequency responses and advanced function requirements due to differences in ear canal morphology (such as short and straight ear canals or collapsed ear canals). For example, if the ear canal is very short and straight, it is relatively easier to produce feedback, and the feedback suppression level needs to be increased. Therefore, this embodiment incorporates individual differences in the user's ear canal structure (such as curvature and volume) into the hearing aid fitting process.

[0019] (4) EEG modalities (such as EEG / ERP signals, etc.) Figure 5EEG signals represent the feedback of the user's brain center after processing speech. In this embodiment, EEG signals are used to perceive changes in the user's hearing threshold and speech recognition ability under noise after hearing aids in specific auditory scenarios. Even for users who have no speech ability or have difficulty describing in words, EEG signals can be used to characterize their neural feedback (such as the auditory cortex's response to specific speech sounds).

[0020] S120. Using the feature extraction modules of each modality, extract the potential features and key hearing information of each modality respectively.

[0021] This embodiment extracts two types of features for each modality of hearing information: one is deep latent features (such as feature vectors or feature matrices) extracted through a neural network model, and the other is key hearing information further identified from the hearing data itself or from the deep latent features. The deep latent features represent the original hearing information using finite-dimensional data, providing a comprehensive information representation. The key hearing information, however, is a crucial component of both the original hearing information and the deep latent features, essential for hearing aid fitting. This embodiment extracts the key hearing information for each modality separately for subsequent calculations of hearing aid parameters to prevent insufficient learning of this key hearing information by subsequent algorithms when learning the deep latent features.

[0022] In one specific implementation, for the text modality, taking a hearing test text as an example, a text modality feature extraction module can be used to extract latent features from the hearing test text and extract keywords related to hearing information from the hearing test text. Optionally, refer to... Figure 6 The feature extraction module for the text modality can employ a pre-trained text encoder. (Text information from the hearing test report.) The text is transformed into a feature vector through a pre-trained text encoder. :

[0023] Simultaneously, keywords and their specific information can be extracted from the hearing report text. Keywords may include various test indicators, disease diagnosis, population classification, and historical fitting records. This process corresponds to... Figure 6 Classify the hearing test information in the document.

[0024] In one specific implementation, for a medical imaging modality, taking ear CT / MRI images as an example, a feature extraction module for the imaging modality can be used to extract latent features from the ear CT / MRI images, and key hearing information in the images can be identified based on these latent features. Optionally, refer to... Figure 6The image modality feature extraction module can employ a VAE-encoder (variational autoencoder) to reduce the dimensionality of extracted images of key structures such as the cochlea and tympanic membrane, obtaining encoder vectors. These vectors are then used as a low-dimensional latent representation of the image features. For example, assuming the original image is... The encoder's mapping function is Then the extracted low-dimensional latent vector The calculation is as follows:

[0025] in, and These are the mean and standard deviation of the latent features output by the encoder, respectively. Let be the disturbance variable. Take . ,according to and It is possible to determine the latent features of image modalities.

[0026] Simultaneously, a feature classification method is employed to classify the latent vectors of image modalities according to specific key hearing information (e.g., whether the external ear is locked, whether there is large vestibular aqueduct syndrome, etc.), with each classification corresponding to one key hearing information. Assume the classifier's mapping function is... The classification result It can be represented as follows:

[0027] In one specific implementation, for EEG modalities, taking EEG / ERP signals as an example, a feature extraction module for EEG modalities can be used to extract latent features of the EEG signals, and key EEG-related hearing information can be identified based on the latent features. Optionally, refer to... Figure 6 The feature extraction module for EEG modalities can adopt an EEGNet structure. Assume the input of the EEG signal is... , Batch size ( These are training parameters; the mode is in use. ), This refers to the number of channels (e.g., the number of electrodes). For time steps, first use... Convolution is performed along the time dimension to obtain features. :

[0028] Then use a deep convolutional network. , to obtain features :

[0029] Finally, a separable convolutional network is used. To obtain the latent features of EEG modalities :

[0030] Based on The data was classified to obtain key auditory information related to the electroencephalogram (EEG):

[0031] Traditional hearing aid fitting does not personalize the fitting based on the user's subjective response. However, this embodiment extracts brainwave test data and extracts text from brainwave features to convert diagnostic conclusions into machine-readable tags, which are then used to adjust hearing aid parameters to match the user's subjective comfort.

[0032] In one specific implementation, for the spatial structure modality, a spatial structure encoder can be used to extract the potential features of the ear canal spatial structure and the hearing aid spatial structure, respectively; and based on each potential feature, key hearing information under the spatial structure modality can be identified. The key hearing information includes the direction matching information of the hearing aid and the ear canal (such as whether the two directions are deviated, and what the deviation angle is), and / or the hearing aid vent information (such as whether there is a vent, and what the diameter of the vent is).

[0033] Optional, refer to Figure 6 The spatial structure encoder can be implemented using a VAE-encoder (with a mapping function of...). ), respectively on the 3D spatial structure of the ear canal 3D spatial structure of hearing aid shell Dimensionality reduction encoding is performed to obtain the following results: latent feature vectors ,as well as latent feature vectors :

[0034]

[0035] To ensure that the latent feature vectors under the spatial structure modality can comprehensively represent the spatial structure information of the ear canal and the hearing aid shell, the spatial structure encoder can be pre-trained as follows: The spatial structure encoder extracts latent features of the ear canal spatial structure; the spatial structure decoder reconstructs the ear canal spatial structure based on these latent features; and the spatial structure encoder and decoder are trained by minimizing the difference between the reconstructed spatial structure and the original spatial structure. Similarly, the hearing aid shell spatial structure data is also trained using the above parameters. The trained spatial structure encoder is used to extract latent features of subsequent ear canal spatial structures or hearing aid spatial structures.

[0036] After obtaining the latent features of the spatial structure, a fusion encoder is first used to fuse the latent features of the ear canal spatial structure and the latent features of the hearing aid spatial structure to obtain fused features. Optionally, the fusion encoder can employ a Transformer architecture to fully leverage the complementarity between 3D modalities and improve model performance.

[0037] in, It can be a feature matrix of a fixed size. and Input after splicing Process it.

[0038] Then, multiple tasks are constructed using a multi-tasking mechanism. Each task is performed based on the fusion features. Optionally, these tasks include at least one of the following: hearing aid shell region segmentation, hearing aid shell type classification, ear canal orientation recognition, and sound outlet orientation recognition, each yielding its respective task result. :

[0039] Finally, based on the identified ear canal direction and sound outlet direction, the direction matching information between the hearing aid and the ear canal is determined. Optionally, this direction matching information includes the cosine similarity and the included angle between the ear canal direction and the sound outlet direction.

[0040] Simultaneously, based on the region segmentation results of the hearing aid shell, the vent information of the hearing aid is determined. Optionally, the region segmentation results include the region attributes of each point, from which points belonging to the vent are extracted; if no such point exists, there is no vent; if such a point exists, circle recognition is performed on the vent points to obtain the vent diameter. Compared to the traditional "ideal cylinder" assumption, 3D ear canal modeling can reduce the acoustic simulation error from ±8dB to ±2dB.

[0041] S130. Based on the potential characteristics of each modality and key hearing information, determine the fitting parameters of the hearing aid.

[0042] After processing by S120, the latent features and key hearing information for each modality are obtained. Optionally, data can be constructed separately for the two types of data: One type is the Feature DB (vector database), which stores latent feature vectors, specifically storing feature representations of textual information, image information, 3D spatial structure information, and electroencephalogram (EEG) information. This vector database enables fast feature matching and supports efficient feature similarity retrieval.

[0043] One type is NoSQL DB (non-relational database), which is used to store key hearing information extracted from various modalities, specifically storing hearing test information, CT image information, EEG signals, and keywords or classification text in spatial structure models.

[0044] Based on the above two types of information, the fitting parameters of the hearing aid can be comprehensively determined. In one specific implementation, key hearing information of each modality can first be embedded and encoded to convert it into embedded features; in addition, user needs in natural language can also be converted into embedded features, including user subjective descriptions, semantic intent space, user preferences, etc. Keywords that cannot be directly processed by the computer need to be parsed to extract semantic emphasis (e.g., high-frequency gain requirement: +3dB) and converted into vectors that the computer can recognize.

[0045] Simultaneously, the latent features corresponding to each modality in the vector library are expanded in dimension (such as EEG features) through a learnable fully connected layer and projected into a high-dimensional space (e.g., unified to 1024 dimensions).

[0046]

[0047]

[0048]

[0049] in, represent .

[0050] Then, each embedded feature It is concatenated with high-dimensional spatial features to form a global feature vector. :

[0051] Then use the hybrid expert model (MoE) to... Weighted fusion is performed. Specifically, gating networks. one The classifier determines which expert is "most suitable" based on the global features of the current input and assigns weights accordingly. It outputs a... A probability distribution vector of dimension, where Total number of experts:

[0052] in, and These represent the weight and bias matrices, respectively.

[0053] Each expert model is a feedforward neural network, including, for example, two fully connected layers, using the GeLU activation function in between. Its output can be represented as:

[0054] in, and , and These represent the weight and bias matrices of the two layers, respectively.

[0055] Each expert network learns to focus on different aspects of the data (each expert learns to process different types of global feature combinations. For example, expert 1 may focus more on features of "text description + 3D geometry", while expert 2 may focus more on features of "image features + classification labels").

[0056] Final output of the MoE layer yes The weighted sum of the outputs of the expert models, with the weights determined by the gating network:

[0057] Will Input an MLP-based prediction head to obtain the final prediction result. :

[0058] In one alternative implementation, the output of the MLP predictor head can be adapted hearing aid parameters. In this approach, a training sample set can be pre-constructed, with each sample including the user's multimodal hearing information and the user's satisfactory hearing aid parameters fitted manually by an audiologist. This sample set is then used to pre-train the entire network (along with the spatial decoder). The training loss function includes the difference between the predicted hearing aid parameters and the labeled hearing aid parameters, as well as the difference between the spatial structure reconstructed by the spatial decoder and the original spatial structure. By minimizing these differences, the prediction head can output the appropriate hearing aid parameters.

[0059] In another alternative implementation, combined with Figure 6 The output of the MLP predictor can also be frequency response curves under different input sound intensities, which represent the target frequency response curve expected to be achieved in hearing aid fitting. Then, based on the reinforcement learning module, the hearing aid chip parameters can be continuously adjusted from the initial parameters until the frequency response curve corresponding to these parameters is essentially consistent with the target frequency response curve, thus obtaining the final hearing aid parameters. Optionally, the policy network and value network in this reinforcement learning model can adopt an MLP structure, where the current hearing aid parameters, their corresponding current frequency response curve, and the target frequency response curve can be concatenated as state variables. The vector formed by concatenating the adjustment actions [-1, 0, 1] of each hearing aid parameter (corresponding to decreasing by one step, keeping it unchanged, and increasing by one step, respectively) is used as the action variable. The reward function is determined based on the difference between the frequency response curve after hearing aid parameter adjustment and the target frequency response curve. The smaller the difference, the larger the reward value, thereby updating the hearing aid parameters. Different hearing aid shell types can correspond to different reinforcement learning module parameters; the hearing aid shell type classification in the previous steps can be applied here.

[0060] Regardless of the method, due to the comprehensive consideration of the user's multimodal hearing information, especially the spatial structural modality of the ear canal and hearing aid, the hearing aid parameters output by the method in this embodiment can provide a good basis for adaptation. Further manual fine-tuning can be carried out based on these parameters, but this embodiment does not impose specific limitations.

[0061] In summary, this embodiment provides a multimodal intelligent hearing aid fitting method based on the spatial structure of the ear canal. Utilizing multimodal data such as audiology, medical imaging, three-dimensional ear canal data, and electroencephalogram (EEG) signals, it captures the user's personalized hearing characteristics, constructs a richer, more multidimensional, and more personalized deep learning network, and realizes an intelligent multimodal hearing aid fitting method. This reduces manual intervention and provides significant support for improving existing fitting efficiency and fit levels. Furthermore, this embodiment uses a hybrid expert model (MoE), which can learn how to optimally integrate features from all modalities to make global decisions, further enhancing fitting accuracy and personalized fit levels.

[0062] In particular, considering that users with the same hearing test data (such as the test threshold) may have vastly different frequency responses and advanced function requirements due to differences in ear canal morphology (such as short and straight ear canals or collapsed ear canals), for example, if the ear canal is very short and straight, it is relatively easier to produce whistling, so the feedback suppression level needs to be increased, this embodiment introduces the individual differences in the user's ear canal structure (such as curvature and volume) into hearing aid fitting. Through pre-trained spatial encoders and spatial decoders, potential features that can represent three-dimensional spatial information are extracted, and multiple tasks such as hearing aid region segmentation, ear canal direction recognition, and sound hole direction recognition are established using a multi-task mechanism. Based on the task execution results, the direction matching information of the hearing aid ear canal and sound hole, as well as the vent information, are further determined, which provides an important foundation for improving the fit of hearing aid fitting.

[0063] Furthermore, this embodiment extracts two types of features for each modality of hearing information: one is deep latent features (such as feature vectors or feature matrices) extracted through a neural network model, and the other is key hearing information further identified from the hearing data itself or the deep latent features. The deep latent features represent the original hearing information through finite-dimensional data, providing a comprehensive information representation. The key hearing information, however, is a crucial component of both the original hearing information and the deep latent features for hearing aid fitting. This embodiment extracts the key hearing information for each modality separately for subsequent calculations of hearing aid parameters to prevent subsequent algorithms from insufficiently learning this key hearing information when learning the deep latent features, thus affecting hearing aid fit.

[0064] It should be noted that all user data involved in this application is information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0065] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 7 As shown, the device includes a processor 60, a memory 61, an input device 62, and an output device 63; the number of processors 60 in the device can be one or more. Figure 7 Taking a processor 60 as an example; the processor 60, memory 61, input device 62, and output device 63 in the device can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0066] The memory 61, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the multimodal hearing aid intelligent fitting method based on ear canal spatial structure in this embodiment of the invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 61, thereby realizing the aforementioned multimodal hearing aid intelligent fitting method based on ear canal spatial structure.

[0067] The memory 61 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 61 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include memory remotely located relative to the processor 60, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0068] Input device 62 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 63 may include display devices such as a display screen.

[0069] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multimodal hearing aid intelligent fitting method based on the ear canal spatial structure of any embodiment.

[0070] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0071] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0072] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0073] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as C or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal intelligent fitting method for hearing aids based on the spatial structure of the ear canal, characterized in that, include: Acquire the user's multimodal hearing information, one of which is the spatial structure mode of the ear canal and hearing aid shell; Using a spatial structure encoder, latent features of the ear canal spatial structure and the hearing aid spatial structure are extracted respectively. Based on each potential feature, key hearing information under the spatial structure mode is identified. The key hearing information includes the direction matching information between the hearing aid and the ear canal, and / or the hearing aid vent information. By utilizing feature extraction modules for other modalities, latent features and key hearing information from other modalities can be extracted; By combining the potential characteristics of each modality and key hearing information, the fitting parameters of the hearing aid are determined.

2. The method according to claim 1, characterized in that, The multimodal hearing information also includes at least one of the following modalities: hearing test text, ear images, and electroencephalogram (EEG) signals.

3. The method according to claim 1, characterized in that, Before extracting the latent features of the ear canal spatial structure and the hearing aid spatial structure using a spatial structure encoder, the method further includes: Using a spatial structure encoder, latent features of the ear canal spatial structure are extracted; The spatial structure decoder is used to reconstruct the ear canal spatial structure based on the potential features; By minimizing the difference between the restored spatial structure and the original spatial structure, the spatial structure encoder and spatial structure decoder are trained. The trained spatial structure encoder is used to extract potential features of the subsequent ear canal spatial structure.

4. The method according to claim 1, characterized in that, The identification of key hearing information under spatial structural modalities based on various potential features includes: A fusion encoder is used to fuse the latent features of the ear canal spatial structure and the latent features of the hearing aid spatial structure. Through a multi-task mechanism, the hearing aid shell region segmentation, hearing aid shell type classification, ear canal direction identification, and sound outlet direction identification are performed based on the fusion features. Based on the identified ear canal direction and sound outlet direction, determine the direction matching information between the hearing aid and the ear canal; Based on the region segmentation results of the hearing aid shell, the information of the hearing aid vent is determined.

5. The method according to claim 4, characterized in that, The step of determining the direction matching information between the hearing aid and the ear canal based on the identified ear canal direction and sound outlet direction includes: Calculate the cosine similarity and angle between the identified ear canal direction and the sound outlet direction.

6. The method according to claim 4, characterized in that, The step of determining the hearing aid vent information based on the region segmentation results of the hearing aid shell includes: From the area segmentation results of the hearing aid shell, extract the vent points; Based on the extraction results, determine whether there are vents; The diameter of the vent is obtained by identifying the points on the vent.

7. The method according to claim 2, characterized in that, The step of using a feature extraction module for other modalities to extract latent features and key hearing information from other modalities includes at least one of the following steps: Using the text modality feature extraction module, the latent features of the hearing test text are extracted, and hearing information keywords are extracted from the hearing test text; The feature extraction module of the image modality is used to extract the latent features of the ear image, and the key hearing information in the image is identified based on the latent features; The feature extraction module of EEG modality is used to extract the potential features of the EEG signal, and the key hearing information related to EEG is identified based on the potential features.

8. The method according to claim 1, characterized in that, The process of integrating the latent features and key hearing information of each modality to determine the fitting parameters of the hearing aid includes: Key hearing information from each modality is embedded and encoded. The embedding vectors and latent features of each modality are jointly provided to the prediction head to predict the fitting parameters of the hearing aid.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal intelligent hearing aid fitting method based on the ear canal spatial structure as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the multimodal intelligent hearing aid fitting method based on the ear canal spatial structure as described in any one of claims 1-8.

Images