Speech audiometry-based hearing impaired patient hearing loss evaluation method and system

Abstract

The invention discloses a speech audiometry-based hearing impairment evaluation method and system, and the method comprises the steps: constructing a speech synthesis model with original audio voiceprint features through the original audio data of speech audiometry, and cleaning the original audio text data and user feedback text data; converting the original audio text data and the user feedback text data into audio signals by adopting a speech synthesis model, obtaining a first audio signal and a second audio signal, performing Fourier transform on the first audio signal and the second audio signal, and calculating the energy of each frequency point so as to obtain multiple groups of audiometric audio data; according to the method, the function of automatically evaluating the hearing state of the patient is achieved, an additional pure tone audiometry process is not needed, meanwhile, the method for automatically evaluating the hearing state of the patient based on the speech audiometry result can fuse personal information of the patient for evaluation, the evaluation efficiency is high, the evaluation result is more accurate, and the method is suitable for popularization and application. And the device is suitable for wide popularization and use.

Description

Technical Field

[0001] The present invention relates to the technical field of hearing status assessment for hearing-impaired patients, and in particular to a method and system for assessing hearing loss for hearing-impaired patients based on speech audiometry. Background Art

[0002] The World Health Organization defines hearing loss as a silent disease that has affected the lives of more than 1.5 billion people worldwide to varying degrees. Hearing loss directly leads to barriers to normal language communication, which in turn leads to reduced social activity ability, autism, psychological disorders and even Alzheimer's disease, which seriously affects the quality of life of people with hearing impairments. Studies have shown that early detection of hearing loss in patients and intervention can prevent further deterioration of hearing conditions.

[0003] At present, the hearing status of hearing-impaired patients is mostly evaluated by pure tone audiometry and speech audiometry. Although pure tone audiometry can relatively accurately measure the hearing threshold of hearing-impaired patients at fixed frequencies, it is relatively insufficient in evaluating the speech comprehension of patients. In addition, speech audiometry focuses more on the speech comprehension of the subject, which better represents the true language perception ability of hearing-impaired patients. When the hearing loss of the test subject is reduced to a certain extent, intervention is required, and wearing hearing aids is currently the most effective means for hearing-impaired patients to improve their hearing. However, the hearing compensation strategy of hearing aids is generally based on the pure tone audiometry results of the patient and the parameters are adjusted based on the speech audiometry results. This makes the traditional hearing aid parameter adjustment process usually cumbersome and very time-consuming. At the same time, there is still a significant difference between the results of pure tone audiometry and the hearing aid compensation, which leads to a large deviation in the initial hearing aid parameter adjustment. Therefore, it is necessary to design a hearing loss evaluation method and system for hearing-impaired patients based on speech audiometry. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and to better and effectively solve the problem that the current hearing compensation strategy of hearing aids is generally based on the pure tone audiometry results of patients and parameter adjustment is performed based on the speech audiometry results. This makes the traditional hearing aid parameter adjustment process usually cumbersome and very time-consuming. At the same time, there is still a significant difference between the results of pure tone audiometry and hearing aid compensation, which leads to a large deviation in the initial hearing aid parameter adjustment. A hearing loss assessment method and system for hearing-impaired patients based on speech audiometry are provided, which realizes the function of automatically assessing the patient's hearing status and does not require an additional pure tone audiometry process. At the same time, the method of automatically assessing the patient's hearing status based on the speech audiometry results can integrate the patient's personal information for assessment, which not only has a high assessment efficiency, but also makes the assessment results more accurate.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A method and system for evaluating hearing loss of hearing-impaired patients based on speech audiometry, comprising the following steps:

[0007] Step A, using the original audio data of speech audiometry to construct a speech synthesis model with the original audio voiceprint features;

[0008] Step B, cleaning the original audio text data and user feedback text data;

[0009] Step C, using a speech synthesis model to convert the original audio text data and the user feedback text data into audio signals to obtain a first audio signal and a second audio signal;

[0010] Step D, performing Fourier transform on the first audio signal and the second audio signal and calculating the energy of each frequency point to obtain multiple groups of audiometry audio data;

[0011] Step E: Establish a hearing assessment network and input the listener's personal information and multiple sets of audio data into the hearing assessment network after regularization. Then, use the deep speech connection, spectral clues and global patterns of different audio samples to conduct a comprehensive a posteriori hearing status assessment to complete the hearing loss assessment task.

[0012] The aforementioned method for assessing hearing loss of hearing-impaired patients based on speech audiometry, step A, uses the original audio data of the speech audiometry to construct a speech synthesis model with the original audio voiceprint features, wherein the speech synthesis model adopts the GPT-SoVITS architecture, and the speech synthesis model is trained using the speech audiometry dataset.

[0013] The aforementioned method for assessing hearing loss of hearing-impaired patients based on speech audiometry, step B, cleans the original audio text data and user feedback text data. The specific cleaning process is to remove the audio content samples that the user is completely unable to repeat and play, and then replace the erroneous content in the repeated text with empty samples and align it with the successfully repeated text.

[0014] The aforementioned method and system for assessing hearing loss of hearing-impaired patients based on speech audiometry, step C, uses a speech synthesis model to convert the original audio text data and user feedback text data into audio signals to obtain a first audio signal and a second audio signal, specifically, the aligned original audio text data and user feedback text data are speech synthesized according to timestamps.

[0015] The aforementioned method for assessing hearing loss of hearing-impaired patients based on speech audiometry, step D, performs Fourier transform on the first audio signal and the second audio signal and calculates the energy of each frequency point to obtain multiple sets of audio data for audio measurement. The specific steps are as follows:

[0016] Step D1, windowing the first audio signal and the second audio signal, and then mapping the audio from the time domain to the frequency domain through a fast Fourier transform operator;

[0017] Step D2: taking the square root of the real part and the imaginary part of each frequency point as the energy of the frequency point.

[0018] The aforementioned method for assessing hearing loss of hearing-impaired patients based on speech audiometry, step E, establishes a hearing assessment network and inputs the listener's personal information and multiple sets of audio data into the hearing assessment network after regularization, and then uses the deep speech connection, spectral clues and global patterns of different audio samples to perform a comprehensive posterior hearing state assessment to complete the hearing loss assessment operation, wherein the personal information includes the patient's gender, age and years of hearing aid use, and the hearing assessment network is used to receive input in different formats of text and audio to assess the posterior hearing state estimation map and hearing loss level. The specific steps are as follows,

[0019] Step E1, receiving text input, specifically using the pre-trained word embedding language model to encode the original text and the user feedback text into vector space and respectively use V l and V u Indicates that, then for the input vector V l and V u Use full connection to calculate the global similarity deep pattern feature V across frames txt , as shown in formula (1),

[0020]

[0021] Among them, tanh represents the activation function, represents Layer Normalization, FC represents the fully connected layer;

[0022] Step E2, receiving audio input, specifically adopting multi-domain feature extraction and factorized feature fusion to extract deep pattern features of user spectral hearing loss status, the specific steps are as follows:

[0023] Step E21, multi-domain feature extraction, specifically, using the cross-correlation feature space and the time-series high-dimensional deep feature space to extract the cross-correlation feature X from the input sub-band energy information corr , time series high-dimensional deep features X RNN and patient-specific characteristics

[0024] Step E22, factorized feature fusion, specifically fusion of the cross-correlation feature X corr , time series high-dimensional deep features X RNN and patient-specific characteristics In this way, the potential patterns of hearing loss in different patients are captured, and then the general features X′ of the deep sequence and the personalized features of the patients are fused through affine transformation. Get the spectrum hearing loss state deep mode feature X audio ;

[0025] Step E3, integrating the text global similarity deep pattern feature V txt and spectrum hearing loss status deep pattern feature X audio And obtain the multi-target fusion feature X fusion , and then obtain the fused feature X fusion Audiogram and hearing loss grade are predicted as shared information.

[0026] The aforementioned method for assessing hearing loss of hearing-impaired patients based on speech audiometry, step E21 specifically comprises the following steps:

[0027] Step E211, extracting sub-band cross-correlation features X corr (b), as shown in formula (1),

[0028]

[0029] Among them, b∈[1,...,B] represents the divided subband, and Represent the sub-band energies of the hearing-impaired patient and the original audio respectively;

[0030] Step E212, extracting time series high-dimensional depth features X RNN , specifically, the concatenation sequence of the hearing-impaired patient and the original audio subband energy vector in the subband dimension Input a multi-layer bidirectional LSTM network, and then extract the internal states of all layers of LSTM networks for weighted sum, as shown in formula (2):

[0031]

[0032] Among them, O l and C l Represent the output and internal state of the l-th layer LSTM network, w l Represents the weight components corresponding to the output and internal state of the LSTM network;

[0033] Step E213, encode the patient's personalized feature information of gender, age and hearing aid wearing years to Z = [age, gender, year used], and then discretize the patient's personalized coding features by codebook sampling, and the mapping function is expressed as Discrete coding From a codebook of V learnable entries Sampling in The index i is calculated by Gumbel-Softmax method, as shown in formula (3):

[0034]

[0035] Where n=-log(-log(u)), and uU(0,1) represents random sampling noise.

[0036] The aforementioned method for assessing hearing loss of hearing-impaired patients based on speech audiometry, step E22 has the following specific steps:

[0037] Step E221, using affine transformation to fuse the general feature X′ of the depth sequence and the personalized feature of the patient Get the feature X after affine transformation fusion stage1 , as shown in formula (4),

[0038]

[0039] in, represents batch normalization, Θ represents the model parameters of the fully connected layer;

[0040] Step E222, through the parameter matrix W fa (b) = [w 1 ,…,w D ](b∈[1,…,B])X stage1 Mapped to D different subspaces Then X corr It is also mapped to D subspaces and used as the weight of each space

[0041] Step E223, using a weighted sum method to fuse and obtain the deep pattern features of the spectrum hearing loss state As shown in formula (5),

[0042]

[0043] The aforementioned method for assessing hearing loss of hearing-impaired patients based on speech audiometry, step E3 specifically comprises the following steps:

[0044] Step E31, integrating the text global similarity deep pattern feature V txt and spectrum hearing loss status deep pattern feature X audio And obtain the multi-target fusion feature X fusion Specifically, the text feature and audio feature expressions are concatenated into a unified vector and then sent to a multimodal fusion network. The multimodal fusion network is composed of decomposed attention and multi-layer dilated convolution, as shown in formula (6).

[0045]

[0046] Step E32, using the fused features as shared information to predict the audiogram and hearing loss level, specifically, the hearing loss level is calculated based on multi-head attention and fully connected network level = FC (MHSA (X fusion )), and then the hearing loss level index and the extracted features are fused and the audiogram is calculated through full connection transformation, as shown in formula (7).

[0047] audiogram=FC(concat(X fusion ,level))(7).

[0048] A hearing loss assessment system for hearing-impaired patients based on speech audiometry, comprising a model building module, a data cleaning module, a signal conversion module, a signal transformation module and a hearing loss assessment module, wherein the model building module is used to construct a speech synthesis model with original audio voiceprint features using original audio data of speech audiometry;

[0049] The data cleaning module is used to clean the original audio text data and the user feedback text data to obtain a first audio signal;

[0050] The signal conversion module is used to convert the original audio text data and the user feedback text data into an audio signal using a speech synthesis model to obtain a second audio signal;

[0051] The signal conversion module is used to perform Fourier transformation on the first audio signal and the second audio signal and calculate the energy of each frequency point to obtain multiple groups of audiometry audio data;

[0052] The hearing loss assessment module is used to establish a hearing assessment network and input the listener's personal information and multiple groups of audio data into the hearing assessment network after regularization, and then use the deep speech connection, spectrum clues and global patterns of different audio samples to perform a comprehensive a posteriori hearing state assessment to complete the hearing loss assessment operation.

[0053] The beneficial effects of the present invention are as follows: a method and system for evaluating hearing loss of hearing-impaired patients based on speech audiometry of the present invention first uses the original audio data of speech audiometry to construct a speech synthesis model with original audio voiceprint features, then cleans the original audio text data and user feedback text data and obtains a first audio signal, then uses the speech synthesis model to convert the original audio text data and the user feedback text data into an audio signal and obtains a second audio signal, then Fourier transforms the first audio signal and the second audio signal and calculates the energy of each frequency point to obtain multiple groups of audiometry audio data, then establishes a hearing evaluation network and inputs the listener's personal information and multiple groups of audiometry audio data into the hearing evaluation network after regularization, and then uses the deep speech connection, spectral clues and global patterns of different audiometry samples to perform a comprehensive a posteriori hearing state evaluation and complete the hearing loss evaluation operation; effectively realizes that the hearing loss evaluation method and system for hearing-impaired patients have the function of automatically evaluating the patient's hearing state, and does not require an additional pure tone audiometry process, and the method for automatically evaluating the patient's hearing state based on the speech audiometry result can integrate the patient's personality information for evaluation, which not only has a high evaluation efficiency, but also makes the evaluation result more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flow chart of a method for evaluating hearing loss of hearing-impaired patients based on speech audiometry of the present invention;

[0055] Figure 2 It is an evaluation principle diagram of a method for evaluating hearing loss of hearing-impaired patients based on speech audiometry of the present invention;

[0056] Figure 3 is a schematic diagram of the hearing assessment network structure of the present invention;

[0057] Figure 4 It is a schematic diagram of the hearing loss evaluation results of the hearing-impaired patient of the present invention. DETAILED DESCRIPTION

[0058] The present invention will be further described below in conjunction with the accompanying drawings.

[0059] like Figure 1 As shown, a method for evaluating hearing loss of hearing-impaired patients based on speech audiometry of the present invention comprises the following steps:

[0060] Step A: Use the original audio data of speech audiometry to build a speech synthesis model with the original audio voiceprint features, wherein the speech synthesis model adopts the GPT-SoVITS architecture, and the speech synthesis model is trained using the speech audiometry dataset.

[0061] Step B, cleaning the original audio text data and user feedback text data. The specific cleaning process is to remove the audio content samples that the user cannot repeat and play at all, and then replace the erroneous content in the repeated text with empty samples and align them with the successfully repeated text.

[0062] like Figure 2 As shown, step C uses a speech synthesis model to convert the original audio text data and the user feedback text data into audio signals to obtain a first audio signal and a second audio signal. Specifically, the aligned original audio text data and the user feedback text data are speech synthesized according to timestamps.

[0063] Step D, performing Fourier transform on the first audio signal and the second audio signal and calculating the energy of each frequency point to obtain multiple sets of audio measurement data. The specific steps are as follows:

[0064] Step D1, windowing the first audio signal and the second audio signal, and then mapping the audio from the time domain to the frequency domain through a fast Fourier transform operator;

[0065] Step D2: taking the square root of the real part and the imaginary part of each frequency point as the energy of the frequency point.

[0066] like Figure 3 As shown, step E, establishes a hearing evaluation network and inputs the listener's personal information and multiple groups of audio data into the hearing evaluation network after regularization, and then uses the deep speech connection, spectral clues and global patterns of different audio samples to perform a comprehensive posterior hearing state evaluation to complete the hearing loss evaluation operation, wherein the personal information includes the patient's gender, age and years of hearing aid use, and the hearing evaluation network is used to receive inputs in different formats of text and audio to evaluate the posterior hearing state estimation map and hearing loss level. The specific steps are as follows,

[0067] Step E1, receiving text input, specifically using the pre-trained word embedding language model to encode the original text and the user feedback text into vector space and respectively use V l and V u Indicates that, then for the input vector V l and V u Use full connection to calculate the global similarity deep pattern feature V across frames txt , as shown in formula (1),

[0068]

[0069] Among them, tanh represents the activation function, represents Layer Normalization, FC represents the fully connected layer;

[0070] Step E2, receiving audio input, specifically adopting multi-domain feature extraction and factorized feature fusion to extract deep pattern features of user spectral hearing loss status, the specific steps are as follows:

[0071] Step E21, multi-domain feature extraction, specifically, using the cross-correlation feature space and the time-series high-dimensional deep feature space to extract the cross-correlation feature X from the input sub-band energy information corr , time series high-dimensional deep features X RNN and patient-specific characteristics The specific steps are as follows:

[0072] Step E211, extracting sub-band cross-correlation features X corr (b), as shown in formula (1),

[0073]

[0074] Among them, b∈[1,...,B] represents the divided subband, and Represent the sub-band energies of the hearing-impaired patient and the original audio respectively;

[0075] Step E212, extracting time series high-dimensional depth features X RNN , specifically, the concatenation sequence of the hearing-impaired patient and the original audio subband energy vector in the subband dimension Input a multi-layer bidirectional LSTM network, and then extract the internal states of all layers of LSTM networks for weighted sum, as shown in formula (2):

[0076]

[0077] Among them, O l and C l Represent the output and internal state of the l-th layer LSTM network, w l Represents the weight components corresponding to the output and internal state of the LSTM network;

[0078] Step E213, encode the patient's personalized feature information of gender, age and hearing aid wearing years to Z = [age, gender, year used], and then discretize the patient's personalized coding features by codebook sampling, and the mapping function is expressed as Discrete coding From a codebook of V learnable entries Sampling in The index i is calculated by Gumbel-Softmax method, as shown in formula (3):

[0079]

[0080] Where n=-log(-log(u)), and uU(0,1) represents random sampling noise.

[0081] Step E22, factorized feature fusion, specifically fusion of the cross-correlation feature X corr , time series high-dimensional deep features X RNN and patient-specific characteristics In this way, the potential patterns of hearing loss in different patients are captured, and then the general features X′ of the deep sequence and the personalized features of the patients are fused through affine transformation. Get the spectrum hearing loss state deep mode feature X audio , the specific steps are as follows,

[0082] Step E221, using affine transformation to fuse the general feature X′ of the depth sequence and the personalized feature of the patient Get the feature X after affine transformation fusion stage1 , as shown in formula (4),

[0083]

[0084] in, represents batch normalization, Θ represents the model parameters of the fully connected layer;

[0085] Step E222, through the parameter matrix W fa (b) = [w 1 ,…,w D ](b∈[1,…,B])X stage1 Mapped to D different subspaces Then X corr It is also mapped to D subspaces and used as the weight of each space

[0086] Step E223, using a weighted sum method to fuse and obtain the deep pattern features of the spectrum hearing loss state As shown in formula (5),

[0087]

[0088] Step E3, integrating the text global similarity deep pattern feature V txt and spectrum hearing loss status deep pattern feature X audio And obtain the multi-target fusion feature X fusion , and then obtain the fused feature X fusion The audiogram and hearing loss level are predicted as shared information. The specific steps are as follows:

[0089] Step E31, integrating the text global similarity deep pattern feature V txt and spectrum hearing loss status deep pattern feature Xaudio And obtain the multi-target fusion feature X fusion Specifically, the text feature and audio feature expressions are concatenated into a unified vector and then sent to a multimodal fusion network. The multimodal fusion network is composed of decomposed attention and multi-layer dilated convolution, as shown in formula (6).

[0090]

[0091] Step E32, using the fused features as shared information to predict the audiogram and hearing loss level, specifically, the hearing loss level is calculated based on multi-head attention and fully connected network level = FC (MHSA (X fusion )), and then the hearing loss level index and the extracted features are fused and the audiogram is calculated through full connection transformation, as shown in formula (7).

[0092] audiogram=FC(concat(X fusion ,level))(7).

[0093] A hearing loss assessment system for hearing-impaired patients based on speech audiometry comprises a model building module, a data cleaning module, a signal conversion module, a signal transformation module and a hearing loss assessment module, wherein the model building module is used to construct a speech synthesis model with original audio voiceprint features using original audio data of speech audiometry; the data cleaning module is used to clean original audio text data and user feedback text data to obtain a first audio signal; the signal conversion module is used to convert the original audio text data and user feedback text data into an audio signal using the speech synthesis model to obtain a second audio signal; the signal transformation module is used to perform Fourier transformation on the first audio signal and the second audio signal and calculate the energy of each frequency point to obtain multiple groups of audiometry audio data; the hearing loss assessment module is used to establish a hearing assessment network and input the personal information of the test subject and multiple groups of audiometry audio data into the hearing assessment network after regularization, and then use the deep speech connection, spectrum clues and global patterns of different audiometry samples to perform a comprehensive a posteriori hearing state assessment to complete the hearing loss assessment task.

[0094] In order to better illustrate the use effect of the present invention, Figure 4 As shown, 30 testers were selected for verification, including 8 patients with moderate hearing loss, 16 patients with moderate to severe hearing loss and 6 patients with severe hearing loss; the test results showed that the maximum deviation of the prediction results of the audiogram by this method reached 7dB.

[0095] In summary, the present invention provides a method and system for evaluating hearing loss of hearing-impaired patients based on speech audiometry. First, the original audio data of speech audiometry is used to construct a speech synthesis model with original audio voiceprint features, and then the original audio text data and user feedback text data are cleaned to obtain a first audio signal. Then, the speech synthesis model is used to convert the original audio text data and the user feedback text data into an audio signal to obtain a second audio signal. Then, the first audio signal and the second audio signal are Fourier transformed and the energy of each frequency point is calculated to obtain multiple groups of audiometry audio data. Subsequently, a hearing evaluation network is established and the personal information of the listener and the multiple groups of audiometry audio data are input into the hearing evaluation network after being regularized. Then, the deep speech connection, spectral clues and global patterns of different audiometry samples are used to perform a comprehensive a posteriori hearing state evaluation and complete the hearing loss evaluation operation. The hearing loss evaluation method and system for hearing-impaired patients effectively realize that the function of automatically evaluating the patient's hearing state is achieved without the need for an additional pure tone audiometry process. At the same time, the method for automatically evaluating the patient's hearing state based on the speech audiometry results can integrate the patient's personality information for evaluation, which not only has a high evaluation efficiency, but also makes the evaluation results more accurate.

[0096] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for evaluating hearing loss of hearing-impaired patients based on speech audiometry, characterized in that: The following steps are included: Step A, using the original audio data of speech audiometry to construct a speech synthesis model with the original audio voiceprint features; Step B, cleaning the original audio text data and user feedback text data; Step C, using a speech synthesis model to convert the original audio text data and the user feedback text data into audio signals to obtain a first audio signal and a second audio signal; Step D, performing Fourier transform on the first audio signal and the second audio signal and calculating the energy of each frequency point to obtain multiple groups of audiometry audio data; Step E: Establish a hearing assessment network and input the listener's personal information and multiple sets of audio data into the hearing assessment network after regularization. Then, use the deep speech connection, spectral clues and global patterns of different audio samples to conduct a comprehensive a posteriori hearing status assessment to complete the hearing loss assessment task.

2. The method for evaluating hearing loss of hearing-impaired patients based on speech audiometry according to claim 1, characterized in that: Step A: Use the original audio data of speech audiometry to build a speech synthesis model with the original audio voiceprint features, wherein the speech synthesis model adopts the GPT-SoVITS architecture, and the speech synthesis model is trained using the speech audiometry dataset.

3. The method for evaluating hearing loss of hearing-impaired patients based on speech audiometry according to claim 2, characterized in that: Step B, cleaning the original audio text data and user feedback text data. The specific cleaning process is to remove the audio content samples that the user cannot repeat and play at all, and then replace the erroneous content in the repeated text with empty samples and align them with the successfully repeated text.

4. The method and system for assessing hearing loss of hearing-impaired patients based on speech audiometry according to claim 3, characterized in that: Step C, using a speech synthesis model to convert the original audio text data and the user feedback text data into audio signals to obtain a first audio signal and a second audio signal, specifically, performing speech synthesis on the aligned original audio text data and the user feedback text data according to timestamps.

5. The method for evaluating hearing loss of hearing-impaired patients based on speech audiometry according to claim 4, characterized in that: Step D, performing Fourier transform on the first audio signal and the second audio signal and calculating the energy of each frequency point to obtain multiple sets of audio measurement data. The specific steps are as follows: Step D1, windowing the first audio signal and the second audio signal, and then mapping the audio from the time domain to the frequency domain through a fast Fourier transform operator; Step D2: taking the square root of the real part and the imaginary part of each frequency point as the energy of the frequency point.

6. The method for evaluating hearing loss of hearing-impaired patients based on speech audiometry according to claim 5, characterized in that: Step E, establish a hearing evaluation network and input the listener's personal information and multiple groups of audio data into the hearing evaluation network after regularization, and then use the deep speech connection, spectral clues and global patterns of different audio samples to perform a comprehensive posterior hearing state evaluation to complete the hearing loss evaluation operation, wherein the personal information includes the patient's gender, age and years of hearing aid use, and the hearing evaluation network is used to receive inputs in different formats of text and audio to evaluate the posterior hearing state estimation map and hearing loss level. The specific steps are as follows: Step E1, receiving text input, specifically using the pre-trained word embedding language model to encode the original text and the user feedback text into vector space and respectively use V l and V u Indicates that, then for the input vector V l and V u Use full connection to calculate the global similarity deep pattern feature V across frames txt , as shown in formula (1), Among them, tanh represents the activation function, represents Layer Normalization, FC represents the fully connected layer; Step E2, receiving audio input, specifically adopting multi-domain feature extraction and factorized feature fusion to extract deep pattern features of user spectral hearing loss status, the specific steps are as follows: Step E21, multi-domain feature extraction, specifically, using the cross-correlation feature space and the time-series high-dimensional deep feature space to extract the cross-correlation feature X from the input sub-band energy information corr , time series high-dimensional deep features X RNN and patient-specific characteristics Step E22, factorized feature fusion, specifically fusion of the cross-correlation feature X corr , time series high-dimensional deep features X RNN and patient-specific characteristics In this way, the potential patterns of hearing loss in different patients are captured, and then the general features X′ of the deep sequence and the personalized features of the patients are fused through affine transformation. Get the spectrum hearing loss state deep mode feature X audio ; Step E3, integrating the text global similarity deep pattern feature V txt and spectrum hearing loss status deep pattern feature X audio And obtain the multi-target fusion feature X fusion , and then obtain the fused feature X fusion Audiogram and hearing loss grade are predicted as shared information.

7. The method for evaluating hearing loss of hearing-impaired patients based on speech audiometry according to claim 6, characterized in that: The specific steps of step E21 are as follows: Step E211, extracting sub-band cross-correlation features X corr (b), as shown in formula (1), Among them, b∈[1,...,B] represents the divided subband, and Represent the sub-band energies of the hearing-impaired patient and the original audio respectively; Step E212, extracting time series high-dimensional depth features X RNN , specifically, the concatenation sequence of the hearing-impaired patient and the original audio subband energy vector in the subband dimension Input a multi-layer bidirectional LSTM network, and then extract the internal states of all layers of LSTM networks for weighted sum, as shown in formula (2): Among them, O l and C l Represent the output and internal state of the l-th layer LSTM network, w l Represents the weight components corresponding to the output and internal state of the LSTM network; Step E213, encode the patient's personalized feature information of gender, age and hearing aid wearing years to Z = [age, gender, year used], and then discretize the patient's personalized coding features by codebook sampling, and the mapping function is expressed as Discrete coding From a codebook of V learnable entries Sampling in The index i is calculated by Gumbel-Softmax method, as shown in formula (3): Where n=-log(-log(u)), and uU(0,1) represents random sampling noise.

8. The method for evaluating hearing loss of hearing-impaired patients based on speech audiometry according to claim 7, characterized in that: The specific steps of step E22 are as follows: Step E221, using affine transformation to fuse the general feature X′ of the depth sequence and the personalized feature of the patient Get the feature X after affine transformation fusion stage1 , as shown in formula (4), in, represents batch normalization, Θ represents the model parameters of the fully connected layer; Step E222, through the parameter matrix W fa (b) = [w 1 ,…,w D ](b∈[1,…,B])X stage1 Mapped to D different subspaces Then X corr It is also mapped to D subspaces and used as the weight of each space Step E223, using a weighted sum method to fuse and obtain the deep pattern features of the spectrum hearing loss state As shown in formula (5), 9. The method for evaluating hearing loss of hearing-impaired patients based on speech audiometry according to claim 8, characterized in that: The specific steps of step E3 are as follows: Step E31, integrating the text global similarity deep pattern feature V txt and spectrum hearing loss status deep pattern feature X audio And obtain the multi-target fusion feature X fusion Specifically, the text feature and audio feature expressions are concatenated into a unified vector and then sent to a multimodal fusion network. The multimodal fusion network is composed of decomposed attention and multi-layer dilated convolution, as shown in formula (6). Step E32, using the fused features as shared information to predict the audiogram and hearing loss level, specifically, the hearing loss level is calculated based on multi-head attention and fully connected network level = FC (MHSA (X fusion )), and then the hearing loss level index and the extracted features are fused and the audiogram is calculated through full connection transformation, as shown in formula (7). audiogram=FC(concat(X fusion ,level))(7)。 10. A hearing loss assessment system for hearing-impaired patients based on speech audiometry, wherein the hearing loss assessment process of the hearing loss assessment system for hearing-impaired patients is based on the hearing loss assessment method according to any one of claims 1 to 9, characterized in that: It includes a model building module, a data cleaning module, a signal conversion module, a signal transformation module and a hearing loss assessment module, wherein the model building module is used to construct a speech synthesis model with original audio voiceprint features using the original audio data of speech audiometry; The data cleaning module is used to clean the original audio text data and user feedback text data; The signal conversion module is used to convert the original audio text data and the user feedback text data into audio signals by using a speech synthesis model to obtain a first audio signal and a second audio signal; The signal conversion module is used to perform Fourier transformation on the first audio signal and the second audio signal and calculate the energy of each frequency point to obtain multiple groups of audiometry audio data; The hearing loss assessment module is used to establish a hearing assessment network and input the listener's personal information and multiple sets of audio data into the hearing assessment network after regularization, and then use the deep speech connection, spectral clues and global patterns of different audio samples to perform a comprehensive a posteriori hearing state assessment to complete the hearing loss assessment task.

Images