Cantonese-mandarin aphasia automatic evaluation method based on transfer learning

Abstract

The invention discloses a Cantonese-Mandarin aphasia automatic evaluation method based on transfer learning, which can make full use of Mandarin and Cantonese voice data to carry out knowledge transmission of different languages, does not need to label the Cantonese and Mandarin voice data, and can carry out aphasia detection in a Cantonese-Mandarin scene. The method comprises the following steps: extracting voice features, semantic features and grammar features of two languages from voice data and text data transcribed from the voice data, and separating the extracted features into common features and private features; common parts of voice features, semantic features and grammatical features of Cantonese and Mandarin are aligned by introducing confrontation domain self-adaption, and mutual information of input voice and text data and extracted voice features, semantic features and grammatical features is maximized by adopting an InfoNCE loss function in contrast learning. Therefore, the extracted features are representative, and the evaluation precision of aphasia is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of aphasia, and in particular to a Cantonese-Mandarin aphasia automatic evaluation method based on transfer learning. Background Art

[0002] Aphasia is a common acquired language disorder caused by dysfunction in specific areas of the brain. It is often associated with neurological diseases such as Alzheimer's disease and stroke. Due to the aggravation of the aging population, the number of patients with these diseases has been increasing year by year. Aphasia can cause defects in the patient's ability to understand and express language, especially the understanding of key parts such as vocabulary and grammar in sentences, which in turn causes a decline in language cognitive ability. The severity of these symptoms depends on the location of the injured area of ​​the brain. Its presence can have a significant negative impact on the patient's daily communication, and cause the patient to have severe inferiority complex and be isolated from society. Therefore, the prevention and treatment of aphasia has become an important public health issue.

[0003] At present, the commonly used aphasia assessment method is mainly to conduct targeted tests on patients by speech-language pathologists (SLPs) combined with corresponding assessment scales. Commonly used scales include the Chinese Standard Aphasia Examination Scale and Chinese Aphasia Battery of the China Rehabilitation Research Center, as well as the Boston Diagnostic Aphasia Examination and Western Aphasia Battery of foreign countries. Although these assessment scales have been clinically tested for a long time and can comprehensively evaluate the strengths and weaknesses of a subject's language ability, they usually take several hours to complete the entire test, and even the shortened version of the test takes at least half an hour. Many aphasia patients, especially those with severe aphasia, are in poor physical and mental condition and usually cannot tolerate the long test process. Moreover, patients often face serious communication difficulties during the test, which can easily lead to emotional breakdown. Even if the entire test is divided into several days, it is easy for patients to have a rejection mentality, so they are unwilling to cooperate with the test and miss the golden opportunity for rehabilitation. In addition, the results of this evaluation method also depend on the experience and ability of SLPs, and are very dependent on subjective judgment, resulting in low accuracy of the evaluation results. Mandarin and Cantonese are two important languages ​​in Chinese. There have been a lot of studies on the automatic detection and evaluation of English aphasia, but there are relatively few studies on the automatic detection and evaluation of Chinese aphasia. The current domestic aphasia evaluation system is far from meeting medical needs, and there is an urgent need to change the traditional manual evaluation model. Using computer technology to assist in the analysis of aphasia is one of the effective ways to address the needs of many patients in my country. Accurate automated aphasia evaluation methods can assist SLPs in evaluation, relieve their work pressure, improve the evaluation efficiency of patients, make the evaluation results of patients more standardized, and effectively save the cost of treatment.

[0004] In recent years, with the rapid development of computer technology and information technology, automated diagnostic technology has also been applied to the evaluation of aphasia, making the diagnosis of aphasia more automated and intelligent. Based on medical theory, domestic and foreign scholars have captured the audio features of patients through artificial intelligence technology, realized the automatic evaluation of the severity of patients' aphasia, and greatly reduced the subjectivity of manual diagnosis. At present, the mainstream research methods are mainly divided into acoustic feature modeling for aphasia and aphasia evaluation algorithms based on deep learning. The acoustic feature modeling methods for aphasia are mainly divided into two directions: feature extraction and feature selection. In feature extraction, speech features such as pitch, formant, Mel frequency cepstral coefficient, linear prediction cepstral coefficient, etc. are often directly extracted from the original audio, but the extracted features are uneven, these features are easily disturbed by noise, and are not robust. In feature selection, different feature selection algorithms are used to screen features, map features from high dimensions to low dimensions, and filter out redundant features. However, due to the weak representation ability of acoustic features themselves, the performance improvement of the model after screening is also limited. Deep learning has achieved good performance in aphasia evaluation because it does not rely on manual feature selection. Deep learning extracts deep features through different network structures, but there are still the following problems: different languages ​​are often distributed differently and it is difficult to obtain labeled data, and most deep learning methods focus on aphasia detection for monolingual people. This application proposes a Cantonese-Mandarin aphasia automatic assessment method based on transfer learning, which realizes aphasia assessment with only a small amount of labeled data. Summary of the invention

[0005] In view of the problems existing in the prior art, the object of the present invention is to provide a Cantonese-Mandarin aphasia automatic assessment method based on transfer learning.

[0006] To solve the above problems, the present invention adopts the following technical solutions.

[0007] The automatic evaluation method of Cantonese-Mandarin aphasia based on transfer learning includes the following steps: Step 1). Collect data: obtain a variety of speech data, including labeled Mandarin or Cantonese speech data and unlabeled Cantonese or Mandarin speech data; Step 2). Data preprocessing: convert the speech data into Mel-spectrogram, transcribe the speech data into text and encode it into vector form; Step 3). Feature extraction and separation: Use Transformer to extract the semantic, grammatical and acoustic features of the text and speech data of Mandarin and Cantonese, fuse the three features and separate them into two parts: private features and common features of Mandarin and Cantonese; Step 4). Feature alignment: Use adversarial domain adaptation to align the common parts of the acoustic features, semantic features, and grammatical features of Mandarin and Cantonese; Step 5). Maximizing mutual information: Use the InfoNCE loss function in contrastive learning to maximize the mutual information between the input data and the extracted features, ensuring that the phonetic features, semantic features, and grammatical features of Cantonese and Mandarin can represent the data characteristics of Cantonese and Mandarin; Step 6). Generate evaluation results: The common features of Cantonese and Mandarin are input into the aphasia assessor , generate evaluation results.

[0008] The voice data of step 1) is as shown in formula (1): (1) in The length of the voice data.

[0009] In the step 2), the voice data Divide into overlapping short time windows, apply Fourier transform to each window, and use Hamming window function To split the signal we can get the complex spectrum: (2) in is the complex spectrum, is the time offset, is the frequency; Map the frequency axis to the Mel scale through the filter bank to obtain the Mel spectrum : (3) in is a Mel filter; Take the logarithm of the Mel spectrum and get the Mel spectrum graph: (4) Transcribe the speech signal into text and encode it into vector form: (5) in is the length of the text sequence.

[0010] Feature extraction in step 3): using a Transformer-based model Extract acoustic features from spectrograms and transcripts , semantic features and grammatical features , where the structure of Transformer is as follows: (6).

[0011] The feature fusion: the acoustic features are combined in series , semantic features and grammatical features Blending Together: (7) The fused features are separated into private features and common features, and the private features and common features of each modality are extracted using private feature extractors and common feature extractors: (8) in yes , and are the corresponding private and common features; The common feature extractor is used to extract the common features of Cantonese and Mandarin. The MMD metric is used to minimize the distribution difference between Cantonese and Mandarin. During the training process, To reduce the number of Cantonese and Mandarin speakers The distribution difference of It can be expressed as: (9) in and The number of samples representing Cantonese and Mandarin, represents the kernel function, The private feature extractor separates private features from common features by increasing the distribution difference between Cantonese and Mandarin. During the training process, the private feature extractor maximizes and Optimize private feature extractors, and It can be expressed as: (10) (11).

[0012] In step 4), the goal of adversarial domain adaptation is to make the features extracted by the common feature extractor not only classify Mandarin aphasia well, but also generate features similar to Mandarin in Cantonese, thereby deceiving the discriminator: (12) in, and This is the data distribution of Mandarin and Cantonese.

[0013] In step 5), when the input is When we get the common features and private features , since the features are divided into common features and private features Two parts, add the two together as input, that is: (13) To maximize Cantonese and Mandarin voice and text data and Features The mutual information between them defines a density ratio function : (14) To calculate the density ratio, the feature and input data The dimensions should be the same, using a fully connected network The characteristics The dimension is transformed to the dimension of the input data, and the transformed features are used = The density ratio function is estimated by the dot product between the original input and the original input, which can be expressed as follows: = (15) In order to maximize the mutual information, the InfoNCE loss in contrastive learning is used. InfoNCE maximizes the mutual information between features and input data by comparing positive and negative pairs. The InfoNCE loss is calculated as follows: (16) in represents the positive sample pairing, ( ) represents negative sample pairing, and InfoNCE loss can be used to represent the feature Mutual information with input data: (17) in, Represents input data Its corresponding features The mutual information of Represents the number of samples in a mini-batch. Mutual information can be maximized by minimizing the InfoNCE loss. , thereby ensuring that the phonetic features, semantic features, and grammatical features of Cantonese and Mandarin can represent the data characteristics of Cantonese and Mandarin.

[0014] In step 6), the common features of Cantonese and Mandarin are input into the aphasia assessor , generate evaluation results, the evaluation results are the scores of the severity of aphasia, and the mean square error (MSE) is used as the prediction loss function : (18) in is the score predicted by the model, It’s a real rating.

[0015] Beneficial Effects of the Invention Compared with the prior art, the advantages of the present invention are: (1) The present invention is used to realize automatic aphasia assessment in a Cantonese-Mandarin scenario with only a small amount of labeled data. This method can make full use of Mandarin (Cantonese) speech data to transfer knowledge between different languages, and does not require labeling of Cantonese (Mandarin) speech data. Aphasia detection can be performed in a Cantonese-Mandarin scenario. By extracting the phonetic features, semantic features and grammatical features of the two languages ​​from the speech data and the text data transcribed from the speech data, and separating the extracted features into common features and private features; (3) Adversarial domain adaptation is introduced to align the common parts of the speech features, semantic features, and grammatical features of Cantonese and Mandarin. The InfoNCE loss function in contrastive learning is used to maximize the mutual information between the input speech and text data and the extracted speech features, semantic features, and grammatical features, ensuring that the extracted features are representative, which greatly improves the accuracy of aphasia assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0017] Figure 2 This is a structural diagram of the Transformer of the present invention. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments, and all other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making creative work are within the scope of protection of the present invention.

[0019] The automatic evaluation method of Cantonese-Mandarin aphasia based on transfer learning includes the following steps. The specific process is shown in the attached Figure 1 As shown: Step 1). Collect data: Obtain a variety of speech data, including labeled Mandarin (Cantonese) speech data and unlabeled Cantonese (Mandarin) speech data.

[0020] Step 2). Data preprocessing: convert the speech data into mel-spectrogram, transcribe the speech data into text and encode it into vector form.

[0021] Step 3). Feature extraction and separation: Use Transformer to extract the semantic, grammatical and acoustic features of text and speech data in Mandarin and Cantonese, fuse the three features and separate them into private features and common features of Mandarin and Cantonese.

[0022] Step 4). Feature alignment: Use adversarial domain adaptation to align the common parts of the acoustic features, semantic features, and grammatical features of Mandarin and Cantonese.

[0023] Step 5). Maximizing mutual information: The InfoNCE loss function in contrastive learning is used to maximize the mutual information between the input data and the extracted features, ensuring that the phonetic features, semantic features, and grammatical features of Cantonese and Mandarin can represent the data characteristics of Cantonese and Mandarin.

[0024] Step 6). Generate evaluation results: Use the model trained in the above steps to perform aphasia evaluation on the test data.

[0025] The voice data in step 1) is as shown in formula (1): (1) in The length of the voice data.

[0026] Step 2) Data preprocessing: The speech data is converted into mel-spectrograms, and the speech data is transcribed into text and encoded into vector form.

[0027] Voice data Divide into overlapping short time windows, apply Fourier transform to each window, and use Hamming window function To split the signal we can get the complex spectrum: (2) in is the complex spectrum, is the time offset, It's the frequency.

[0028] Map the frequency axis to the Mel scale through the filter bank to obtain the Mel spectrum : (3) in is a Mel filter; Take the logarithm of the Mel spectrum and get the Mel spectrum graph: (4) Transcribe the speech signal into text and encode it into vector form: (5) in is the length of the text sequence.

[0029] Step 3) Feature extraction and separation: Use Transformer to extract semantic, syntactic, and acoustic features from the text and speech data of Mandarin and Cantonese. After fusing the three features, separate them into two parts: the private features and the common features of Mandarin and Cantonese.

[0030] Acoustic features refer to the physical properties related to pronunciation in the speech signal, such as frequency, pitch, duration, intensity, etc. These features are mainly represented by the waveform and spectrum of the speech signal. For example, Mandarin has four tones, and Cantonese has six to nine tones. Semantic features refer to the meaning level of language units (such as words, phrases, sentences). In Cantonese and Mandarin, many words have the same meaning but different pronunciations; some words have the same pronunciation but different meanings; and there are also some words with different meanings in the two language systems. Syntactic features involve the structural rules of language, including word order, syntactic structure, tense, mood, etc. Although Cantonese and Mandarin are similar in basic word order, there are differences in some syntactic structures and expressions. For example, in Mandarin, words such as "了", "过", "正在" are used to indicate tenses such as completion, experience, and ongoing, while in Cantonese, words such as "咗", "过", "紧" are used to indicate similar tenses. Generally speaking, Mandarin and Cantonese have both similarities and differences in acoustic features, semantic features, and syntactic features.

[0031] Feature extraction: Use a Transformer-based model to extract acoustic features from spectrograms and transcribed texts , semantic features , and syntactic features Figure 2 , where the structure of Transformer is as

[0032] (6) Feature fusion: Fuse acoustic features , semantic features , and syntactic features together through concatenation: (7) Separate the fused features into private features and common features, and use a private feature extractor and a common feature extractor to extract the private features and common features in each modality: (8) where is , and are the corresponding private features and common features.

[0033] The common feature extractor is mainly used to extract the common features of Cantonese and Mandarin. Here we use the MMD metric to minimize the distribution difference between Cantonese and Mandarin. During the training process, we minimize To reduce the number of Cantonese and Mandarin speakers The distribution difference of It can be expressed as: (9) in and The number of samples representing Cantonese and Mandarin, Represents the kernel function.

[0034] The private feature extractor separates private features from common features by increasing the distribution difference between Cantonese and Mandarin. During the training process, we maximize and Optimize private feature extractors, and It can be expressed as: (10) (11) Step 4) Feature alignment and separation: Use adversarial domain adaptation to align the common parts of the acoustic features, semantic features, and grammatical features of Mandarin and Cantonese.

[0035] The goal of adversarial domain adaptation is to make the features extracted by the common feature extractor not only classify Mandarin aphasia well, but also generate features similar to Mandarin in Cantonese, thereby deceiving the discriminator: (12) in, and This is the data distribution of Mandarin and Cantonese.

[0036] Step 5) Maximizing mutual information: The InfoNCE loss function in contrastive learning is used to maximize the mutual information between the input data and the extracted features, ensuring that the phonetic features, semantic features, and grammatical features of Cantonese and Mandarin can represent the data characteristics of Cantonese and Mandarin.

[0037] Specifically, we applied the InfoNCE loss function in contrastive learning to maximize the mutual information between the extracted features and the input data, where mutual information means the amount of information contained in one random variable about another random variable. If the extracted features contain more information about the input data, that is, the greater the mutual information, the more the extracted features can represent the data characteristics of the language.

[0038] When input is From this we can get the common characteristics and private features , since the features are divided into common features and private features Two parts, here we add the two as input, that is: (13) To maximize Cantonese and Mandarin voice and text data and Features The mutual information between them, we define a density ratio function : (14) To calculate the density ratio, the feature and input data The dimensions should be the same, so we use a fully connected network with one layer The characteristics The dimension is transformed to the dimension of the input data. We use the transformed features = The density ratio function is estimated by the dot product between the original input and the original input, which can be expressed as follows: = (15) To maximize the mutual information, we use the InfoNCE loss in contrastive learning. InfoNCE maximizes the mutual information between features and input data by contrasting positive and negative pairs. The InfoNCE loss is calculated as follows: (16) in represents the positive sample pairing, ( ) represents negative sample pairing. According to previous research, we can use InfoNCE loss to represent the feature Mutual information with input data: (17) in, Represents input data Its corresponding features The mutual information of Represents the number of samples in a mini-batch. We can see that mutual information can be maximized by minimizing the InfoNCE loss. , thereby ensuring that the phonetic features, semantic features, and grammatical features of Cantonese and Mandarin can represent the data characteristics of Cantonese and Mandarin.

[0039] 6) Evaluation module Common features of Cantonese and Mandarin were input into the aphasia assessor , generating an evaluation result, which is a score of the severity of aphasia. The present invention uses the mean square error (MSE) as the prediction loss function : (18) in is the score predicted by the model, It’s a real rating.

[0040] In summary, the present invention proposes a Cantonese-Mandarin aphasia automatic assessment method based on transfer learning. First, the features (acoustic, semantic, and speech features) under different modalities are obtained through speech data, and the common features and private features are extracted through common feature extractors and private feature extractors after the features are fused; in order to better align the common features between Cantonese and Mandarin, adversarial domain adaptation is used to reduce the distribution differences of acoustic features, semantic features, and speech features between Cantonese and Mandarin; in order to ensure that the speech features, semantic features, and grammatical features of Cantonese and Mandarin can represent the data characteristics of Cantonese and Mandarin, the InfoNCE loss function in contrastive learning is used to maximize the mutual information between the input speech data and the extracted features, thereby improving the assessment accuracy of aphasia in the Cantonese-Mandarin scenario.

[0041] The above is only a preferred specific implementation of the present invention; however, the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solution and its improved conception within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A Cantonese-Mandarin aphasia automatic assessment method based on transfer learning, characterized in that: The following steps are involved: Step 1). Collect data: obtain a variety of speech data, including labeled Mandarin or Cantonese speech data and unlabeled Cantonese or Mandarin speech data; Step 2). Data preprocessing: convert the speech data into Mel-spectrogram, transcribe the speech data into text and encode it into vector form; Step 3). Feature extraction and separation: Use Transformer to extract the semantic, grammatical and acoustic features of the text and speech data of Mandarin and Cantonese, fuse the three features and separate them into two parts: private features and common features of Mandarin and Cantonese; Step 4). Feature alignment: Use adversarial domain adaptation to align the common parts of the acoustic features, semantic features, and grammatical features of Mandarin and Cantonese; Step 5). Maximizing mutual information: Use the InfoNCE loss function in contrastive learning to maximize the mutual information between the input data and the extracted features, ensuring that the phonetic features, semantic features, and grammatical features of Cantonese and Mandarin can represent the data characteristics of Cantonese and Mandarin; Step 6). Generate evaluation results: The common features of Cantonese and Mandarin are input into the aphasia assessor , generate evaluation results.

2. The method for automatic assessment of Cantonese-Mandarin aphasia based on transfer learning according to claim 1, characterized in that: The voice data of step 1) is as shown in formula (1): (1) in The length of the voice data.

3. The method for automatic assessment of aphasia between Cantonese and Mandarin based on transfer learning according to claim 2, characterized in that: In the step 2), the voice data Divide into overlapping short time windows, apply Fourier transform to each window, and use Hamming window function To split the signal we can get the complex spectrum: (2) in is the complex spectrum, is the time offset, is the frequency; Map the frequency axis to the Mel scale through the filter bank to obtain the Mel spectrum : (3) in is a Mel filter; Take the logarithm of the Mel spectrum and get the Mel spectrum graph: (4) Transcribe the speech signal into text and encode it into vector form: (5) in is the length of the text sequence.

4. The method for automatic assessment of Cantonese-Mandarin aphasia based on transfer learning according to claim 3, characterized in that: Feature extraction in step 3): using a Transformer-based model Extract acoustic features from spectrograms and transcripts , semantic features and grammatical features , where the structure of Transformer is as follows: (6).

5. The method for automatic assessment of Cantonese-Mandarin aphasia based on transfer learning according to claim 4, characterized in that: The feature fusion: the acoustic features are combined in series , semantic features and grammatical features Blending Together: (7) The fused features are separated into private features and common features, and the private features and common features of each modality are extracted using private feature extractors and common feature extractors: (8) in yes , and are the corresponding private and common features; The common feature extractor is used to extract the common features of Cantonese and Mandarin. The MMD metric is used to minimize the distribution difference between Cantonese and Mandarin. During the training process, To reduce the number of Cantonese and Mandarin speakers The distribution difference of It can be expressed as: (9) in and The number of samples representing Cantonese and Mandarin, represents the kernel function, The private feature extractor separates private features from common features by increasing the distribution difference between Cantonese and Mandarin. During the training process, the private feature extractor maximizes and Optimize private feature extractors, and It can be expressed as: (10) (11)。 6. The method for automatic assessment of Cantonese-Mandarin aphasia based on transfer learning according to claim 5, characterized in that: In step 4), the goal of adversarial domain adaptation is to make the features extracted by the common feature extractor not only classify Mandarin aphasia well, but also generate features similar to Mandarin in Cantonese, thereby deceiving the discriminator: (12) in, and This is the data distribution of Mandarin and Cantonese.

7. The method for automatic assessment of aphasia between Cantonese and Mandarin based on transfer learning according to claim 6, characterized in that: In step 5), when the input is When we get the common features and private features , since the features are divided into common features and private features Two parts, add the two together as input, that is: (13) To maximize Cantonese and Mandarin voice and text data and Features The mutual information between them defines a density ratio function : (14) To calculate the density ratio, the feature and input data The dimensions should be the same, using a fully connected network The characteristics The dimension is transformed to the dimension of the input data, and the transformed features are used = The density ratio function is estimated by the dot product between the original input and the original input, which can be expressed as follows: = (15) In order to maximize the mutual information, the InfoNCE loss in contrastive learning is used. InfoNCE maximizes the mutual information between features and input data by comparing positive and negative pairs. The InfoNCE loss is calculated as follows: (16) in represents the positive sample pairing, ( ) represents negative sample pairing, and InfoNCE loss can be used to represent the feature Mutual information with the input data: (17) in, Represents input data Its corresponding features The mutual information of Represents the number of samples in a mini-batch. Mutual information can be maximized by minimizing the InfoNCE loss. , thereby ensuring that the phonetic features, semantic features, and grammatical features of Cantonese and Mandarin can represent the data characteristics of Cantonese and Mandarin.

8. The method for automatic assessment of aphasia between Cantonese and Mandarin based on transfer learning according to claim 7, characterized in that: In step 6), the common features of Cantonese and Mandarin are input into the aphasia assessor , generating evaluation results, which are the scores of the severity of aphasia, using mean square error as the prediction loss function : (18) in is the score predicted by the model, It’s a real rating.

Images