Pronunciation recognition learning auxiliary method and system based on speech processing

Abstract

The present invention discloses a pronunciation recognition learning assistance method and system based on speech processing, which relates to the technical field of speech pronunciation recognition, and comprises the following steps: recognizing a user's input speech through speech recognition and extracting input text; constructing a pronunciation feature extraction model; extracting the pronunciation features of the input speech and input text through the pronunciation feature extraction model to obtain speech features and text features respectively; performing pronunciation training on the user, and comparing the differences between the speech features and the text features to determine whether the user's pronunciation is standard. The present invention is used to solve the problem that the existing speech pronunciation recognition technology still has an unreasonable method for determining whether the user's pronunciation is standard and the data in the analysis process does not have characteristic features, resulting in the inability to effectively help the user to learn and correct pronunciation.

Description

Technical Field

[0001] The present invention relates to the technical field of pronunciation recognition, and in particular to a pronunciation recognition learning auxiliary method and system based on speech processing. Background Art

[0002] Speech pronunciation recognition technology refers to a technology that converts human speech into text or instructions by analyzing the characteristics of speech signals, combining acoustic models and language models, and evaluating the accuracy of pronunciation, thereby assisting users in optimizing pronunciation learning.

[0003] Speech pronunciation recognition technology is commonly used to help users learn pronunciation. However, existing speech pronunciation recognition technology only recognizes the correct text based on the user's speech and considers the user's pronunciation correct. Pronunciation learners are typically students or adults, not children. Therefore, pronunciation recognition requires more stringent judgment criteria to help users correct their pronunciation. Furthermore, existing speech pronunciation recognition technology does not convert speech into characteristic data and extract its pronunciation features when processing speech, resulting in inaccurate judgment of whether the user's pronunciation is standard. For example, patent application publication number CN109461436A discloses a "method and system for correcting speech recognition pronunciation errors." This solution does not provide a detailed analysis process for determining whether a user's pronunciation is standard. It only uses speech recognition text and the correctness of the recognized text to determine whether the user's pronunciation is standard. However, pronunciation learners are typically not children, and more stringent judgment criteria are required to help users correct their pronunciation. Existing speech pronunciation recognition technology also suffers from unreasonable judgment methods for determining whether a user's pronunciation is standard and the lack of characteristic data in the analysis process, resulting in an inability to effectively help users learn and correct their pronunciation. Summary of the Invention

[0004] The present invention aims to solve one of the technical problems in the prior art to at least a certain extent, and recognizes the user's input voice through voice recognition, extracts input text, pre-processes the voice to be analyzed, obtains a time domain graph of the voice to be analyzed, frames it, divides the time domain graph into different frame data, and then performs short-time Fourier transform on the frame data to convert it into a spectrum of the frame data, named frame spectrum, converts the input voice into a spectrogram based on the frame spectrum, and then extracts the pronunciation features of the voice to be analyzed based on the spectrogram, extracts the pronunciation features of the input voice and input text through a pronunciation feature extraction model, and obtains voice features and text features respectively, finally performs pronunciation training on the user, and compares the differences between the voice features and text features to determine whether the user's pronunciation is standard, so as to solve the problem that the existing voice pronunciation recognition technology still has an unreasonable way of judging whether the user's pronunciation is standard and the data in the analysis process does not have characteristics, resulting in an inability to effectively help users learn and correct pronunciation.

[0005] To achieve the above objectives, in a first aspect, the present application provides a pronunciation recognition learning assistance method based on speech processing, comprising the following steps:

[0006] Recognize the user's input voice through voice recognition and extract the input text;

[0007] Construct a pronunciation feature extraction model to convert the speech to be analyzed into a spectrogram and extract the pronunciation features in the spectrogram;

[0008] The pronunciation features of the input speech and input text are extracted through the pronunciation feature extraction model to obtain speech features and text features respectively;

[0009] Provide pronunciation training for users, and compare the differences in voice features and text features to determine whether the user's pronunciation is standard.

[0010] Furthermore, the speech recognition is performed using speech-to-text technology to convert the input speech into text, which is named input text.

[0011] Furthermore, constructing a pronunciation feature extraction model, converting the speech to be analyzed into a spectrogram and extracting pronunciation features from the spectrogram includes the following sub-steps:

[0012] After preprocessing the speech to be analyzed, a time domain image of the speech to be analyzed is obtained, and the time domain image is divided into different frame data;

[0013] Perform short-time Fourier transform on the frame data to convert it into a spectrum of the frame data, named frame spectrum, and convert the input speech into a spectrogram based on the frame spectrum;

[0014] Extract the pronunciation features of the speech to be analyzed based on the spectrogram.

[0015] Furthermore, after the speech to be analyzed is preprocessed, a time domain graph of the speech to be analyzed is obtained and framed. Dividing the time domain graph into different frame data includes the following sub-steps:

[0016] Construct a pronunciation feature extraction model, and name the speech input into the pronunciation feature extraction model as the speech to be analyzed;

[0017] After filtering the speech to be analyzed through a first-order high-pass filter, a time domain image of the speech to be analyzed is obtained;

[0018] Set the frame length, marked as n, obtain the speech duration of the speech to be analyzed, marked as T, calculate T / n, and mark the result as m, where m is rounded up to an integer;

[0019] The time domain diagram is evenly divided into m parts, each part is a frame data, and the frame data is marked as F(n,m). F(n,m) represents the waveform in the time domain diagram within the time range of [n×mn,n×m].

[0020] Furthermore, the step of performing short-time Fourier transform on the frame data to convert the frame data into a spectrum of the frame data is named as a frame spectrum, and converting the input speech into a spectrogram based on the frame spectrum includes the following sub-steps:

[0021] Performing a short-time Fourier transform on F(n,m) to convert it into a frame spectrum of F(n,m), which is labeled S(n,m). The frame spectrum is specifically a two-dimensional coordinate system with the X axis being the frequency and the Y axis being the amplitude value;

[0022] Get the minimum and maximum amplitude values, mark them as A1 and A2 respectively, form the range [A1, A2], and name it amplitude range;

[0023] The amplitude range is evenly divided into 256 sub-ranges, named as amplitude sub-ranges, and the amplitude sub-ranges are sorted and numbered in ascending order. i Represents, where i is a positive integer and i is the serial number of P;

[0024] There are color values ​​0 to 255, 0 represents black, 255 represents white, i The color value is set to C i , the C i is 256-i;

[0025] Name the waveform in S(n,m) the wave to be converted, mark the length of the wave to be converted on the X-axis as L, construct a line with a length of L and a width of n, name it the speech line, record the speech line into S(n,m), and align the speech line with the wave to be converted;

[0026] For any value of X in S(n,m), name the amplitude value corresponding to X as the target amplitude, and find the P to which the target amplitude belongs. i Corresponding C i , marked as CD i , change the color value of the pixel at X on the spectrum line to CD i , analyze each value of X, and the final spectrum line is a line segment with color changes;

[0027] Extract the spectrum line of S(n,m) and rotate it 90° counterclockwise to obtain Q(n,m);

[0028] A two-dimensional coordinate system is established with m as the horizontal axis and frequency as the vertical axis, named spectrogram, and Q(n,m) is entered into the spectrogram.

[0029] Furthermore, the extraction of pronunciation features of the speech to be analyzed based on the spectrogram includes the following sub-steps:

[0030] Number the input text from left to right, using the symbol W j Represents, where j is a positive integer and j is the sequence number of W;

[0031] When converting input speech into input text, mark W j The pronunciation time of the word is obtained, and the frame data within the pronunciation time is searched in the spectrogram and marked as the word pronunciation data;

[0032] The word pronunciation data is W j pronunciation characteristics.

[0033] Furthermore, the extraction of pronunciation features of the input speech and input text by the pronunciation feature extraction model includes the following sub-steps:

[0034] Find the standard pronunciation of the input text and name it as standard pronunciation;

[0035] The pronunciation features of the input speech are extracted through the pronunciation feature extraction model and named as speech features;

[0036] The pronunciation features of standard speech are extracted through the pronunciation feature extraction model and named as text features.

[0037] Furthermore, the method of performing pronunciation training on the user and comparing the differences between the voice features and the text features to determine whether the user's pronunciation is standard includes the following sub-steps:

[0038] The user inputs or selects training content;

[0039] Standard voice to announce training content to users;

[0040] Users learn standard voices and upload input voices;

[0041] Determine whether the input voice meets the standard voice.

[0042] Furthermore, the step of determining whether the input speech conforms to the standard pronunciation includes the following sub-steps:

[0043] Compare the input speech with the same W in the standard speech j The speech features and text features are summarized into a group of character feature groups;

[0044] Analyze any character feature group and mark the speech feature and text feature in the character feature group as the first feature and the second feature respectively;

[0045] The frame data in the first feature are numbered from left to right, and the symbol Z1 hIndicates that the frame data in the second feature are numbered from left to right, and the symbol Z2 h Denotes, where h is a positive integer and h is the sequence number of Z1 and Z2;

[0046] Extract Z1 in order from bottom to top h and Z2 h CD i , and form a one-dimensional matrix, marked as G1 h and G2 h ;

[0047] Calculate G1 h -G2 h , mark the calculation result as G3 h and G3 h Take the absolute value of all the values ​​in G4 h , in order from left to right, G4 h The values ​​in are numbered and represented by the symbol K1(h,t), and G2 is sorted from left to right. h The values ​​in are numbered and represented by the symbol K2(h,t), where t is a positive integer and (h,t) is the sequence number of K1 and K2;

[0048] calculate , the calculation result is named pronunciation accuracy, where max() is the maximum value operator;

[0049] Find the first number of volunteers, ask them to perform pronunciation training for the second number, and calculate the pronunciation accuracy of each training, which is named as the regular accuracy. Find the minimum value of the regular accuracy, which is named as the accuracy threshold.

[0050] The pronunciation accuracy of the user's input speech is calculated, named input accuracy, and the input accuracy is compared with the accuracy threshold. If the input accuracy is less than the accuracy threshold, the user's input speech is marked as not meeting the standard speech. Otherwise, the user's input speech is marked as meeting the standard speech.

[0051] In a second aspect, the present application provides a pronunciation recognition learning auxiliary system based on speech processing, comprising a speech input module, a pronunciation feature extraction model construction module, a pronunciation feature extraction module, and a pronunciation standard judgment module; the speech input module, the pronunciation feature extraction model construction module, and the pronunciation feature extraction module are respectively data-connected to the pronunciation standard judgment module;

[0052] The voice input module is used to recognize the user's input voice through voice recognition and extract the input text;

[0053] The pronunciation feature extraction model construction module is used to construct a pronunciation feature extraction model, convert the speech to be analyzed into a spectrogram and extract the pronunciation features in the spectrogram;

[0054] The pronunciation feature extraction module is used to extract the pronunciation features of the input speech and input text through the pronunciation feature extraction model to obtain speech features and text features respectively;

[0055] The pronunciation standard judgment module is used to perform pronunciation training on the user and compare the differences between voice features and text features to determine whether the user's pronunciation is standard.

[0056] Beneficial effects of the present invention: The present invention recognizes the user's input voice through voice recognition, extracts input text, pre-processes the voice to be analyzed, obtains a time domain graph of the voice to be analyzed, frames it, divides the time domain graph into different frame data, and then performs short-time Fourier transform on the frame data to convert it into a spectrum of the frame data, named frame spectrum, converts the input voice into a spectrogram based on the frame spectrum, and then extracts the pronunciation features of the voice to be analyzed based on the spectrogram. The advantage is that the voice to be analyzed is converted into a spectrogram, and the spectrogram shows the data of the voice to be analyzed in three dimensions of time, frequency and amplitude value. Then, effective pronunciation features are extracted from the spectrogram to judge whether the user's pronunciation is standard, thereby improving the accuracy and effectiveness of the pronunciation standard judgment;

[0057] The present invention extracts the pronunciation features of the input speech and input text through a pronunciation feature extraction model to obtain speech features and text features respectively, and finally performs pronunciation training on the user. At the same time, the differences between the speech features and the text features are compared to judge whether the user's pronunciation is standard. The advantage is that by extracting the speech features and the text features and analyzing the differences between them, if the deviation is within a reasonable range, the user's pronunciation is judged to be standard. If the deviation is too large, it means that the user's pronunciation is not standard, thereby improving the accuracy and rationality of the pronunciation standard judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a functional block diagram of the system of the present invention;

[0059] Figure 2 It is a time domain diagram of the present invention;

[0060] Figure 3 Schematic diagram of frame data F(25,1) of the present invention;

[0061] Figure 4 is a schematic diagram of a frame spectrum of the present invention;

[0062] Figure 5 A schematic diagram of placing the speech line of the present invention into the frame spectrum;

[0063] Figure 6 A schematic diagram of a speech spectrum line after the processing of the present invention is completed;

[0064] Figure 7 is a schematic diagram of a spectrogram of the present invention;

[0065] Figure 8 Flow chart of the steps of the method of the present invention. DETAILED DESCRIPTION

[0066] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0067] Example 1, please refer to Figure 1 As shown, the present application provides a pronunciation recognition learning auxiliary system based on speech processing, including a speech input module, a pronunciation feature extraction model construction module, a pronunciation feature extraction module and a pronunciation standard judgment module; the speech input module, the pronunciation feature extraction model construction module and the pronunciation feature extraction module are respectively connected to the pronunciation standard judgment module data;

[0068] The voice input module is used to recognize the user's input voice through voice recognition and extract the input text; through voice recognition, the voice-to-text technology is used to convert the input voice into text, which is named input text;

[0069] In practical applications, existing speech recognition technology is used to extract input text.

[0070] The pronunciation feature extraction model construction module is used to construct a pronunciation feature extraction model, convert the speech to be analyzed into a spectrogram and extract the pronunciation features in the spectrogram; the pronunciation feature extraction model construction module includes a framing unit, a spectrogram conversion unit and a feature extraction unit;

[0071] The framing unit is used to obtain a time domain image of the speech to be analyzed after pre-processing the speech to be analyzed, and to frame the time domain image into different frame data;

[0072] The framing unit is configured with a framing strategy, which includes:

[0073] Construct a pronunciation feature extraction model, and name the speech input into the pronunciation feature extraction model as the speech to be analyzed;

[0074] See also Figure 2 As shown, after filtering the speech to be analyzed through a first-order high-pass filter, a time domain graph of the speech to be analyzed is obtained;

[0075] Set the frame length, marked as n, obtain the speech duration of the speech to be analyzed, marked as T, calculate T / n, and mark the result as m, where m is rounded up to an integer;

[0076] The time domain graph is evenly divided into m parts, each part is a frame data, and the frame data is marked as F(n,m). F(n,m) represents the waveform in the time domain graph within the time range of [n×mn,n×m].

[0077] In practical applications, after filtering the speech to be analyzed through a first-order high-pass filter, the time domain image is obtained as follows Figure 2 As shown, the X-axis is time in seconds, and the Y-axis is amplitude. According to the existing frame length standard, the frame length is usually 10ms to 40ms. In this embodiment, the median is taken and the frame length is set to 25ms, that is, n is 25ms, and the speech duration T is 0.535s, that is, 535ms. 535 / 25=22 frame data are obtained, that is, F(25,1) to F(25,22), and m is rounded to an integer.

[0078] The spectrum conversion unit is used to perform short-time Fourier transform on the frame data and convert it into a spectrum of the frame data, named frame spectrum, and convert the input speech into a spectrogram based on the frame spectrum;

[0079] The speech spectrum conversion unit is configured with a speech spectrum conversion strategy, which includes:

[0080] See also Figures 3 and 4 As shown, F(n,m) is converted into a frame spectrum of F(n,m) by short-time Fourier transform, which is marked as S(n,m). The frame spectrum is specifically a two-dimensional coordinate system with the X axis being the frequency and the Y axis being the amplitude value.

[0081] Get the minimum and maximum amplitude values, mark them as A1 and A2 respectively, form the range [A1, A2], and name it amplitude range;

[0082] The amplitude range is evenly divided into 256 sub-ranges, named as amplitude sub-ranges, and the amplitude sub-ranges are sorted and numbered in ascending order. i Represents, where i is a positive integer and i is the serial number of P;

[0083] There are color values ​​0 to 255, 0 represents black, 255 represents white, i The color value is set to C i , C i is 256-i;

[0084] In practical applications, taking F(25,1) as an example, F(25,1) is Figure 3 As shown, it is converted to S(25,1) by short-time Fourier transform as Figure 4 As shown in the spectrum diagram, the amplitude range is [-1,1]. The amplitude range is divided into 256 amplitude sub-ranges, and [1-(-1)] / 256=0.0078125 is calculated. The span of each amplitude sub-range is 0.0078125, and the number is P. i , 1≤i≤256, if the amplitude value is within [-1,-0.9921875], it belongs to P1, and the color value C1 of P1 is 256-1=255;

[0085] See also Figure 5 As shown, the waveform in S(n,m) is named the wave to be converted, the length of the wave to be converted on the X-axis is marked as L, a line with a length of L and a width of n is constructed, named the spectrum line, and the spectrum line is recorded into S(n,m), and the spectrum line is aligned with the wave to be converted;

[0086] For any value of X in S(n,m), name the amplitude value corresponding to X as the target amplitude, and find the P to which the target amplitude belongs. i Corresponding C i , marked as CD i , change the color value of the pixel at X on the spectrum line to CD i , analyze each value of X, and the final spectrum line is a line segment with color changes;

[0087] See also Figure 6 As shown, extract the spectrum line of S(n,m) and rotate it 90° counterclockwise to obtain Q(n,m);

[0088] See also Figure 7 As shown, a two-dimensional coordinate system is established with m as the horizontal axis and frequency as the vertical axis, named spectrogram, and Q(n,m) is entered into the spectrogram;

[0089] In practical applications, the length of the wave to be converted on the X-axis is the speech duration T, and L=T=0.535 is obtained. A spectral line with a length of 0.535 and a width of n=25ms is constructed and recorded in S(25,1). The spectral line in S(25,1) is as follows: Figure 5 As shown by Figure 5 After zooming in on the middle part of the spectrum line, we can see that the spectrum line is composed of several pixel grids. According to the X-axis coordinate of each pixel grid, the amplitude value of the wave to be converted at this X value is queried as the target amplitude value, and then the P value to which the target amplitude value belongs is found. i Corresponding C i , and finally get CD i And change the color value of the corresponding pixel grid to CD i , the final spectrum line is a line segment with color changes, and the spectrum line Q(25,1) is obtained by rotating it 90° counterclockwise. Figure 6 As shown, each frame data is converted into a spectral line and combined in the order of m from small to large, and finally the spectrogram is obtained as shown in Figure 7 As shown;

[0090] The feature extraction unit is used to extract the pronunciation features of the speech to be analyzed based on the spectrogram;

[0091] The feature extraction unit is configured with a feature extraction strategy, which includes:

[0092] Number the input text from left to right, using the symbol W j Represents, where j is a positive integer and j is the sequence number of W;

[0093] When converting input speech into input text, mark W j The pronunciation time of the word is obtained, and the frame data within the pronunciation time is searched in the spectrogram and marked as the word pronunciation data;

[0094] The pronunciation data of the word is W j The pronunciation characteristics of

[0095] In actual applications, for example, if the input text is "afraid of the sun", W1 and W2 are extracted, and the pronunciation time can be directly extracted based on speech recognition technology. Among them, the pronunciation time of W1 is 0s to 0.25s, and the pronunciation time of W2 is 0.25s to 0.535s. The frame data between 0s and 0.25s is marked as the word pronunciation data of W1, and the frame data between 0.25s and 0.535s is marked as the word pronunciation data of W2.

[0096] The pronunciation feature extraction module is used to extract the pronunciation features of the input speech and input text through the pronunciation feature extraction model to obtain speech features and text features respectively;

[0097] The pronunciation feature extraction module is configured with a pronunciation feature extraction strategy, which includes:

[0098] Find the standard pronunciation of the input text and name it as standard pronunciation;

[0099] The pronunciation features of the input speech are extracted through the pronunciation feature extraction model and named as speech features;

[0100] The pronunciation features of standard speech are extracted through the pronunciation feature extraction model and named as text features;

[0101] In practical applications, standard speech is provided by the vocabulary library on the Internet, and speech features and text features are obtained through extraction.

[0102] The pronunciation standard judgment module is used to train the user's pronunciation and compare the differences between voice features and text features to determine whether the user's pronunciation is standard. The pronunciation standard judgment module includes a pronunciation training unit and a standard judgment unit.

[0103] The pronunciation training unit is configured with a pronunciation training strategy, which includes:

[0104] The user inputs or selects training content;

[0105] Standard voice to announce training content to users;

[0106] Users learn standard voices and upload input voices;

[0107] In actual applications, users can choose to input training content themselves or select built-in training content from the model. After the user completes the selection, the standard voice of the training content will be played to the user. The user learns and reads according to the standard voice, and then uploads the input voice during reading;

[0108] The standard judgment unit is used to judge whether the input speech conforms to the standard speech;

[0109] The standard judgment unit is configured with a standard judgment strategy, which includes:

[0110] Compare the input speech with the same W in the standard speech j The speech features and text features are summarized into a group of character feature groups;

[0111] Analyze any character feature group and mark the speech feature and text feature in the character feature group as the first feature and the second feature respectively;

[0112] The frame data in the first feature are numbered from left to right, and the symbol Z1 h Indicates that the frame data in the second feature are numbered from left to right, and the symbol Z2 h Denotes, where h is a positive integer and h is the sequence number of Z1 and Z2;

[0113] Extract Z1 in order from bottom to top h and Z2 h CD i , and form a one-dimensional matrix, marked as G1 h and G2 h ;

[0114] Calculate G1 h -G2 h , mark the calculation result as G3 h and G3 h Take the absolute value of all the values ​​in G4h , in order from left to right, G4 h The values ​​in are numbered and represented by the symbol K1(h,t), and G2 is sorted from left to right. h The values ​​in are numbered and represented by the symbol K2(h,t), where t is a positive integer and (h,t) is the sequence number of K1 and K2;

[0115] In actual applications, there are W in both the input speech and the standard speech. j , the two W1 of the input speech and the standard speech form a word feature group, and the two W2 form a word feature group. This embodiment takes the word feature group corresponding to W1 as an example, which includes the first feature and the second feature. W1 is within the time from 0s to 0.25s, which includes 10 frames of data, that is, the first feature and the second feature each include 10 frames of data, and the mark is Z1 h and Z2 h , 1≤h≤10, based on the CD of the spectral line i , extract in order from bottom to top, and construct a one-dimensional matrix. For example, in Z11, the color values ​​of the spectrum lines are 213, 189, 177, 205, 192 and 153 from bottom to top, and the G11 is constructed as [213 189 177 205 192 153]. h -G2 h When , h must be equal. For example, if G21 is [209 175 163 213 205 164], the calculated G31 is [4 14 14-8 -13 -11]. The absolute value of G41 is [4 14 14 8 13 11]. The numbering of K1(1,1) to K1(1,6) is 4, 14, 14, 8, 13 and 11 respectively. At the same time, the numbering of K2(1,1) to K2(1,6) is 209, 175, 163, 213, 205 and 164 respectively.

[0116] calculate , the calculation result is named pronunciation accuracy, where max() is the maximum value operator;

[0117] Find the first number of volunteers, who must have a Mandarin proficiency certificate, and require them to perform the second number of pronunciation training. Count the pronunciation accuracy of each training session, which is named the regular accuracy. Find the minimum value of the regular accuracy, which is named the accuracy threshold.

[0118] Calculate the pronunciation accuracy of the user's input speech, named input accuracy, and compare the input accuracy with the accuracy threshold. If the input accuracy is less than the accuracy threshold, mark the user's input speech as not meeting the standard speech; otherwise, mark the user's input speech as meeting the standard speech.

[0119] In actual application, each frame data in the character feature group is analyzed to finally obtain K1(h, t) and K2(h, t), where 1≤h≤10, 1≤t≤6, and the value range of t is the result of the example in this embodiment, not the actual data. In actual calculation, the value of t is large and it is inconvenient to show it in detail in this embodiment. Therefore, only part of the data is used as a case for explanation. The pronunciation accuracy is then calculated by the formula. The pronunciation accuracy reflects the similarity between the user's pronunciation and the standard pronunciation. However, in actual pronunciation, the user cannot be 100% identical to the standard pronunciation. There will always be a certain deviation. It is necessary to determine the maximum allowable deviation as a judgment basis. Therefore, a first number of volunteers are found, and the volunteers must have a Mandarin proficiency certificate. This is used to analyze the deviation range between the user and the standard pronunciation under normal circumstances. There are no requirements for the setting of the first and second numbers, and the developer can set them at will. The minimum value of the conventional accuracy is the maximum allowable deviation between the user's pronunciation and the standard voice under normal circumstances. Therefore, it is set as the accuracy threshold to determine whether the user's pronunciation is standard, thereby helping users achieve the purpose of learning standard Mandarin pronunciation.

[0120] Example 2, please refer to Figure 8 As shown, the present application provides a pronunciation recognition learning auxiliary method based on speech processing, comprising the following steps:

[0121] Step S1: Recognize the user's input voice through voice recognition and extract input text; use voice-to-text technology through voice recognition to convert the input voice into text, which is named input text;

[0122] Step S2: construct a pronunciation feature extraction model, convert the speech to be analyzed into a spectrogram and extract the pronunciation features in the spectrogram; Step S2 includes the following sub-steps:

[0123] Step S201, after pre-processing the speech to be analyzed, obtaining a time domain graph of the speech to be analyzed, dividing it into frames, and dividing the time domain graph into different frame data;

[0124] Step S201 includes the following sub-steps:

[0125] Step S201.1, constructing a pronunciation feature extraction model, and naming the speech input into the pronunciation feature extraction model as the speech to be analyzed;

[0126] Step S201.2, filtering the speech to be analyzed by a first-order high-pass filter to obtain a time domain image of the speech to be analyzed;

[0127] Step S201.3: Set the frame length, denoted as n, obtain the speech duration of the speech to be analyzed, denoted as T, calculate T / n, and denoted as m, where m is rounded up to an integer.

[0128] Step S201.4, evenly divide the time domain graph into m parts, each part is a frame data, and the frame data is marked as F(n,m), where F(n,m) represents the waveform in the time domain graph within the time range of [n×mn,n×m].

[0129] Step S202, performing short-time Fourier transform on the frame data to convert it into a spectrum of the frame data, named frame spectrum, and converting the input speech into a spectrogram based on the frame spectrum;

[0130] Step S202 includes the following sub-steps:

[0131] Step S202.1, perform short-time Fourier transform on F(n,m) to convert it into a frame spectrum of F(n,m), denoted as S(n,m). The frame spectrum is specifically a two-dimensional coordinate system with frequency on the X axis and amplitude on the Y axis;

[0132] Step S202.2: Get the minimum and maximum amplitude values, mark them as A1 and A2 respectively, and form a range [A1, A2], named amplitude range;

[0133] Step S202.3, the amplitude range is evenly divided into 256 sub-ranges, named as amplitude sub-ranges, and the amplitude sub-ranges are sorted and numbered in ascending order, using the symbol P i Represents, where i is a positive integer and i is the serial number of P;

[0134] Step S202.4: There are color values ​​from 0 to 255, 0 represents black, 255 represents white, and P i The color value is set to C i , C i is 256-i;

[0135] Step S202.5: Name the waveform in S(n,m) the waveform to be converted, mark the length of the waveform to be converted on the X-axis as L, construct a line of length L and width n, name it the speech line, enter the speech line into S(n,m), and align the speech line with the waveform to be converted.

[0136] Step S202.6: For any value of X in S(n,m), name the amplitude value corresponding to X as the target amplitude, and search for the P to which the target amplitude belongs. i Corresponding C i , marked as CD i , change the color value of the pixel at X on the spectrum line to CD i, analyze each value of X, and the final spectrum line is a line segment with color changes;

[0137] Step S202.7, extract the spectrum line of S(n,m), rotate it 90° counterclockwise to obtain Q(n,m);

[0138] Step S202.8: Create a two-dimensional coordinate system with m as the horizontal axis and frequency as the vertical axis, named spectrogram, and enter Q(n,m) into the spectrogram;

[0139] Step S203, extracting pronunciation features of the speech to be analyzed based on the spectrogram;

[0140] Step S203 includes the following sub-steps:

[0141] Step S203.1: Number the input text from left to right, using the symbol W. j Represents, where j is a positive integer and j is the sequence number of W;

[0142] Step S203.2: When converting the input speech into input text, mark W j The pronunciation time of the word is obtained, and the frame data within the pronunciation time is searched in the spectrogram and marked as the word pronunciation data;

[0143] Step S203.3, the pronunciation data of the word is W j The pronunciation characteristics of

[0144] Step S3, extracting the pronunciation features of the input speech and input text through the pronunciation feature extraction model to obtain speech features and text features respectively; Step S3 includes the following sub-steps:

[0145] Step S301, searching for the standard pronunciation of the input text and naming it as the standard pronunciation;

[0146] Step S302: extracting the pronunciation features of the input speech through a pronunciation feature extraction model, and naming them as speech features;

[0147] Step S303: extracting the pronunciation features of the standard speech through the pronunciation feature extraction model, and naming them as text features;

[0148] Step S4: Perform pronunciation training on the user, and compare the differences between the voice features and the text features to determine whether the user's pronunciation is standard. Step S4 includes the following sub-steps:

[0149] Step S401, the user inputs or selects training content;

[0150] Step S402: broadcasting the training content to the user in a standard voice;

[0151] Step S403: The user learns the standard voice and uploads the input voice;

[0152] Step S404, determining whether the input voice conforms to the standard voice;

[0153] Step S404 includes the following sub-steps:

[0154] Step S404.1: compare the input speech with the same W in the standard speech. j The speech features and text features are summarized into a group of character feature groups;

[0155] Step S404.2: Analyze any character feature group and mark the speech features and text features in the character feature group as first features and second features respectively;

[0156] Step S404.3: number the frame data in the first feature from left to right, using the symbol Z1 h Indicates that the frame data in the second feature are numbered from left to right, and the symbol Z2 h Denotes, where h is a positive integer and h is the sequence number of Z1 and Z2;

[0157] Step S404.4: Extract Z1 from bottom to top h and Z2 h CD i , and form a one-dimensional matrix, marked as G1 h and G2 h ;

[0158] Step S404.5, calculate G1 h -G2 h , mark the calculation result as G3 h and G3 h Take the absolute value of all the values ​​in G4 h , in order from left to right, G4 h The values ​​in are numbered and represented by the symbol K1(h,t), and G2 is sorted from left to right. h The values ​​in are numbered and represented by the symbol K2(h,t), where t is a positive integer and (h,t) is the sequence number of K1 and K2;

[0159] Step S404.6, calculate , the calculation result is named pronunciation accuracy, where max() is the maximum value operator;

[0160] Step S404.7: Find a first number of volunteers, who must possess a Mandarin proficiency certificate. Require the volunteers to perform a second number of pronunciation training sessions. Count the pronunciation accuracy of each session, which is called the regular accuracy. Find the minimum value of the regular accuracy, which is called the accuracy threshold.

[0161] Step S404.8, calculate the pronunciation accuracy of the user's input voice, named input accuracy, and compare the input accuracy with the accuracy threshold. If the input accuracy is less than the accuracy threshold, mark the user's input voice as not meeting the standard voice; otherwise, mark the user's input voice as meeting the standard voice.

[0162] In embodiment 3, the present application provides an electronic device, which may include: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus. The memory stores computer-readable instructions, and the processor can call the instructions in the memory. When the computer-readable instructions are executed by the processor, the steps in the pronunciation recognition learning auxiliary method based on speech processing are executed to achieve the following functions: recognizing the user's input speech through speech recognition and extracting the input text; constructing a pronunciation feature extraction model; extracting the pronunciation features of the input speech and input text through the pronunciation feature extraction model to obtain speech features and text features respectively; performing pronunciation training on the user, and comparing the differences between the speech features and the text features to determine whether the user's pronunciation is standard.

[0163] In addition, the logical instructions in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0164] Example 4. The present application also provides a computer-readable storage medium. The present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the pronunciation recognition learning assistance method based on speech processing are executed to achieve the following functions: recognize the user's input speech through speech recognition and extract the input text; construct a pronunciation feature extraction model; extract the pronunciation features of the input speech and input text through the pronunciation feature extraction model to obtain speech features and text features respectively; perform pronunciation training on the user, and compare the differences between the speech features and the text features to determine whether the user's pronunciation is standard.

[0165] Through the description of the above embodiments, the embodiments of the present invention can be provided as methods, systems, or computer program products. Based on this understanding, the essence of the above technical solutions or the portion that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for enabling a computer device (such as a personal computer, server, or network device) to execute the methods described in various embodiments or certain portions of the embodiments.

[0166] In the embodiments provided in this application, it should be understood that the disclosed system or method can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of systems, modules and units can be electrical, mechanical or other forms.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A pronunciation recognition learning assistance method based on speech processing, characterized in that: The steps include: Recognize the user's input voice through voice recognition and extract the input text; Construct a pronunciation feature extraction model to convert the speech to be analyzed into a spectrogram and extract the pronunciation features in the spectrogram; The pronunciation features of the input speech and input text are extracted through the pronunciation feature extraction model to obtain speech features and text features respectively; Provide pronunciation training for users and compare the differences between voice features and text features to determine whether the user's pronunciation is standard; Constructing a pronunciation feature extraction model, converting the speech to be analyzed into a spectrogram and extracting pronunciation features from the spectrogram includes the following sub-steps: After preprocessing the speech to be analyzed, a time domain image of the speech to be analyzed is obtained, and the time domain image is divided into different frame data; Perform short-time Fourier transform on the frame data to convert it into a spectrum of the frame data, named frame spectrum, and convert the input speech into a spectrogram based on the frame spectrum; Extract the pronunciation features of the speech to be analyzed based on the spectrogram; After the speech to be analyzed is preprocessed, a time domain graph of the speech to be analyzed is obtained and framed. Dividing the time domain graph into different frame data includes the following sub-steps: Construct a pronunciation feature extraction model, and name the speech input into the pronunciation feature extraction model as the speech to be analyzed; After filtering the speech to be analyzed through a first-order high-pass filter, a time domain image of the speech to be analyzed is obtained; Set the frame length, marked as n, obtain the speech duration of the speech to be analyzed, marked as T, calculate T / n, and mark the result as m, where m is rounded up to an integer; The time domain graph is evenly divided into m parts, each part is a frame data, and the frame data is marked as F(n,m). F(n,m) represents the waveform in the time domain graph within the time range of [n×mn,n×m]. The short-time Fourier transform of the frame data is converted into a spectrum of the frame data, which is named as a frame spectrum. Converting the input speech into a spectrogram based on the frame spectrum includes the following sub-steps: Performing a short-time Fourier transform on F(n,m) to convert it into a frame spectrum of F(n,m), which is labeled S(n,m). The frame spectrum is specifically a two-dimensional coordinate system with the X axis being the frequency and the Y axis being the amplitude value; Get the minimum and maximum amplitude values, mark them as A1 and A2 respectively, form the range [A1, A2], and name it amplitude range; The amplitude range is evenly divided into 256 sub-ranges, named as amplitude sub-ranges, and the amplitude sub-ranges are sorted and numbered in ascending order. i Represents, where i is a positive integer and i is the serial number of P; There are color values ​​0 to 255, 0 represents black, 255 represents white, i The color value is set to C i , the C i is 256-i; Name the waveform in S(n,m) the wave to be converted, mark the length of the wave to be converted on the X-axis as L, construct a line with a length of L and a width of n, name it the speech line, record the speech line into S(n,m), and align the speech line with the wave to be converted; For any value of X in S(n,m), name the amplitude value corresponding to X as the target amplitude, and find the P to which the target amplitude belongs. i Corresponding C i , marked as CD i , change the color value of the pixel at X on the spectrum line to CD i , analyze each value of X, and the final spectrum line is a line segment with color changes; Extract the spectrum line of S(n,m) and rotate it 90° counterclockwise to obtain Q(n,m); A two-dimensional coordinate system is established with m as the horizontal axis and frequency as the vertical axis, named spectrogram, and Q(n,m) is entered into the spectrogram; The method of extracting the pronunciation features of the speech to be analyzed based on the spectrogram includes the following sub-steps: Number the input text from left to right, using the symbol W j Represents, where j is a positive integer and j is the sequence number of W; When converting input speech into input text, mark W j The pronunciation time of the word is obtained, and the frame data within the pronunciation time is searched in the spectrogram and marked as the word pronunciation data; The word pronunciation data is W j The pronunciation characteristics of Determining whether the user's pronunciation is standard includes the following sub-steps: Compare the input speech with the same W in the standard speech j The speech features and text features are summarized into a group of character feature groups; Analyze any character feature group and mark the voice feature and text feature in the character feature group as the first feature and the second feature respectively; The frame data in the first feature are numbered from left to right, and the symbol Z1 h Indicates that the frame data in the second feature are numbered from left to right, and the symbol Z2 h Denotes, where h is a positive integer and h is the sequence number of Z1 and Z2; Extract Z1 in order from bottom to top h and Z2 h CD i , and form a one-dimensional matrix, marked as G1 h and G2 h ; Calculate G1 h -G2 h , mark the calculation result as G3 h and G3 h Take the absolute value of all the values ​​in G4 h , in order from left to right, G4 h The values ​​in are numbered and represented by the symbol K1(h,t), and G2 is sorted from left to right. h The values ​​in are numbered and represented by the symbol K2(h,t), where t is a positive integer and (h,t) is the sequence number of K1 and K2; calculate , the calculation result is named pronunciation accuracy, where max() is the maximum value operator; Find the first number of volunteers, ask them to perform pronunciation training for the second number, and calculate the pronunciation accuracy of each training, which is named as the regular accuracy. Find the minimum value of the regular accuracy, which is named as the accuracy threshold. The pronunciation accuracy of the user's input speech is calculated, named input accuracy, and the input accuracy is compared with the accuracy threshold. If the input accuracy is less than the accuracy threshold, the user's input speech is marked as not meeting the standard speech. Otherwise, the user's input speech is marked as meeting the standard speech.

2. The pronunciation recognition learning auxiliary method based on speech processing according to claim 1 is characterized in that: The speech recognition is performed by using speech-to-text technology to convert the input speech into text, which is named input text.

3. The pronunciation recognition learning auxiliary method based on speech processing according to claim 2 is characterized in that: The method of extracting the pronunciation features of the input speech and input text by using the pronunciation feature extraction model includes the following sub-steps: Find the standard pronunciation of the input text and name it as standard pronunciation; The pronunciation features of the input speech are extracted through the pronunciation feature extraction model and named as speech features; The pronunciation features of standard speech are extracted through the pronunciation feature extraction model and named as text features.

4. The pronunciation recognition learning auxiliary method based on speech processing according to claim 3 is characterized in that: The method of performing pronunciation training on the user and comparing the differences between the voice features and the text features to determine whether the user's pronunciation is standard includes the following sub-steps: The user inputs or selects training content; Standard voice to announce training content to users; Users learn standard voices and upload input voices; Determine whether the input voice meets the standard voice.

5. A pronunciation recognition learning assistance system based on speech processing, used to implement the pronunciation recognition learning assistance method based on speech processing according to any one of claims 1 to 4, characterized in that: It includes a speech input module, a pronunciation feature extraction model construction module, a pronunciation feature extraction module and a pronunciation standard judgment module; the speech input module, the pronunciation feature extraction model construction module and the pronunciation feature extraction module are respectively connected to the pronunciation standard judgment module data; The voice input module is used to recognize the user's input voice through voice recognition and extract the input text; The pronunciation feature extraction model construction module is used to construct a pronunciation feature extraction model, convert the speech to be analyzed into a spectrogram and extract the pronunciation features in the spectrogram; The pronunciation feature extraction module is used to extract the pronunciation features of the input speech and input text through the pronunciation feature extraction model to obtain speech features and text features respectively; The pronunciation standard judgment module is used to perform pronunciation training on the user and compare the differences between voice features and text features to determine whether the user's pronunciation is standard.

Images