Voice recognition method of point reading pen
By comparing the demonstration pronunciation and user follow-up data, combining the lip shaped data collected by the camera, a comprehensive analysis of phonemes and lip shaped is solved, and the analysis results in the existing technology are not detailed enough, and the speech recognition correction of the dot reading pen is improved Improved effects of error accuracy and user pronunciation habits.
Patent Information
- Application Number
- CN202311779723.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-22
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-12-22
AI Technical Summary
The existing dot reading pen voice recognition method only analyzes the speech content, resulting in the analysis results that are not detailed and accurate enough, and cannot fully understand the user's pronunciation errors, which affects the accuracy of error correction.
By obtaining demonstration pronunciation data and user follow-up data, following pronunciation scores are performed, and lip typing data is collected using the camera, combining phoneme error analysis and lip typing change characteristics, comprehensive lip typing abnormal data is generated, error correction and training is performed, and pronunciation is improved accuracy.
It realizes fine-grained analysis of user pronunciations and accurate identification of lip typographic errors, improving the accuracy of error correction and improving the user's pronunciation habits.
Smart Images

Figure CN117935780B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech recognition method of a point-reading pen. Background Art
[0002] The speech recognition of the reading pen allows users to interact with the device in spoken language, promoting a more natural and direct way of communication. This interactive method can stimulate interest and make learning more lively and interesting. Speech recognition enables the reading pen to adapt to the needs of different users. It can provide personalized feedback and learning content based on the user's accent, speaking speed and language habits, which is closer to individual needs. Speech recognition adds more possibilities to the reading pen. It can be used for teaching applications such as voice search, interactive teaching, and voice evaluation, enriching teaching methods and experience. However, the existing speech recognition method of the reading pen only analyzes the voice content, resulting in insufficient details and accuracy in the analysis results, and users cannot fully understand the error points. Summary of the invention
[0003] Based on this, it is necessary to provide a speech recognition method for a reading pen to solve at least one of the above technical problems.
[0004] To achieve the above object, a speech recognition method for a point reading pen is provided, the method comprising the following steps:
[0005] Step S1: Acquire demonstration pronunciation data and user follow-up reading data; perform follow-up reading pronunciation scoring on the user follow-up reading data based on the demonstration pronunciation data to generate a follow-up reading pronunciation score;
[0006] Step S2: screening low-score data of the pronunciation scores of the follow-up reading using a preset score threshold to generate unqualified pronunciation data; performing fine-grained analysis on the unqualified pronunciation data to generate phoneme error data;
[0007] Step S3: using a camera device to collect lip shape data and generate follow-up lip shape data; based on the unqualified pronunciation data, the follow-up lip shape data is screened for incorrect lip shapes and incorrect lip shape data is generated;
[0008] Step S4: Analyze the change characteristics of the erroneous lip shape data to generate lip shape change characteristic data; mark abnormal points according to the lip shape change characteristic data to generate comprehensive lip shape abnormal data;
[0009] Step S5: Perform lip shape demonstration correction based on the comprehensive lip shape abnormality data to generate correction process data; perform difficulty clustering based on the correction process data to generate key and difficult pronunciation data; perform pronunciation training based on the key and difficult pronunciation data to generate optimized pronunciation data.
[0010] The present invention can quantify the pronunciation accuracy of the user and generate a pronunciation score by comparing the demonstration pronunciation and the user's follow-up reading data. A clear score is provided for the user to help understand the pronunciation performance and motivate the improvement of spoken language. The pronunciation data with unqualified pronunciation is screened by threshold value, focusing on the data with obvious pronunciation problems for further analysis. The phoneme error data is analyzed to identify the specific phoneme pronunciation problems and provide a specific direction for correction. The camera is used to collect lip shape data to find incorrect lip shapes. By screening the lip shape data for errors, the lip shape problems existing in the pronunciation process of the user can be more accurately located. By analyzing the changes in the lip shape data, the specific patterns or laws of the lip shape problems are found. The lip shape abnormalities are identified to further highlight the problems in the lip shape. Demonstration error correction is performed based on the lip shape abnormality data to improve the lip shape problems and provide a lip shape demonstration that is more in line with the standard. By clustering and dividing the difficult pronunciation data, targeted training is carried out to strengthen the pronunciation ability of the user on specific difficult points. Therefore, the speech recognition method of the point-reading pen of the present invention not only obtains the phoneme details through fine-grained analysis, but also performs lip shape error analysis on the user's follow-up lip shape data, thereby comprehensively evaluating the user's incorrect pronunciation points to improve the accuracy of error correction and improve the user's pronunciation habits.
[0011] Preferably, step S1 comprises the following steps:
[0012] Step S11: Acquire demonstration pronunciation data and user follow-up reading data; mark special pronunciation points based on the demonstration pronunciation data to generate special pronunciation requirement data;
[0013] Step S12: performing short-time energy and fundamental frequency change analysis on the user's follow-up reading data to generate speech speed and intonation data; performing rhythm and prosody analysis on the speech speed and intonation data to generate rhythm and prosody feature data;
[0014] Step S13: performing signal conversion on the user's follow-up reading data to generate a follow-up reading spectrogram; performing voiceprint feature analysis based on the follow-up reading spectrogram to generate voiceprint feature data;
[0015] Step S14: using the rhythm and prosody feature data to correct feature deviations of the voiceprint feature data, and generating follow-up feature data;
[0016] Step S15: Perform comprehensive follow-up pronunciation scoring based on the follow-up feature data to generate comprehensive scoring factor data;
[0017] Step S16: scoring the special pronunciation of the rhythmic and prosodic feature data according to the special pronunciation requirement data to generate key scoring factor data;
[0018] Step S17: Perform a total pronunciation score based on the comprehensive scoring factor data and the key scoring factor data to generate a follow-up pronunciation score.
[0019] The present invention provides key guidance for users to pay attention to these pronunciations by marking special pronunciation points in the demonstration pronunciation, which may include tone changes, special phonemes, etc. Short-term energy and fundamental frequency changes analyze the speech speed and tone, while rhythm and prosody analysis provides information such as the rhythm and tone of the voice. These data can help users understand their own speech rhythm and tone, and help improve oral fluency and naturalness. Based on rhythm and prosody and voiceprint feature data, the features in the follow-up data are corrected to reduce pronunciation deviations and improve accuracy. The corrected data helps to improve the accuracy of users' pronunciation and make them closer to standard pronunciation. Combined with the follow-up feature data, multiple factors such as speech speed, tone, voiceprint, etc. are comprehensively considered to provide a comprehensive evaluation of the user's pronunciation performance. The comprehensive scoring factor data provides a multi-dimensional evaluation of the user's pronunciation ability, which helps users understand their performance in different aspects. According to the special pronunciation requirement data, special scoring is performed on key pronunciation points, focusing on the parts with special pronunciation requirements. These data highlight the performance of users at specific pronunciation points, so that the scoring pays more attention to the key points and areas that need improvement in pronunciation. Based on the comprehensive scoring factor data and key scoring factor data, the final pronunciation score is generated, taking into account all aspects of performance. The final pronunciation score provides an assessment of the user's overall pronunciation ability, provides comprehensive feedback to the user, and indicates the direction of improvement.
[0020] Preferably, step S2 comprises the following steps:
[0021] Step S21: screening low-score data of pronunciation scores of follow-up reading using a preset score threshold to generate pronunciation unqualified data;
[0022] Step S22: segmenting the follow-up reading feature data of the unqualified pronunciation data into error segments according to the comprehensive scoring factor data to generate follow-up reading error feature data;
[0023] Step S23: Marking phoneme boundaries based on the special pronunciation requirement data to generate phoneme boundary division data;
[0024] Step S24: extracting deviation phonemes from the rhythmic prosody feature data using the phoneme boundary division data to generate phoneme error data.
[0025] The present invention screens out low-scoring data through a score threshold, locates specific data with unqualified pronunciation, and provides quantitative information on specific problems. This helps to conduct error analysis and improvements more accurately in the future. The error segments in the unqualified pronunciation data are segmented according to the comprehensive scoring factor data to help locate the specific parts of the pronunciation errors. Targeted error segments are provided to make the problem more specific and more analyzable, providing a clearer direction for improvement. The phonemes are marked with boundaries according to special pronunciation requirement data so as to more accurately identify the specific positions of the phonemes. The division of phoneme boundaries helps to more accurately find the specific phonemes with pronunciation problems, making subsequent analysis more accurate. The phoneme boundary division data is used to extract the phoneme parts with deviations in the rhythm and prosody feature data. An accurate pronunciation problem location is provided to help more accurately identify specific pronunciation errors and make targeted improvements.
[0026] Preferably, step S3 comprises the following steps:
[0027] Step S31: using a camera device to obtain face image data;
[0028] Step S32: using a placement orientation calculation formula and performing device placement orientation calculation based on pre-collected frontal face image data and face image data to generate pre-placement position data;
[0029] Step S33: adjusting the position of the camera device according to the pre-placement position data, and collecting the lip shape to generate the lip shape data;
[0030] Step S34: Screening the mouth shape data of the follow-up reading based on the unqualified pronunciation data to generate wrong mouth shape data.
[0031] The present invention captures facial image data through a camera device for subsequent extraction and analysis of lip shape information. Facial image data helps to analyze the shape and changes of the lip shape, and provides additional information for pronunciation evaluation. The placement position of the device is calculated using the pre-collected frontal face image data and the current face image data to ensure better capture of lip shape information. The measurement and adjustment of the device position helps to improve the accuracy and reliability of lip shape acquisition. Under the calibrated device position, a camera is used to capture lip shape data to record the user's lip shape changes and pronunciation process. The collection of lip shape data helps to capture dynamic information during lip shape changes and pronunciation. The lip shape data is screened based on the unqualified pronunciation data, and the lip shape data related to the pronunciation problem is extracted. The screened erroneous lip shape data helps to more accurately locate the lip shape problems related to the unqualified pronunciation, and provides direction for subsequent improvements.
[0032] Preferably, the placement orientation calculation formula in step S32 is as follows:
[0033] ;
[0034] ;
[0035] In the formula, For pre-placement position data, is the horizontal coordinate of the position of the point reading pen, is the device distance parameter, Place the device at an angle. is the vertical coordinate of the position of the point reading pen, is the inverse tangent function, is the ordinate of the nose tip position, is the horizontal coordinate of the nose tip position, is the maximum face length of the pre-collected frontal face image, is the maximum face length of the face image, is the maximum face width of the pre-collected frontal face image, is the maximum face width of the face image, is the offset angle of the device placement. To correct the angle.
[0036] The present invention constructs a placement orientation calculation formula, which is used to calculate the device placement orientation based on the pre-collected face front image data and face image data, and generate pre-placement position data. The formula fully considers the horizontal coordinate of the point reading pen position. , device distance parameters , equipment placement angle , the vertical coordinate of the point reading pen position , the inverse tangent function , the vertical coordinate of the nose tip position , the horizontal coordinate of the nose tip position , the maximum face length of the pre-collected frontal face image , the maximum face length of the face image , the maximum face width of the pre-collected frontal face image , the maximum face width of the face image , the offset angle of the device placement , Correction Angle And the interaction relationship between variables, forming the following functional relationship:
[0037] ;
[0038] ;
[0039] pass and Determines the relative reference point of the device in the pre-placement position data. By determining the coordinates of the reference point, the placement of the device can be clarified and affect the determination of the final pre-placement position. Indicates the distance between the device and the reference object. By adjusting the distance parameter, you can control the distance between the device and the reference object. A larger distance parameter will make the device farther away from the reference object, while a smaller distance parameter will make the device closer to the reference object. It is the device placement azimuth. By calculating the angle difference between the reading pen and the face position, combined with the offset angle, the device placement angle can be determined. By adjusting this angle, the device orientation can be changed to face different directions of the target object. and is the coordinate of the nose tip in the face image data. By comparing the position of the reading pen with the position coordinate of the face, the device placement angle can be calculated. Changes in these coordinates will directly affect The calculation result of The calculation can be done by calculating the azimuth of the device placement The change in the relative size of the face is taken into account to further accurately determine the placement of the device. A larger difference value indicates that the height change is more significant than the width change, and the angle value will be larger; conversely, a smaller difference value indicates that the height and width change trends are similar, and the angle value will be closer to zero. By comparing the facial midline position of the acquired facial image data with the actual midline position, the offset angle of the device placement can be obtained. , thereby correcting the deviation of the device placement to adapt to the user usage at different times and locations. This functional relationship can accurately evaluate the degree of deviation between the device and the face position, thereby calculating the reasonable device placement position to obtain clear face image data, providing a basis for subsequent lip shape collection and analysis. And using To correct the angle, adjust the functional relationship to reduce the impact of data deviation or error. For example, when the light is poor, the possible placement angle can be determined through foresight experience, so as to Make adjustments and corrections to more accurately generate the device placement angle , improve the accuracy of equipment orientation adjustment. At the same time, the coordinate values and correction values in the formula are adjusted according to the actual situation and applied to different users and usage environments, which improves the flexibility and applicability of the algorithm.
[0040] Preferably, step S4 comprises the following steps:
[0041] Step S41: performing lip opening and closing analysis on the incorrect lip shape data to generate lip opening and closing movement data; performing opening time image interception on the incorrect lip shape data based on the lip opening and closing movement data to generate opening image data;
[0042] Step S42: performing tooth opening and closing analysis based on the tooth opening and closing image data to generate tooth opening and closing motion data; performing tooth opening time image interception on the tooth opening image data based on the tooth opening and closing motion data to generate tooth opening image data;
[0043] Step S43: performing noise reduction and smoothing processing on the open-teeth image data to generate clear open-teeth image data; marking the tongue tip position according to the clear open-teeth image data to generate tongue tip motion trajectory data;
[0044] Step S44: performing time-series correlation on the lip opening and closing movement data, the tooth position opening and closing movement data, and the tongue tip movement trajectory data to generate mouth shape change feature data;
[0045] Step S45: marking abnormal change points according to the lip shape change feature data to generate lip shape abnormal range data;
[0046] Step S46: marking the abnormal time point of the lip shape abnormal range data according to the phoneme error data to generate comprehensive lip shape abnormal data.
[0047] The present invention performs lip opening and closing analysis on mouth shape data and extracts lip movement data, which is helpful to understand the movement of lips during pronunciation. Based on the lip opening and closing movement data, the mouth opening image data is intercepted to record the opening and closing of lips during pronunciation. The tooth position opening and closing analysis is performed using the mouth opening image data, and the tooth position opening and closing movement data is extracted to understand the dynamic change of the tooth position during pronunciation. Based on the tooth position opening and closing movement data, the tooth opening image data is intercepted to record the change of the tooth position during pronunciation. The tooth opening image data is subjected to noise reduction and smoothing processing to reduce the noise in the image and extract clear tooth opening image data. Based on the clear tooth opening image data, the tongue tip position is marked and the tongue tip movement trajectory data is extracted to record the movement trajectory of the tongue tip during pronunciation. The lip opening and closing movement data, the tooth position opening and closing movement data and the tongue tip movement trajectory data are time-series associated to integrate and form mouth shape change feature data. A more comprehensive mouth shape feature is provided, which is helpful to fully understand the mouth shape change. The abnormal change point is marked according to the mouth shape change feature data, and the mouth shape abnormal range is determined, which may involve abnormal lip, tooth position or tongue tip movement. Determine the range of lip shape abnormalities so that subsequent analysis and improvements can be more focused on specific lip shape abnormality intervals. Mark the abnormal time points of the lip shape abnormality range data based on the phoneme error data, and associate the lip shape abnormality with specific phoneme errors. Associate the lip shape abnormality data with pronunciation errors to provide more accurate correlation analysis between lip shape and pronunciation errors for improvement.
[0048] Preferably, step S44 includes the following steps:
[0049] Step S441: Time-sequentially associate the lip opening and closing movement data, the tooth position opening and closing movement data, and the tongue tip movement trajectory data to generate mouth shape movement change data;
[0050] Step S442: performing trajectory image conversion on the lip movement change data to generate a two-dimensional lateral movement graph;
[0051] Step S443: using the angle change evaluation formula and the two-dimensional lateral motion diagram to perform motion tendency analysis on the tongue tip motion trajectory data to generate motion direction feature data;
[0052] Step S444: using the motion direction feature data to perform trajectory fuzzy estimation on the tongue tip motion trajectory data to generate tongue tip motion optimization data;
[0053] Step S445: performing motion supplementation on the two-dimensional lateral motion map according to the tongue tip motion optimization data to generate a lateral motion optimization map;
[0054] Step S446: Perform motion feature evaluation based on the lateral motion optimization graph to generate lip shape change feature data.
[0055] The present invention generates lip movement change data by temporally associating the lip opening and closing, tooth opening and closing, and tongue tip movement data. The movement changes of various parts of the lip shape are integrated to provide more comprehensive lip movement features, which helps to understand the lip shape changes more deeply. The lip shape movement change data is converted into a two-dimensional lateral movement graph to display the side perspective of the movement changes of the oral organs in an image. The two-dimensional lateral movement graph helps to intuitively display the movement of the lip shape and provide visual information of the lip shape movement. The two-dimensional lateral movement graph is analyzed using the angle change evaluation formula to evaluate the movement tendency of the tongue tip movement trajectory. Movement direction feature data is generated to mark the tendency of the tongue tip movement, which may be a tendency to move upward, downward or horizontally. Trajectory fuzzy estimation is performed based on the movement direction feature data to optimize the tongue tip movement trajectory data. The tongue tip movement is estimated and optimized to improve the accuracy and clarity of the lip shape movement data to more accurately reflect the movement of the tongue tip. The two-dimensional lateral movement graph is supplemented with movement based on the tongue tip movement optimization data to generate an optimized lateral lip shape movement graph. The optimized lateral movement graph shows the lip shape movement more clearly and provides more intuitive lip shape movement visual information. Based on the optimized lateral motion graph, the mouth movement characteristics are evaluated to analyze the characteristics and rules of mouth shape changes. This provides a more detailed description of the specific characteristics of mouth shape changes in spoken pronunciation.
[0056] Preferably, the angle change evaluation formula in step S443 is as follows:
[0057] ;
[0058] In the formula, is the average change in angle, is the start time of the assessment, is the end time of the assessment, is the index value of time change, is the arccosine function, For time The horizontal coordinate of the tongue tip position at For time The horizontal coordinate of the tongue tip position at For time The ordinate of the tongue tip position at For time The ordinate of the tongue tip position at For time The horizontal coordinate of the tongue tip position at For time The ordinate of the tongue tip position at is the angle deviation correction value.
[0059] The present invention constructs an angle change evaluation formula for analyzing the movement tendency of the tongue tip movement trajectory data based on the two-dimensional lateral movement graph to generate movement direction feature data. The formula fully considers the start time of the evaluation. , the end time of the evaluation , the index value of time change , the arccosine function , time is The horizontal coordinate of the tongue tip position at , time is The horizontal coordinate of the tongue tip position at , time is The vertical coordinate of the tongue tip position at , time is The ordinate of the tongue tip position at , time is The horizontal coordinate of the tongue tip position at , time is The vertical coordinate of the tongue tip position at , Angle deviation correction value And the interaction relationship between variables, forming the following functional relationship:
[0060] ;
[0061] pass and , indicating the position of the tongue tip at time point and In the formula, they form the vectors of two adjacent time points and are used to calculate the angle change. The third point used in the formula to calculate the angle between two vectors helps determine the direction of the two vectors. Calculates the angle between two vectors. This function divides the inner product by the product of the modulos and then inversely calculates the angle between two vectors. Its function is to convert the angle between the vectors into an actual angle. The angle is summed over a specified time range by using the summation symbol to calculate the total change in angle over that time range. It is used to normalize the sum of the angle values to the average value within the time range. This average value represents the average rate of change of the angle. This functional relationship can quickly evaluate the position and speed of the tongue tip change during each pronunciation process, so as to predict the movement state of the tongue tip during the mouth closure period, so as to improve the accuracy of the subsequent pronunciation error position assessment. Using the angle deviation correction value Correcting the functional relationship reduces the error impact caused by abnormal data and error data, thereby generating the average angle change value more accurately. , which improves the accuracy and reliability of the calculation results. At the same time, the correction value in the formula can be adjusted according to the actual situation and applied to different trajectory data to calculate their motion tendency, which improves the flexibility and applicability of the algorithm.
[0062] Preferably, step S5 comprises the following steps:
[0063] Step S51: performing lip shape demonstration error correction based on the comprehensive lip shape abnormality data to generate error correction follow-up reading data;
[0064] Step S52: re-scoring the pronunciation of the error correction and follow-up reading data to generate re-scoring data; using a preset score threshold to evaluate the eligibility of the re-scoring data, if the re-scoring data is greater than or equal to the preset score threshold, summarizing the error correction process and generating error correction process data;
[0065] Step S53: if the re-scoring data is less than the preset score threshold, secondary unqualified data is generated, and repeated error analysis and error correction are performed based on the secondary unqualified data until error correction process data is generated;
[0066] Step S54: performing clustering of difficult points according to the error correction process data to generate important and difficult point pronunciation data;
[0067] Step S55: Formulate training strategies based on the key and difficult pronunciation data to generate training strategy data;
[0068] Step S56: Perform pronunciation training according to the training strategy data to generate optimized pronunciation data.
[0069] The present invention performs lip shape demonstration correction by integrating abnormal lip shape data, and generates error correction and follow-up reading data. The error correction and follow-up reading data provides a demonstration of the correct lip shape to help users understand and imitate the correct lip shape. The pronunciation of the error correction and follow-up reading data is re-scored to generate re-scoring data. A pre-set score threshold is used for qualification assessment. If it meets the pre-set standard, error correction process data is generated to summarize the error correction process. If the re-scoring data is lower than the pre-set score threshold, secondary unqualified data is generated to perform iterative error analysis and error correction. The error correction process is iteratively analyzed to continuously improve the spoken pronunciation until the pre-set qualification assessment standard is reached. Based on the error correction process data, the difficulties in the error correction process are clustered and divided, and the key and difficult pronunciation data are distinguished. Through clustering, specific key and difficult points in pronunciation are determined to help carry out subsequent training more targetedly. According to the key and difficult pronunciation data, targeted training strategies are formulated, including specific exercises, skills and methods. Targeted training strategies are formulated to make the training closer to the individual's pronunciation problems and improve the training effect.
[0070] Preferably, step S54 includes the following steps:
[0071] Step S541: aggregate the same error words in the error correction process data to generate aggregated word data;
[0072] Step S542: Estimating the error frequency based on the aggregated vocabulary data to generate error correction frequency distribution data;
[0073] Step S543: performing phoneme error frequency estimation based on the error correction frequency distribution data to generate phoneme frequency distribution data;
[0074] Step S544: clustering the same phonemes according to the phoneme frequency distribution data to generate aggregated phoneme data;
[0075] Step S545: re-divide the aggregated phoneme data by phoneme position to generate important and difficult pronunciation data.
[0076] The present invention aggregates the same erroneous words in the error correction process data to generate aggregated vocabulary data. Aggregating the same erroneous words with high frequency helps to identify common pronunciation problems and centrally process them. Estimating the error frequency based on the aggregated vocabulary data generates error correction frequency distribution data. Providing the distribution of pronunciation errors helps to understand the frequency and severity of different errors. Estimating the phoneme error frequency based on the error correction frequency distribution data generates phoneme frequency distribution data. Providing the frequency and distribution of specific phoneme errors helps to identify the frequent errors of specific phonemes in pronunciation problems. According to the phoneme frequency distribution data, the data of the same phoneme in the pronunciation errors are aggregated and divided to generate aggregated phoneme data. By aggregating the phoneme data, the same phonemes with high frequency are clustered so as to more effectively identify the key phonemes in the pronunciation. Based on the aggregated phoneme data, the positions of the phonemes are re-divided to distinguish the key and difficult pronunciation data in the pronunciation process. In order to more accurately determine which phonemes are more prone to errors or more difficult to pronounce in the pronunciation process, further targeted training and improvement can be carried out.
[0077] The beneficial effect of the present application is that the present invention generates a pronunciation score by comparing and scoring the demonstration pronunciation and the user's follow-up reading data. This step evaluates the accuracy and quality of the user's pronunciation. A standardized scoring method is provided to help users understand their pronunciation level. It also provides basic data for subsequent error analysis. By setting a preset score threshold, data with unqualified pronunciation are identified and screened out. These data may include parts with low pronunciation accuracy. The unqualified pronunciation data are analyzed in detail to identify the phoneme error data therein. These error data identify specific pronunciation problems and help to accurately locate errors. The lip shape data is collected by a camera device for analyzing the user's lip shape movements. Based on the unqualified pronunciation data, the data with incorrect mouth shape is screened, and these data can include the incorrect parts of the user's lip shape movements when pronouncing. The change feature analysis of the wrong mouth shape data is performed to identify the change feature of the mouth shape movement. According to the mouth shape change feature data, the abnormal points in the mouth shape movement are marked and the mouth shape abnormal data is identified. Based on the comprehensive mouth shape abnormal data, the lip shape demonstration error correction is performed to guide the user's correct lip shape movement. According to the error correction process data, the key and difficult points in the pronunciation process are identified, which may be specific parts of the mouth shape movement. Training is carried out on the pronunciation data of the key and difficult points to help users improve their pronunciation and improve the correctness and accuracy of their mouth shape movements. Therefore, the speech recognition method of the point-reading pen of the present invention not only obtains the phoneme details through fine-grained analysis, but also performs mouth shape error analysis on the user's follow-up mouth shape data, thereby comprehensively evaluating the user's incorrect pronunciation points to improve the accuracy of error correction and improve the user's pronunciation habits. BRIEF DESCRIPTION OF THE DRAWINGS
[0078] Figure 1A schematic diagram of the steps of a speech recognition method for a point-reading pen;
[0079] Figure 2 for Figure 1 Detailed implementation steps of step S2 in the flowchart;
[0080] Figure 3 for Figure 1 Detailed implementation steps of step S3 in FIG.
[0081] Figure 4 for Figure 1 Detailed implementation steps of step S4 in FIG.
[0082] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0083] The following is a clear and complete description of the technical method of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.
[0084] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.
[0085] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.
[0086] To achieve this, please refer to Figures 1 to 4 , a speech recognition method for a point reading pen, comprising the following steps:
[0087] Step S1: Acquire demonstration pronunciation data and user follow-up reading data; perform follow-up reading pronunciation scoring on the user follow-up reading data based on the demonstration pronunciation data to generate a follow-up reading pronunciation score;
[0088] Step S2: screening low-score data of the pronunciation scores of the follow-up reading using a preset score threshold to generate unqualified pronunciation data; performing fine-grained analysis on the unqualified pronunciation data to generate phoneme error data;
[0089] Step S3: using a camera device to collect lip shape data and generate follow-up lip shape data; based on the unqualified pronunciation data, the follow-up lip shape data is screened for incorrect lip shapes and incorrect lip shape data is generated;
[0090] Step S4: Analyze the change characteristics of the erroneous lip shape data to generate lip shape change characteristic data; mark abnormal points according to the lip shape change characteristic data to generate comprehensive lip shape abnormal data;
[0091] Step S5: Perform lip shape demonstration correction based on the comprehensive lip shape abnormality data to generate correction process data; perform difficulty clustering based on the correction process data to generate key and difficult pronunciation data; perform pronunciation training based on the key and difficult pronunciation data to generate optimized pronunciation data.
[0092] The present invention can quantify the pronunciation accuracy of the user and generate a pronunciation score by comparing the demonstration pronunciation and the user's follow-up reading data. A clear score is provided for the user to help understand the pronunciation performance and motivate the improvement of spoken language. The pronunciation data with unqualified pronunciation is screened by threshold value, focusing on the data with obvious pronunciation problems for further analysis. The phoneme error data is analyzed to identify the specific phoneme pronunciation problems and provide a specific direction for correction. The camera is used to collect lip shape data to find incorrect lip shapes. By screening the lip shape data for errors, the lip shape problems existing in the pronunciation process of the user can be more accurately located. By analyzing the changes in the lip shape data, the specific patterns or laws of the lip shape problems are found. The lip shape abnormalities are identified to further highlight the problems in the lip shape. Demonstration error correction is performed based on the lip shape abnormality data to improve the lip shape problems and provide a lip shape demonstration that is more in line with the standard. By clustering and dividing the difficult pronunciation data, targeted training is carried out to strengthen the pronunciation ability of the user on specific difficult points. Therefore, the speech recognition method of the point-reading pen of the present invention not only obtains the phoneme details through fine-grained analysis, but also performs lip shape error analysis on the user's follow-up lip shape data, thereby comprehensively evaluating the user's incorrect pronunciation points to improve the accuracy of error correction and improve the user's pronunciation habits.
[0093] In the embodiment of the present invention, reference Figure 1 The above is a schematic flow chart of the steps of a speech recognition method of a point reading pen of the present invention. In this example, the speech recognition method of the point reading pen includes the following steps:
[0094] Step S1: Acquire demonstration pronunciation data and user follow-up reading data; perform follow-up reading pronunciation scoring on the user follow-up reading data based on the demonstration pronunciation data to generate a follow-up reading pronunciation score;
[0095] In the embodiment of the present invention, special pronunciation points are marked by demonstrating pronunciation data. The speed, intonation and rhythm and prosody features of the user's follow-up reading data are analyzed. The user's follow-up reading data is converted into a spectrogram for voiceprint feature analysis. The voiceprint features are modified according to the speed, intonation and rhythm and prosody features to generate follow-up reading feature data. The follow-up reading pronunciation quality and key pronunciation points are comprehensively evaluated, and special follow-up reading pronunciation scores are performed for special pronunciation requirements. The total pronunciation score is performed by combining the total score factors and the key score factors.
[0096] Step S2: screening low-score data of the pronunciation scores of the follow-up reading using a preset score threshold to generate unqualified pronunciation data; performing fine-grained analysis on the unqualified pronunciation data to generate phoneme error data;
[0097] In an embodiment of the present invention, low-scoring data is screened out by setting a score threshold. The score threshold standard is determined, for example, to determine which scores are considered unqualified. The unqualified pronunciation data is carefully analyzed, divided into error segments, and the specific erroneous parts are located. The segmentation standard and method of the error segments are determined, for example, based on time or phoneme division. According to special pronunciation requirements, the phoneme boundaries are divided, and the phonemes in the pronunciation process are marked or segmented. According to the rules of special pronunciation requirements, the phoneme boundaries are accurately divided and marked. From the marked phoneme boundary data, the deviation phoneme data is extracted. These phonemes are analyzed to identify the wrong or inaccurate pronunciation parts.
[0098] Step S3: using a camera device to collect lip shape data and generate follow-up lip shape data; based on the unqualified pronunciation data, the follow-up lip shape data is screened for incorrect lip shapes and incorrect lip shape data is generated;
[0099] In an embodiment of the present invention, the user's lip shape is collected in real time by using a camera device to generate lip shape data. Use appropriate camera angles and resolutions to ensure that lip shape movements can be captured clearly and accurately. According to previously identified unqualified pronunciation data, the lip shape data is screened to find lip shape data related to incorrect pronunciation. The timestamp or specific mark of the incorrect pronunciation data is used to locate and filter out the corresponding lip shape data. Feature extraction is performed on the filtered lip shape data to analyze lip shape features such as mouth movement and morphology. Features related to incorrect pronunciation in the lip shape data are extracted, such as lip shape, tongue position, etc. By analyzing the lip shape feature data, abnormal lip shape data, that is, abnormal lip shape movements related to incorrect pronunciation, are identified. Abnormal lip shape data is marked or filtered out to accurately identify lip shape errors related to unqualified pronunciation.
[0100] Step S4: Analyze the change characteristics of the erroneous lip shape data to generate lip shape change characteristic data; mark abnormal points according to the lip shape change characteristic data to generate comprehensive lip shape abnormal data;
[0101] In an embodiment of the present invention, relevant data is obtained by analyzing the lip opening and closing movements of incorrect mouth shape data. The lip opening and closing movements are analyzed to extract their changing characteristics. The tooth opening and closing movements are analyzed in the mouth shape data to extract relevant data. The changing characteristics of the tooth opening and closing movements are analyzed. The tongue tip movement trajectory is marked according to the tongue tip position data in the mouth shape data. The characteristics of the tongue tip movement and the abnormal tongue tip movement related to pronunciation errors are analyzed. Combined with the data of the lip, tooth position and tongue tip movement, comprehensive change feature data of the mouth shape are extracted. These features may include comprehensive features of mouth shape changes such as the movement speed of the lips and teeth, the position change of the tongue tip, etc. Based on the mouth shape change feature data, abnormal points or abnormal changes in the mouth shape data are marked to indicate abnormal conditions of the mouth shape movement. These abnormal point marks may cover abnormal lip, tooth position or tongue tip movements, which are directly related to pronunciation errors.
[0102] Step S5: Perform lip shape demonstration correction based on the comprehensive lip shape abnormality data to generate correction process data; perform difficulty clustering based on the correction process data to generate key and difficult pronunciation data; perform pronunciation training based on the key and difficult pronunciation data to generate optimized pronunciation data.
[0103] In an embodiment of the present invention, lip shape demonstration correction is performed by utilizing lip shape abnormality data, and correction operations are performed according to different types of lip shape abnormalities to generate corresponding correction process data. The pronunciation of the corrected follow-up data is re-scored. A preset score threshold is set, and a qualification assessment is performed to determine whether the preset score standard is met. If the re-scored data does not meet the preset qualification standard, secondary unqualified data is generated. Iterative error analysis is performed based on the secondary unqualified data, and further lip shape correction is performed. Clustering of difficult pronunciation data is performed based on the correction process data. Targeted training strategies are formulated based on the clustering results.
[0104] Preferably, step S1 comprises the following steps:
[0105] Step S11: Acquire demonstration pronunciation data and user follow-up reading data; mark special pronunciation points based on the demonstration pronunciation data to generate special pronunciation requirement data;
[0106] Step S12: performing short-time energy and fundamental frequency change analysis on the user's follow-up reading data to generate speech speed and intonation data; performing rhythm and prosody analysis on the speech speed and intonation data to generate rhythm and prosody feature data;
[0107] Step S13: performing signal conversion on the user's follow-up reading data to generate a follow-up reading spectrogram; performing voiceprint feature analysis based on the follow-up reading spectrogram to generate voiceprint feature data;
[0108] Step S14: using the rhythm and prosody feature data to correct feature deviations of the voiceprint feature data, and generating follow-up feature data;
[0109] Step S15: Perform comprehensive follow-up pronunciation scoring based on the follow-up feature data to generate comprehensive scoring factor data;
[0110] Step S16: scoring the special pronunciation of the rhythmic and prosodic feature data according to the special pronunciation requirement data to generate key scoring factor data;
[0111] Step S17: Perform a total pronunciation score based on the comprehensive scoring factor data and the key scoring factor data to generate a follow-up pronunciation score.
[0112] The present invention provides key guidance for users to pay attention to these pronunciations by marking special pronunciation points in the demonstration pronunciation, which may include tone changes, special phonemes, etc. Short-term energy and fundamental frequency changes analyze the speech speed and tone, while rhythm and prosody analysis provides information such as the rhythm and tone of the voice. These data can help users understand their own speech rhythm and tone, and help improve oral fluency and naturalness. Based on rhythm and prosody and voiceprint feature data, the features in the follow-up data are corrected to reduce pronunciation deviations and improve accuracy. The corrected data helps to improve the accuracy of users' pronunciation and make them closer to standard pronunciation. Combined with the follow-up feature data, multiple factors such as speech speed, tone, voiceprint, etc. are comprehensively considered to provide a comprehensive evaluation of the user's pronunciation performance. The comprehensive scoring factor data provides a multi-dimensional evaluation of the user's pronunciation ability, which helps users understand their performance in different aspects. According to the special pronunciation requirement data, special scoring is performed on key pronunciation points, focusing on the parts with special pronunciation requirements. These data highlight the performance of users at specific pronunciation points, so that the scoring pays more attention to the key points and areas that need improvement in pronunciation. Based on the comprehensive scoring factor data and key scoring factor data, the final pronunciation score is generated, taking into account all aspects of performance. The final pronunciation score provides an assessment of the user's overall pronunciation ability, provides comprehensive feedback to the user, and indicates the direction of improvement.
[0113] In an embodiment of the present invention, standard demonstration pronunciation data is obtained from recordings or other voice data sources, which can come from a voice library, voice teaching materials, or specific voice training resources. Collecting user follow-up data can be achieved by recording the user's voice or recording the user's oral expression. The demonstration pronunciation data is marked or annotated to identify special pronunciation points or specific pronunciation requirements. These marks can be acoustic features, pronunciation focuses, or special pronunciation guidance to help users better understand and imitate. Use speech signal processing technology (such as short-time Fourier transform, etc.) to analyze the energy and fundamental frequency (frequency of sound vibration) changes of the user follow-up data. Short-time energy can be used to detect the intensity of speech, and the fundamental frequency corresponds to the pitch of the sound. Analyze the changes in data such as energy and fundamental frequency, and derive the characteristic data of speech speed and intonation, including information such as speech speed, intonation rise and fall. Based on the speech speed and intonation data, the rhythm and prosody features in the speech are further analyzed, such as the stress position of the syllable, the pause between syllables, etc. This information helps to understand the fluency and rhythm of the speech. The user's follow-up voice signal is digitized and converted into a spectrogram (a display of the spectrum in time), which may use methods such as short-time Fourier transform or Mel frequency cepstrum coefficients. The generated spectrogram helps to visualize the user's voice characteristics for further analysis. Voiceprint feature extraction based on the spectrogram may include acoustic features such as formant frequency and vocal tract length. These voiceprint feature data help to identify and describe the individual's voice characteristics and may be used for tasks such as speech recognition or speech classification. The extracted rhythm and prosody feature data is used to analyze possible deviations or errors in the voiceprint feature data. By comparing the rhythm and prosody features with the voiceprint features, statistical methods or models may be used to correct the deviations in the voiceprint feature data to ensure that the user's actual pronunciation characteristics are more accurately reflected. After correction, the obtained voiceprint feature data is called follow-up feature data, which should more accurately describe the user's pronunciation characteristics. Based on the corrected follow-up feature data, a comprehensive score calculation of the follow-up pronunciation is performed. A variety of algorithms or models may be used to consider acoustic features, voice quality, accuracy and other factors to generate comprehensive scoring factor data. Based on the special pronunciation requirement data, the rhythmic prosody feature data is specially scored for specific pronunciation requirements or pronunciation points. This may include special evaluations of the accuracy, speaking speed, intonation, etc. of special pronunciation points. Key scoring factor data are extracted from the evaluation results of the special pronunciation requirement data, which may have an important impact on the final pronunciation score. Based on the comprehensive scoring factor data and the key scoring factor data, combined with the evaluation results of the special pronunciation requirements, the final total score of the follow-up pronunciation is calculated. Taking various factors into consideration, for example, a higher weight can be set for the key scoring phoneme data, so as to adjust the final score and generate the final follow-up pronunciation score, reflecting the user's overall pronunciation performance.
[0114] Preferably, step S2 comprises the following steps:
[0115] Step S21: screening low-score data of pronunciation scores of follow-up reading using a preset score threshold to generate pronunciation unqualified data;
[0116] Step S22: segmenting the follow-up reading feature data of the unqualified pronunciation data into error segments according to the comprehensive scoring factor data to generate follow-up reading error feature data;
[0117] Step S23: Marking phoneme boundaries based on the special pronunciation requirement data to generate phoneme boundary division data;
[0118] Step S24: extracting deviation phonemes from the rhythmic prosody feature data using the phoneme boundary division data to generate phoneme error data.
[0119] The present invention screens out low-scoring data through a score threshold, locates specific data with unqualified pronunciation, and provides quantitative information on specific problems. This helps to conduct error analysis and improvements more accurately in the future. The error segments in the unqualified pronunciation data are segmented according to the comprehensive scoring factor data to help locate the specific parts of the pronunciation errors. Targeted error segments are provided to make the problem more specific and more analyzable, providing a clearer direction for improvement. The phonemes are marked with boundaries according to special pronunciation requirement data so as to more accurately identify the specific positions of the phonemes. The division of phoneme boundaries helps to more accurately find the specific phonemes with pronunciation problems, making subsequent analysis more accurate. The phoneme boundary division data is used to extract the phoneme parts with deviations in the rhythm and prosody feature data. An accurate pronunciation problem location is provided to help more accurately identify specific pronunciation errors and make targeted improvements.
[0120] As an example of the present invention, refer to Figure 2 As shown, in this example, step S2 includes:
[0121] Step S21: screening low-score data of pronunciation scores of follow-up reading using a preset score threshold to generate pronunciation unqualified data;
[0122] In an embodiment of the present invention, a score threshold is set as a standard for judging the pronunciation quality. This threshold can first obtain a benchmark value based on expert evaluation, training results of a speech recognition model or other standards. For example, it is generally believed that a follow-up pronunciation below 70 points is unqualified, and this is used as a benchmark. It is then adjusted according to the age of the user. For example, the threshold corresponding to a younger user can be lower, so that the threshold is adjusted to 60 points. The user's follow-up pronunciation score is compared with a preset score threshold, and the data below the threshold is marked as unqualified pronunciation data. The follow-up data that does not reach the preset score threshold is regarded as unqualified pronunciation data and is marked or recorded.
[0123] Step S22: segmenting the follow-up reading feature data of the unqualified pronunciation data into error segments according to the comprehensive scoring factor data to generate follow-up reading error feature data;
[0124] In the embodiment of the present invention, based on the comprehensive scoring factor data, the problem in the pronunciation unqualified data is found, which may include problems in speech speed, pronunciation accuracy, intonation, etc. The follow-up feature data in the pronunciation unqualified data is analyzed in detail, and specific error segments are segmented and marked. For example, according to the specific paragraphs of the spectrogram or sound signal, the segments of possible errors or irregular pronunciations can be marked. The method of speech signal processing can be adopted, such as framing, time domain / frequency domain analysis or pattern recognition technology, to identify and segment specific error segments. These segmented segments will help to identify the specific parts of inaccurate or wrong pronunciation, and provide important basis for subsequent error correction and training.
[0125] Step S23: Marking phoneme boundaries based on the special pronunciation requirement data to generate phoneme boundary division data;
[0126] In an embodiment of the present invention, by analyzing special pronunciation requirement data, these data may come from the requirements put forward by phoneticians or voice training experts for specific pronunciations. These may include the pronunciation standard of a specific phoneme, the phonetic features during pronunciation, etc. According to the special pronunciation requirement data, the speech signal is analyzed to identify the boundary position of each phoneme. The phoneme boundary is the switching point between phonemes, which is used to divide the speech signal into different phoneme segments. This may require the use of signal processing technology, speech analysis algorithms or professional annotation tools to determine the switching positions of different phonemes in the speech signal. The identified phoneme boundary positions are recorded and marked to form phoneme boundary division data. These data will be used for subsequent phoneme error extraction.
[0127] Step S24: extracting deviation phonemes from the rhythmic prosody feature data using the phoneme boundary division data to generate phoneme error data.
[0128] In the embodiment of the present invention, by analyzing the rhythm and prosody feature data, these features may involve information on speech speed, pitch, rhythm, etc. The previously generated phoneme boundary division data is used to compare, extract and analyze the parts corresponding to specific phonemes in the rhythm and prosody feature data. These parts may have deviations or errors. According to the phoneme boundary division data, the parts of specific phonemes are extracted from the rhythm and prosody feature data, and further analyzed whether these parts have deviations outside the standards defined in the special pronunciation requirement data. For example, it is possible to identify possible pronunciation errors or deviations by comparing the differences between the extracted phoneme features and the standard phoneme requirement data. The found deviations or erroneous phoneme feature data are recorded as phoneme error data. These data will help to identify and correct specific problems in pronunciation and provide a basis for subsequent training.
[0129] Preferably, step S3 comprises the following steps:
[0130] Step S31: using a camera device to obtain face image data;
[0131] Step S32: using a placement orientation calculation formula and performing device placement orientation calculation based on pre-collected frontal face image data and face image data to generate pre-placement position data;
[0132] Step S33: adjusting the position of the camera device according to the pre-placement position data, and collecting the lip shape to generate the lip shape data;
[0133] Step S34: Screening the mouth shape data of the follow-up reading based on the unqualified pronunciation data to generate wrong mouth shape data.
[0134] The present invention captures facial image data through a camera device for subsequent extraction and analysis of lip shape information. Facial image data helps to analyze the shape and changes of the lip shape, and provides additional information for pronunciation evaluation. The placement position of the device is calculated using the pre-collected frontal face image data and the current face image data to ensure better capture of lip shape information. The measurement and adjustment of the device position helps to improve the accuracy and reliability of lip shape acquisition. Under the calibrated device position, a camera is used to capture lip shape data to record the user's lip shape changes and pronunciation process. The collection of lip shape data helps to capture dynamic information during lip shape changes and pronunciation. The lip shape data is screened based on the unqualified pronunciation data, and the lip shape data related to the pronunciation problem is extracted. The screened erroneous lip shape data helps to more accurately locate the lip shape problems related to the unqualified pronunciation, and provides direction for subsequent improvements.
[0135] As an example of the present invention, refer to Figure 3 As shown, in this example, step S3 includes:
[0136] Step S31: using a camera device to obtain face image data;
[0137] In the embodiment of the present invention, the camera device is ensured to acquire clear and high-quality facial image data. This may include the appropriate camera type, camera resolution, lighting conditions, etc. The camera device is used to acquire facial images. The subject may be required to aim at the camera and remain stable to acquire a clear facial image. The acquired facial image data needs to be saved and processed. This may include storage of image files, format conversion, or data transmission, etc.
[0138] Step S32: using a placement orientation calculation formula and performing device placement orientation calculation based on pre-collected frontal face image data and face image data to generate pre-placement position data;
[0139] In an embodiment of the present invention, a set of facial image data with a known placement orientation (usually the front) is collected in advance and used as reference data for measurement. A preset placement orientation measurement formula is used, which fully considers parameters such as the position coordinates of the reading pen and the position coordinates of the nose tip, as well as the pre-collected frontal face image data and the currently collected face image data, for comparison and calculation, so as to accurately evaluate the degree of deviation between the device and the face position, and generate pre-placement position data that recommends changing the placement orientation of the current camera based on the calculated data. In addition to the preset formula, the device placement orientation can also be measured and pre-placement position data can be generated through methods such as feature point detection and matching, three-dimensional reconstruction technology, neural networks or machine learning models.
[0140] Step S33: adjusting the position of the camera device according to the pre-placement position data, and collecting the lip shape to generate the lip shape data;
[0141] In an embodiment of the present invention, the orientation of the camera device is adjusted by using the pre-placement position data to ensure that it can correctly align with the user's mouth shape. This may include adjusting the angle, position or focus of the camera to ensure that clear mouth shape image data is obtained. After adjusting the orientation, the mouth shape is captured. This step requires the user to cooperate, keep the posture stable and clearly show the oral area. The mouth shape image data is captured by the camera device for subsequent analysis and processing. The collected mouth shape image data is saved, marked and stored for subsequent processing and analysis.
[0142] Step S34: Screening the mouth shape data of the follow-up reading based on the unqualified pronunciation data to generate wrong mouth shape data.
[0143] In an embodiment of the present invention, by analyzing previously generated unqualified pronunciation data, these data may contain information about lip shape errors or other pronunciation problems. Use the standards or indicators in the unqualified pronunciation data to screen the lip shape image data. This may include image screening and recognition based on specific standards (such as incorrect lip shape position, movement, shape, etc.). Identify the lip shape data that does not meet the requirements and record it as incorrect lip shape data. Record, classify or further analyze the identified incorrect lip shape data. This may include recording the type, location, duration and other information of the lip shape error for subsequent targeted training or correction.
[0144] Preferably, the placement orientation calculation formula in step S32 is as follows:
[0145] ;
[0146] ;
[0147] In the formula, For pre-placement position data, is the horizontal coordinate of the position of the point reading pen, is the device distance parameter, Place the device at an angle. is the vertical coordinate of the position of the point reading pen, is the inverse tangent function, is the ordinate of the nose tip position, is the horizontal coordinate of the nose tip position, is the maximum face length of the pre-collected frontal face image, is the maximum face length of the face image, is the maximum face width of the pre-collected frontal face image, is the maximum face width of the face image, is the offset angle of the device placement. To correct the angle.
[0148] The present invention constructs a placement orientation calculation formula, which is used to calculate the device placement orientation based on the pre-collected face front image data and face image data, and generate pre-placement position data. The formula fully considers the horizontal coordinate of the point reading pen position. , device distance parameters , equipment placement angle , the vertical coordinate of the point reading pen position , the inverse tangent function , the vertical coordinate of the nose tip position , the horizontal coordinate of the nose tip position , the maximum face length of the pre-collected frontal face image , the maximum face length of the face image , the maximum face width of the pre-collected frontal face image , the maximum face width of the face image , the offset angle of the device placement , Correction Angle And the interaction relationship between variables, forming the following functional relationship:
[0149] ;
[0150] ;
[0151] pass and Determines the relative reference point of the device in the pre-placement position data. By determining the coordinates of the reference point, the placement of the device can be clarified and affect the determination of the final pre-placement position. Indicates the distance between the device and the reference object. By adjusting the distance parameter, you can control the distance between the device and the reference object. A larger distance parameter will make the device farther away from the reference object, while a smaller distance parameter will make the device closer to the reference object. It is the device placement azimuth. By calculating the angle difference between the reading pen and the face position, combined with the offset angle, the device placement angle can be determined. By adjusting this angle, the device orientation can be changed to face different directions of the target object. and is the coordinate of the nose tip in the face image data. By comparing the position of the reading pen with the position coordinate of the face, the device placement angle can be calculated. Changes in these coordinates will directly affect The calculation result of The calculation can be done by calculating the azimuth of the device placement The change in the relative size of the face is taken into account to further accurately determine the placement of the device. A larger difference value indicates that the height change is more significant than the width change, and the angle value will be larger; conversely, a smaller difference value indicates that the height and width change trends are similar, and the angle value will be closer to zero. By comparing the facial midline position of the acquired facial image data with the actual midline position, the offset angle of the device placement can be obtained. , thereby correcting the deviation of the device placement to adapt to the user usage at different times and locations. This functional relationship can accurately evaluate the degree of deviation between the device and the face position, thereby calculating the reasonable device placement position to obtain clear face image data, providing a basis for subsequent lip shape collection and analysis. And using To correct the angle, adjust the functional relationship to reduce the impact of data deviation or error. For example, when the light is poor, the possible placement angle can be determined through foresight experience, so as to Make adjustments and corrections to more accurately generate the device placement angle , improve the accuracy of equipment orientation adjustment. At the same time, the coordinate values and correction values in the formula are adjusted according to the actual situation and applied to different users and usage environments, which improves the flexibility and applicability of the algorithm.
[0152] Preferably, step S4 comprises the following steps:
[0153] Step S41: performing lip opening and closing analysis on the incorrect lip shape data to generate lip opening and closing movement data; performing opening time image interception on the incorrect lip shape data based on the lip opening and closing movement data to generate opening image data;
[0154] Step S42: performing tooth opening and closing analysis based on the tooth opening and closing image data to generate tooth opening and closing motion data; performing tooth opening time image interception on the tooth opening image data based on the tooth opening and closing motion data to generate tooth opening image data;
[0155] Step S43: performing noise reduction and smoothing processing on the open-teeth image data to generate clear open-teeth image data; marking the tongue tip position according to the clear open-teeth image data to generate tongue tip motion trajectory data;
[0156] Step S44: performing time-series correlation on the lip opening and closing motion data, the tooth position opening and closing motion data, and the tongue tip motion trajectory data to generate mouth shape change feature data;
[0157] Step S45: marking abnormal change points according to the lip shape change feature data to generate lip shape abnormal range data;
[0158] Step S46: marking the abnormal time point of the lip shape abnormal range data according to the phoneme error data to generate comprehensive lip shape abnormal data.
[0159] The present invention performs lip opening and closing analysis on mouth shape data and extracts lip movement data, which is helpful to understand the movement of lips during pronunciation. Based on the lip opening and closing movement data, the mouth opening image data is intercepted to record the opening and closing of lips during pronunciation. The tooth position opening and closing analysis is performed using the mouth opening image data, and the tooth position opening and closing movement data is extracted to understand the dynamic change of the tooth position during pronunciation. Based on the tooth position opening and closing movement data, the tooth opening image data is intercepted to record the change of the tooth position during pronunciation. The tooth opening image data is subjected to noise reduction and smoothing processing to reduce the noise in the image and extract clear tooth opening image data. Based on the clear tooth opening image data, the tongue tip position is marked and the tongue tip movement trajectory data is extracted to record the movement trajectory of the tongue tip during pronunciation. The lip opening and closing movement data, the tooth position opening and closing movement data and the tongue tip movement trajectory data are time-series associated to integrate and form mouth shape change feature data. A more comprehensive mouth shape feature is provided, which is helpful to fully understand the mouth shape change. The abnormal change point is marked according to the mouth shape change feature data, and the mouth shape abnormal range is determined, which may involve abnormal lip, tooth position or tongue tip movement. Determine the range of lip shape abnormalities so that subsequent analysis and improvements can be more focused on specific lip shape abnormality intervals. Mark the abnormal time points of the lip shape abnormality range data based on the phoneme error data, and associate the lip shape abnormality with specific phoneme errors. Associate the lip shape abnormality data with pronunciation errors to provide more accurate correlation analysis between lip shape and pronunciation errors for improvement.
[0160] As an example of the present invention, refer to Figure 4 As shown, in this example, step S4 includes:
[0161] Step S41: performing lip opening and closing analysis on the incorrect lip shape data to generate lip opening and closing movement data; performing opening time image interception on the incorrect lip shape data based on the lip opening and closing movement data to generate opening image data;
[0162] In the embodiment of the present invention, the position and boundary of the lips are first detected by image processing technology. By comparing the changes in the position of the lips in consecutive frames, the opening and closing movement of the lips is calculated, and a specific image processing algorithm, such as a method based on edge detection, template matching or neural network, can be used. According to the lip opening and closing movement data, the image frame at the opening moment is identified. The frame image related to the opening of the mouth is intercepted and extracted from the video stream or image sequence.
[0163] Step S42: performing tooth opening and closing analysis based on the tooth opening and closing image data to generate tooth opening and closing motion data; performing tooth opening time image interception on the tooth opening image data based on the tooth opening and closing motion data to generate tooth opening image data;
[0164] In the embodiment of the present invention, the position and opening and closing of teeth in the oral region are detected by image processing technology. The opening and closing movement of teeth is calculated by comparing the position and opening and closing of teeth in consecutive frames. The image frame at the moment of opening teeth is identified based on the tooth position opening and closing movement data. The frame image related to the opening and closing of teeth is intercepted from the video stream or image sequence.
[0165] Step S43: performing noise reduction and smoothing processing on the open-teeth image data to generate clear open-teeth image data; marking the tongue tip position according to the clear open-teeth image data to generate tongue tip motion trajectory data;
[0166] In an embodiment of the present invention, the intercepted open-tooth image data is subjected to noise reduction and smoothing processing to improve the image quality and clarity, and image processing technologies such as image enhancement, filtering, and denoising can be used. Image processing technologies such as edge detection, feature point extraction (such as Harris corner point detection, SIFT, SURF, etc.), color analysis, etc. are used to identify the position of the tongue tip in the oral cavity. According to the characteristics of the oral image, the oral region is segmented to locate the area of the tongue tip. The contours of various parts in the oral cavity are obtained by edge detection and other technologies, and the edge shape is further analyzed to locate the tongue tip. If the color of the tongue tip is different from that of the surrounding tissue, a color segmentation technology (such as color threshold segmentation, color space conversion, etc.) can be used to detect a specific color area as the tongue tip. A target tracking algorithm (such as Kalman filtering, mean tracking, motion model, etc.) is used to track the position of the tongue tip in continuous frame images. The tongue tip positions in continuous frames are associated to construct time series data of the tongue tip movement. The tongue tip position data in continuous image frames are used to obtain the tongue tip movement trajectory data in the oral cavity.
[0167] Step S44: performing time-series correlation on the lip opening and closing movement data, the tooth position opening and closing movement data, and the tongue tip movement trajectory data to generate mouth shape change feature data;
[0168] In an embodiment of the present invention, the lip opening and closing, tooth opening and closing, and tongue tip motion trajectory data are time-aligned to ensure that they correspond to the same lip shape state at the same time point. These data are synchronized using methods such as timestamps or inter-frame matching. The characteristics of lip opening and closing, tooth opening and closing, and tongue tip motion trajectory data are combined to extract the characteristics of lip shape changes. The speed, amplitude, continuity, and other characteristics of lip shape changes can be analyzed, and statistical methods, time series models, or machine learning techniques can be used to capture the patterns and characteristics of lip shape changes.
[0169] Step S45: marking abnormal change points according to the lip shape change feature data to generate lip shape abnormal range data;
[0170] In an embodiment of the present invention, the abnormal points in the lip shape change are identified by comparing the lip shape change feature data with the change features of the demonstration pronunciation data or using an abnormality detection algorithm (for example, based on thresholds, volatility, deviations, etc.). Detecting points in the lip shape change data that deviate greatly from the demonstration pronunciation data indicates abnormal lip shape or irregular oral movement. Based on the abnormal points and the surrounding data, the time range or interval of the lip shape abnormality is defined. The abnormal range can be determined based on the temporal position of the abnormal point and the changes in the lip shape change data around it. The error range can also be directly determined by following the error feature data, thereby intercepting the abnormal change points in the lip shape change feature data and generating lip shape abnormal range data.
[0171] Step S46: marking the abnormal time point of the lip shape abnormal range data according to the phoneme error data to generate comprehensive lip shape abnormal data.
[0172] In the embodiment of the present invention, the abnormal time range extracted from the lip shape abnormal range data is time-sequentially associated with the phoneme error data. The time point corresponding to the factor error data is marked in the lip shape abnormal range data. The lip shape abnormal range data, the abnormal time point and the related phoneme error data are integrated. Comprehensive lip shape abnormal data can be established, including the abnormal time range, type and corresponding lip shape and phoneme error information.
[0173] Preferably, step S44 includes the following steps:
[0174] Step S441: Time-sequentially associate the lip opening and closing movement data, the tooth position opening and closing movement data, and the tongue tip movement trajectory data to generate mouth shape movement change data;
[0175] Step S442: performing trajectory image conversion on the lip movement change data to generate a two-dimensional lateral movement graph;
[0176] Step S443: using the angle change evaluation formula and the two-dimensional lateral motion diagram to perform motion tendency analysis on the tongue tip motion trajectory data to generate motion direction feature data;
[0177] Step S444: using the motion direction feature data to perform trajectory fuzzy estimation on the tongue tip motion trajectory data to generate tongue tip motion optimization data;
[0178] Step S445: performing motion supplementation on the two-dimensional lateral motion map according to the tongue tip motion optimization data to generate a lateral motion optimization map;
[0179] Step S446: Perform motion feature evaluation based on the lateral motion optimization graph to generate lip shape change feature data.
[0180] In an embodiment of the present invention, the synchronization of data at the same time point is ensured by aligning the data in time. The lip motion data from different sources are temporally associated by using methods such as time series analysis and correlation analysis. The lip motion data after time series association is converted into a visualized two-dimensional image. According to the motion data of the lips, teeth and tongue tip, a corresponding two-dimensional lateral motion image is constructed, that is, a visualized two-dimensional lip motion image observed from the side. The angle change of the motion trajectory is calculated using the position data of the relevant points in the lip motion image. The angle change evaluation formula is used to evaluate the change trend of the motion. The formula fully considers variables such as the evaluation time and the horizontal and vertical coordinates of the tongue tip position, so as to efficiently calculate the tendency of the tongue tip change direction and speed in each pronunciation process, so as to predict the tongue tip movement status that cannot be recorded when the teeth are closed. According to the result of the angle change evaluation, the tendency and direction of the tongue tip movement are judged. Based on the two-dimensional lateral motion map, key motion direction feature data, such as the horizontal or vertical direction of the tongue tip movement, are extracted. The previously obtained motion direction feature data is used to estimate the tongue tip motion trajectory data. Use fuzzy logic, statistical methods or deep learning technology to fuzzily estimate the tongue tip motion trajectory. This helps to deal with the changes and uncertainties of the trajectory motion. Integrate the tongue tip motion trajectory data after fuzzy estimation into a set of optimized data. Mark this data for use in subsequent steps. Combine the tongue tip motion optimization data with the existing two-dimensional lateral motion image. Integrate the optimization data, supplement and update the two-dimensional lateral motion map, and ensure that the optimization data is reflected and presented in the image. Use computer vision or image processing technology to detect key feature points or contours in the lateral motion optimization map. Apply edge detection algorithms (such as Sobel, Canny, etc.) to extract image edges or regions of interest. Extract key contour information of mouth shape movement from the image through methods such as curve fitting and shape analysis. Use feature descriptors (such as SIFT, SURF, HOG, etc.) to extract description information of key features from the image. Convert the extracted feature descriptors into numerical vectors. Integrate the motion feature data extracted from the lateral motion optimization map with other feature data related to mouth shape movement to generate mouth shape change feature data, including information such as lips, teeth positions, and direction and speed of tongue tip movement.
[0181] Preferably, the angle change evaluation formula in step S443 is as follows:
[0182] ;
[0183] In the formula, is the average change in angle, is the start time of the assessment, is the end time of the assessment, is the index value of time change, is the arccosine function, For time The horizontal coordinate of the tongue tip position at For time The horizontal coordinate of the tongue tip position at For time The ordinate of the tongue tip position at For time The ordinate of the tongue tip position at For time The horizontal coordinate of the tongue tip position at For time The ordinate of the tongue tip position at is the angle deviation correction value.
[0184] The present invention constructs an angle change evaluation formula for analyzing the movement tendency of the tongue tip movement trajectory data based on the two-dimensional lateral movement graph to generate movement direction feature data. The formula fully considers the start time of the evaluation. , the end time of the evaluation , the index value of time change , the arccosine function , time is The horizontal coordinate of the tongue tip position at , time is The horizontal coordinate of the tongue tip position at , time is The ordinate of the tongue tip position at , time is The ordinate of the tongue tip position at , time is The horizontal coordinate of the tongue tip position at , time is The ordinate of the tongue tip position at , Angle deviation correction value And the interaction relationship between variables, forming the following functional relationship:
[0185] ;
[0186] pass and , indicating the position of the tongue tip at time point and In the formula, they form the vectors of two adjacent time points and are used to calculate the angle change. The third point used in the formula to calculate the angle between two vectors helps determine the direction of the two vectors. Calculates the angle between two vectors. This function divides the inner product by the product of the modulos and then inversely calculates the angle between two vectors. Its function is to convert the angle between the vectors into an actual angle. The angle is summed over a specified time range by using the summation symbol to calculate the total change in angle over that time range. It is used to normalize the sum of the angle values to the average value within the time range. This average value represents the average rate of change of the angle. This functional relationship can quickly evaluate the position and speed of the tongue tip change during each pronunciation process, so as to predict the movement state of the tongue tip during the mouth closure period, so as to improve the accuracy of the subsequent pronunciation error position assessment. Using the angle deviation correction value Correcting the functional relationship reduces the error impact caused by abnormal data and error data, thereby generating the average angle change value more accurately. , which improves the accuracy and reliability of the calculation results. At the same time, the correction value in the formula can be adjusted according to the actual situation and applied to different trajectory data to calculate their motion tendency, which improves the flexibility and applicability of the algorithm.
[0187] Preferably, step S5 comprises the following steps:
[0188] Step S51: performing lip shape demonstration error correction based on the comprehensive lip shape abnormality data to generate error correction follow-up reading data;
[0189] Step S52: re-scoring the pronunciation of the error correction and follow-up reading data to generate re-scoring data; using a preset score threshold to evaluate the eligibility of the re-scoring data, if the re-scoring data is greater than or equal to the preset score threshold, summarizing the error correction process and generating error correction process data;
[0190] Step S53: if the re-scoring data is less than the preset score threshold, secondary unqualified data is generated, and repeated error analysis and error correction are performed based on the secondary unqualified data until error correction process data is generated;
[0191] Step S54: performing clustering of difficult points according to the error correction process data to generate important and difficult point pronunciation data;
[0192] Step S55: Formulate training strategies based on the key and difficult pronunciation data to generate training strategy data;
[0193] Step S56: Perform pronunciation training according to the training strategy data to generate optimized pronunciation data.
[0194] The present invention performs lip shape demonstration correction by integrating abnormal lip shape data, and generates error correction and follow-up reading data. The error correction and follow-up reading data provides a demonstration of the correct lip shape to help users understand and imitate the correct lip shape. The pronunciation of the error correction and follow-up reading data is re-scored to generate re-scoring data. A pre-set score threshold is used for qualification assessment. If it meets the pre-set standard, error correction process data is generated to summarize the error correction process. If the re-scoring data is lower than the pre-set score threshold, secondary unqualified data is generated to perform iterative error analysis and error correction. The error correction process is iteratively analyzed to continuously improve the spoken pronunciation until the pre-set qualification assessment standard is reached. Based on the error correction process data, the difficulties in the error correction process are clustered and divided, and the key and difficult pronunciation data are distinguished. Through clustering, specific key and difficult points in pronunciation are determined to help carry out subsequent training more targetedly. According to the key and difficult pronunciation data, targeted training strategies are formulated, including specific exercises, skills and methods. Targeted training strategies are formulated to make the training closer to the individual's pronunciation problems and improve the training effect.
[0195] In an embodiment of the present invention, the lip shape abnormality present in the shadow reading is determined by using the lip shape abnormality data. The abnormal lip shape phenomenon is analyzed to determine the possible causes, such as inaccurate pronunciation position, irregular lip shape, etc. According to the detected abnormal lip shape, a demonstration or guidance is provided to correct the problem, such as using a demonstration video, a demonstration picture or a text description. The correct lip shape, tongue position, lip shape, etc. are explained in detail to guide the user to imitate the correct lip shape. The guidance information for correcting the lip shape is applied to the shadow reading data to generate error correction shadow reading data, that is, new shadow reading data after error correction. The pronunciation of the error correction shadow reading data is scored, and the scoring method mentioned in the aforementioned step S1 is used, and the preset score threshold mentioned in the aforementioned step S2 is used to determine whether the re-scored data meets the qualification standard. If the re-scored data meets or exceeds the preset score threshold, it is deemed qualified; otherwise, it is deemed unqualified. The data that fails the qualification assessment is marked as secondary unqualified data. The secondary unqualified data is returned to step S2 and then executed to generate phoneme error data in the unqualified data, comprehensive lip shape abnormal data, etc., and error correction is performed again according to the generated analysis data, and scoring and re-judgment is performed. If the score is still unqualified, the process of analysis, error correction, scoring and re-judgment is repeated until the score reaches the preset score threshold, and the error position, cause and other data obtained from each analysis are summarized to generate error correction process data. The characteristics of various lip shape errors, such as pronunciation error type, frequency, lip shape characteristics, etc., are extracted from the error correction process data. Cluster analysis is performed on the lip shape error data using a clustering algorithm (such as K-means, DBSCAN, etc.), and similar errors are classified. According to the clustering results, the key and difficult pronunciation data, i.e., those lip shape errors that are difficult to master or occur frequently, are determined. The characteristics and commonalities of the difficult pronunciation data are analyzed, and training strategies are formulated to correct these errors in a targeted manner. According to the pronunciation characteristics of individuals, personalized training programs are formulated, including practice content, frequency, intensity, etc. To help users practice oral English more effectively. Implement training plans for difficult pronunciations, which can be oral practice, pronunciation courses, repeated reading, etc. Regularly evaluate the training results, collect user pronunciation data, and re-score or compare to test progress. Adjust the training strategy based on the evaluation results, which may require modifying the training content and strengthening specific exercises to achieve better pronunciation results.
[0196] Preferably, step S54 includes the following steps:
[0197] Step S541: aggregate the same error words in the error correction process data to generate aggregated word data;
[0198] Step S542: Estimating the error frequency based on the aggregated vocabulary data to generate error correction frequency distribution data;
[0199] Step S543: performing phoneme error frequency estimation based on the error correction frequency distribution data to generate phoneme frequency distribution data;
[0200] Step S544: clustering the same phonemes according to the phoneme frequency distribution data to generate aggregated phoneme data;
[0201] Step S545: re-divide the aggregated phoneme data by phoneme position to generate important and difficult pronunciation data.
[0202] The present invention aggregates the same erroneous words in the error correction process data to generate aggregated vocabulary data. Aggregating the same erroneous words with high frequency helps to identify common pronunciation problems and centrally process them. Estimating the error frequency based on the aggregated vocabulary data generates error correction frequency distribution data. Providing the distribution of pronunciation errors helps to understand the frequency and severity of different errors. Estimating the phoneme error frequency based on the error correction frequency distribution data generates phoneme frequency distribution data. Providing the frequency and distribution of specific phoneme errors helps to identify the frequent errors of specific phonemes in pronunciation problems. According to the phoneme frequency distribution data, the data of the same phoneme in the pronunciation errors are aggregated and divided to generate aggregated phoneme data. By aggregating the phoneme data, the same phonemes with high frequency are clustered so as to more effectively identify the key phonemes in the pronunciation. Based on the aggregated phoneme data, the positions of the phonemes are re-divided to distinguish the key and difficult pronunciation data in the pronunciation process. In order to more accurately determine which phonemes are more prone to errors or more difficult to pronounce in the pronunciation process, further targeted training and improvement can be carried out.
[0203] In an embodiment of the present invention, the same erroneous words are classified and categorized according to the erroneous words in the error correction process data. The words of the same erroneous pronunciation or error type are aggregated to form aggregated vocabulary data. The aggregated vocabulary data is statistically analyzed to calculate the frequency of each erroneous pronunciation or error type. The number of times each error occurs in the data set is determined, and the frequency distribution is calculated. The number of times each phoneme is pronounced incorrectly is extracted from the aggregated vocabulary data. The frequency distribution of each phoneme error is calculated based on the number of occurrences of the phoneme error. Based on the phoneme frequency distribution data, the same phonemes in different vocabularies are aggregated to form aggregated phoneme data. For each phoneme in the aggregated phoneme data, their positions in the vocabulary are marked. The same phoneme may appear in any position of the front, middle, or back of a vocabulary. The aggregated phoneme data is redivided based on the position in the vocabulary. This identifies and determines which phonemes are more likely to make mistakes in specific positions or specific environments, and marks them as key and difficult pronunciation data.
[0204] The beneficial effect of the present application is that the present invention generates a pronunciation score by comparing and scoring the demonstration pronunciation and the user's follow-up reading data. This step evaluates the accuracy and quality of the user's pronunciation. A standardized scoring method is provided to help users understand their pronunciation level. It also provides basic data for subsequent error analysis. By setting a preset score threshold, data with unqualified pronunciation are identified and screened out. These data may include parts with low pronunciation accuracy. The unqualified pronunciation data are analyzed in detail to identify the phoneme error data therein. These error data identify specific pronunciation problems and help to accurately locate errors. The lip shape data is collected by a camera device for analyzing the user's lip shape movements. Based on the unqualified pronunciation data, the data with incorrect mouth shape is screened, and these data can include the incorrect parts of the user's lip shape movements when pronouncing. The change feature analysis of the wrong mouth shape data is performed to identify the change feature of the mouth shape movement. According to the mouth shape change feature data, the abnormal points in the mouth shape movement are marked and the mouth shape abnormal data is identified. Based on the comprehensive mouth shape abnormal data, the lip shape demonstration error correction is performed to guide the user's correct lip shape movement. According to the error correction process data, the key and difficult points in the pronunciation process are identified, which may be specific parts of the mouth shape movement. Training is carried out on the pronunciation data of the key and difficult points to help users improve their pronunciation and improve the correctness and accuracy of their mouth shape movements. Therefore, the speech recognition method of the point-reading pen of the present invention not only obtains the phoneme details through fine-grained analysis, but also performs mouth shape error analysis on the user's follow-up mouth shape data, thereby comprehensively evaluating the user's incorrect pronunciation points to improve the accuracy of error correction and improve the user's pronunciation habits.
[0205] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.
[0206] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.
Claims
1. A speech recognition method for a point reading pen, characterized in that: The following steps are involved: Step S1: Acquire demonstration pronunciation data and user follow-up reading data; Score the user's follow-up pronunciation data based on the demonstration pronunciation data to generate a follow-up pronunciation score; Step S2: screening low-score data of the pronunciation scores of the follow-up reading using a preset score threshold to generate pronunciation-unqualified data; Perform fine-grained analysis on the pronunciation error data to generate phoneme error data; Step S3: using a camera device to collect lip shape data and generate lip shape data; Based on the unqualified pronunciation data, the incorrect lip shape data is screened to generate incorrect lip shape data; Step S4: Analyze the change characteristics of the erroneous lip shape data to generate lip shape change characteristic data; Marking abnormal points according to the lip shape change feature data to generate comprehensive lip shape abnormal data; wherein step S4 includes the following steps: Step S41: performing lip opening and closing analysis on the incorrect lip shape data to generate lip opening and closing movement data; performing opening time image interception on the incorrect lip shape data based on the lip opening and closing movement data to generate opening image data; Step S42: performing tooth opening and closing analysis based on the tooth opening and closing image data to generate tooth opening and closing motion data; performing tooth opening time image interception on the tooth opening image data based on the tooth opening and closing motion data to generate tooth opening image data; Step S43: performing noise reduction and smoothing processing on the open-teeth image data to generate clear open-teeth image data; marking the tongue tip position according to the clear open-teeth image data to generate tongue tip motion trajectory data; Step S44: Time-sequentially associate the lip opening and closing movement data, the tooth opening and closing movement data, and the tongue tip movement trajectory data to generate mouth shape change feature data; wherein step S44 includes the following steps: Step S441: Time-sequentially associate the lip opening and closing movement data, the tooth position opening and closing movement data, and the tongue tip movement trajectory data to generate mouth shape movement change data; Step S442: performing trajectory image conversion on the lip movement change data to generate a two-dimensional lateral movement graph; Step S443: using the angle change evaluation formula and performing a motion tendency analysis on the tongue tip motion trajectory data according to the two-dimensional lateral motion diagram to generate motion direction feature data; wherein the angle change evaluation formula is as follows: ; In the formula, is the average change in angle, is the start time of the assessment, is the end time of the assessment, is the index value of time change, is the arccosine function, For time The horizontal coordinate of the tongue tip position at For time The horizontal coordinate of the tongue tip position at For time The ordinate of the tongue tip position at For time The ordinate of the tongue tip position at For time The horizontal coordinate of the tongue tip position at For time The ordinate of the tongue tip position at is the angle deviation correction value; Step S444: using the motion direction feature data to perform trajectory fuzzy estimation on the tongue tip motion trajectory data to generate tongue tip motion optimization data; Step S445: performing motion supplementation on the two-dimensional lateral motion map according to the tongue tip motion optimization data to generate a lateral motion optimization map; Step S446: performing motion feature evaluation based on the lateral motion optimization graph to generate lip shape change feature data; Step S45: marking abnormal change points according to the lip shape change feature data to generate lip shape abnormal range data; Step S46: marking the abnormal time point of the lip shape abnormal range data according to the phoneme error data to generate comprehensive lip shape abnormal data; Step S5: Perform lip shape demonstration correction based on the comprehensive lip shape abnormality data to generate correction process data; perform difficulty clustering based on the correction process data to generate key and difficult pronunciation data; perform pronunciation training based on the key and difficult pronunciation data to generate optimized pronunciation data.
2. The speech recognition method of the point reading pen according to claim 1, characterized in that: Step S1 includes the following steps: Step S11: Acquire demonstration pronunciation data and user follow-up reading data; mark special pronunciation points based on the demonstration pronunciation data to generate special pronunciation requirement data; Step S12: performing short-time energy and fundamental frequency change analysis on the user's follow-up reading data to generate speech speed and intonation data; performing rhythm and prosody analysis on the speech speed and intonation data to generate rhythm and prosody feature data; Step S13: performing signal conversion on the user's follow-up reading data to generate a follow-up reading spectrogram; performing voiceprint feature analysis based on the follow-up reading spectrogram to generate voiceprint feature data; Step S14: using the rhythm and prosody feature data to correct feature deviations of the voiceprint feature data, and generating follow-up feature data; Step S15: Perform comprehensive follow-up pronunciation scoring based on the follow-up feature data to generate comprehensive scoring factor data; Step S16: scoring the special pronunciation of the rhythmic and prosodic feature data according to the special pronunciation requirement data to generate key scoring factor data; Step S17: Perform a total pronunciation score based on the comprehensive scoring factor data and the key scoring factor data to generate a follow-up pronunciation score.
3. The speech recognition method of the point reading pen according to claim 2, characterized in that: Step S2 includes the following steps: Step S21: screening low-score data of pronunciation scores of follow-up reading using a preset score threshold to generate pronunciation unqualified data; Step S22: segmenting the follow-up reading feature data of the unqualified pronunciation data into error segments according to the comprehensive scoring factor data to generate follow-up reading error feature data; Step S23: Marking phoneme boundaries based on the special pronunciation requirement data to generate phoneme boundary division data; Step S24: extracting deviation phonemes from the rhythmic prosody feature data using the phoneme boundary division data to generate phoneme error data.
4. The speech recognition method of the point reading pen according to claim 3, characterized in that: Step S3 includes the following steps: Step S31: using a camera device to obtain face image data; Step S32: using a placement orientation calculation formula and performing device placement orientation calculation based on pre-collected frontal face image data and face image data to generate pre-placement position data; Step S33: adjusting the position of the camera device according to the pre-placement position data, and collecting the lip shape to generate the lip shape data; Step S34: Screening the mouth shape data of the follow-up reading based on the unqualified pronunciation data to generate wrong mouth shape data.
5. The speech recognition method of the point reading pen according to claim 4, characterized in that: The placement orientation calculation formula in step S32 is as follows: ; ; In the formula, For pre-placement position data, is the horizontal coordinate of the position of the point reading pen, is the device distance parameter, Place the device at an angle. is the vertical coordinate of the position of the point reading pen, is the inverse tangent function, is the ordinate of the nose tip position, is the horizontal coordinate of the nose tip position, is the maximum face length of the pre-collected frontal face image, is the maximum face length of the face image, is the maximum face width of the pre-collected frontal face image, is the maximum face width of the face image, is the offset angle of the device placement. To correct the angle.
6. The speech recognition method of the point reading pen according to claim 1, characterized in that: Step S5 includes the following steps: Step S51: performing lip shape demonstration error correction based on the comprehensive lip shape abnormality data to generate error correction follow-up reading data; Step S52: re-scoring the pronunciation of the error correction and follow-up reading data to generate re-scoring data; using a preset score threshold to evaluate the eligibility of the re-scoring data, if the re-scoring data is greater than or equal to the preset score threshold, summarizing the error correction process and generating error correction process data; Step S53: if the re-scoring data is less than the preset score threshold, secondary unqualified data is generated, and repeated error analysis and error correction are performed based on the secondary unqualified data until error correction process data is generated; Step S54: performing clustering of difficult points according to the error correction process data to generate important and difficult point pronunciation data; Step S55: Formulate training strategies based on the key and difficult pronunciation data to generate training strategy data; Step S56: Perform pronunciation training according to the training strategy data to generate optimized pronunciation data.
7. The speech recognition method of the point reading pen according to claim 6, characterized in that: Step S54 includes the following steps: Step S541: aggregate the same error words in the error correction process data to generate aggregated word data; Step S542: Estimating the error frequency based on the aggregated vocabulary data to generate error correction frequency distribution data; Step S543: performing phoneme error frequency estimation based on the error correction frequency distribution data to generate phoneme frequency distribution data; Step S544: clustering the same phonemes according to the phoneme frequency distribution data to generate aggregated phoneme data; Step S545: re-divide the aggregated phoneme data by phoneme position to generate important and difficult pronunciation data.
