Human language auxiliary learning device and working method thereof

Abstract

The invention belongs to the technical field of human language auxiliary learning, and discloses a human language auxiliary learning device and a working method thereof. The human language auxiliary learning device comprises a data acquisition module which is used for acquiring voice of a single conversation of a speaker and video data of a mouth shape corresponding to the voice; the feature extraction module is used for obtaining voice and a corresponding mouth shape video feature coding sequence; the data processing module is used for acquiring a final language and character information sequence which can be understood by the physiological age group of the preset hearing-impaired person; and the feedback module is used for retrieving and matching the pre-stored standard mouth shape video clip so as to feed back and guide the preset hearing-impaired person to simulate and sound through the vocal cord. According to the technical scheme disclosed by the invention, the education investment of families and society on the congenital hearing disorder can be greatly reduced, and the language learning and communication efficiency and effect of the families and society are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of human language-assisted learning, and relates to the processing and analysis of voice and video data, and in particular to a human language-assisted learning device and a working method thereof. Background Art

[0002] Globally, the vast majority of hearing-impaired people do not have congenital language defects. The functions of the language-related organs in their bodies are normal when they are born. However, due to their congenital hearing impairments and lack of effective education and guidance during their personal growth process, they lose both their hearing and language functions in adulthood.

[0003] At present, the existing technical means cannot solve the problem of oral communication between people with congenital hearing impairment (hereinafter referred to as hearing-impaired people) and people with normal language function (hereinafter referred to as normal people). The specific situation includes the following two aspects. On the one hand, although there are many cases of successful oral communication between hearing-impaired people around the world (for example, Helen Keller, the author of "If I Had Three Days of Light", etc.), traditional deaf-mute education institutions need to make uninterrupted investment for many years in the process of educating and guiding them to communicate orally, and often due to factors such as educational methods and individual differences in education, there is a problem of poor oral communication effect. On the other hand, when the existing technical solutions realize communication between hearing-impaired people and normal people, they mainly use technical means such as converting the voice or mouth shape of normal people into text and converting the sign language of hearing-impaired people into voice. It is not possible to realize direct communication between the two parties through voice, and there are problems such as information omission or distortion and low communication efficiency. Summary of the invention

[0004] The purpose of the present invention is to provide a human language auxiliary learning device and a working method thereof to solve one or more of the above-mentioned technical problems. The technical solution disclosed in the present invention can guide hearing-impaired people of different ages who suffer from hearing impairment but have normal language-related organs to communicate with normal people through voice in a step-by-step manner in their daily lives without sending them to special education institutions and without increasing the burden of raising and educating their parents. This can greatly reduce the educational investment of families and society in hearing-impaired people and significantly improve the efficiency and effectiveness of their language learning and oral communication.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a human language auxiliary learning device, comprising: A data acquisition module, used to acquire the speech of a single conversation of a speaker and the video data of the mouth shape corresponding to the speech based on the conversation scene in which the human language auxiliary learning device is located; wherein the speaker is a preset hearing-impaired person or a normal person; A feature extraction module is used to extract features from the speech and the video data of the speech corresponding to the mouth shape, and obtain a speech and corresponding mouth shape video feature coding sequence; The data processing module is used to use the trained natural language recognition and standard scoring model adapted to the selected physiological age group to perform recognition processing on the speech and the corresponding lip shape video feature coding sequence to obtain the initial language and text information sequence that can be expressed by the speaker's preset physiological age group, and synchronously convert the initial language and text information sequence into the final language and text information sequence that can be understood by the preset hearing-impaired physiological age group; the module also uses the trained natural language recognition and standard scoring model adapted to the selected physiological age group to perform scoring processing on the speech and the corresponding lip shape video feature coding sequence to obtain the final scoring result; A feedback module is used to retrieve and match pre-stored standard mouth shape video clips based on the language and text information sequence, so as to guide the preset hearing-impaired person to imitate and make sounds through the vocal cords through feedback; wherein, if the speaker is the preset hearing-impaired person, then when the final score result is lower than a preset threshold, multiple video clips corresponding to the final score result that can be understood by the physiological age group of the preset hearing-impaired person are pushed multiple times to guide the correction of the vocal cord pronunciation of the preset hearing-impaired person.

[0006] A further improvement of the present invention is that In the feature extraction module, the step of performing feature extraction on the speech and the video data of the speech corresponding to the mouth shape to obtain the speech and the corresponding mouth shape video feature coding sequence specifically includes: The audio data corresponding to the speech is segmented according to the time interval to obtain the speech segmentation result; based on the speech segmentation result, the video data is segmented into a video including multiple continuous frames according to the pronunciation of the word; the mouth area of ​​each frame in each continuous frame is segmented, and the segmented pictures of each frame are aligned, and then the lip edge segmentation of each frame segmented picture is performed through edge detection to obtain lip continuous frames; In addition, a human face detection model is used to perform human face detection on multiple continuous frames and remove the background to obtain continuous frames of human faces.

[0007] A further improvement of the present invention is that The step of performing lip edge segmentation on each frame segmentation picture through edge detection to obtain lip continuous frames comprises: Based on the segmented pictures of each frame, a Gaussian filter is used to smooth the image to obtain a two-dimensional Gaussian distribution; Based on the obtained two-dimensional Gaussian distribution, the Sobel operator is used to calculate the first-order derivatives of the image in the horizontal and vertical directions to obtain the horizontal gradient map And the vertical gradient map ; Among them, the Sobel operator is and ; According to the horizontal gradient map obtained And the vertical gradient map , calculate the gradient of the boundary and direction ; where the gradient of the boundary and direction The calculation expressions are: ; ; Boundary-based gradient and direction , use non-maximum suppression and double threshold detection to determine the edge points, connect the edge points and segment the shape of the lips to obtain continuous lip frames.

[0008] A further improvement of the present invention is that In the data processing module, the steps of using the trained natural language recognition and standard scoring model adapted to the selected physiological age group to perform recognition processing on the speech and the corresponding mouth shape video feature coding sequence, obtaining an initial language and text information sequence that can be expressed by the speaker's preset physiological age group, and synchronously converting the initial language and text information sequence into a final language and text information sequence that can be understood by the preset hearing-impaired physiological age group include: Based on the acquired continuous lip frames and continuous face frames, the initial language and text information sequence that can be expressed by the current speaker's preset physiological age group is identified through a natural language recognition and standard scoring model that has been trained to adapt to the selected physiological age group, and the words are concatenated and synchronously converted into a final language and text information sequence that can be understood by the preset hearing-impaired person's physiological age group using a language understanding model; The natural language recognition and standard scoring model adapted to the selected physiological age group includes: a physiological age group recognition model based on human face and a language recognition model based on continuous lip frames.

[0009] A further improvement of the present invention is that The face-based physiological age group recognition model adopts a convolutional neural network; During the training process of the face-based physiological age recognition model, The loss function used It is expressed as: ; In the formula, represents the predicted age; Represents the image label; represents the standard deviation of the label; The evaluation index used is the mean absolute error MAE, and the calculation expression of MAE is: ; In the formula, N Indicates the total quantity; , Estimated age and real age respectively.

[0010] A further improvement of the present invention is that During the training process of the language recognition model based on continuous lip frames, The training is performed using a lip reading video training set; wherein the recognition result of the face-based physiological age group recognition model is used as a basis, and lip reading video data of different age groups are used to train multiple model parameter versions; During the training process, spatial convolution is used to perform three-dimensional convolution on continuous frames to use time sequence and image information to recognize lip pronunciation; based on the word results obtained by spatial convolution, Bi-LSTM is used to convert the words into understandable language and text information sequences; the CTC loss function is used to optimize the model parameters; When in use, according to the age result determined by the physiological age group recognition model based on the face, a language recognition model based on lip continuous frames with different age group parameter versions is adopted to perform lip reading recognition.

[0011] A further improvement of the present invention is that In the step of using the trained natural language recognition and standard scoring model adapted to the selected physiological age group to score the speech and the corresponding mouth shape video feature coding sequence to obtain the final scoring result, Final rating results S The calculation expression is: ; In the formula, , They are audio score and mouth shape score respectively; , They are the weighted values ​​of audio score and mouth shape score respectively.

[0012] A further improvement of the present invention is that ; In the formula, is the recognition result confidence of the language recognition model based on continuous lip frames; is the track similarity; Among them, the track similarity The acquisition steps are: using the audio track feature extractor to extract the video audio track features, and performing cosine similarity comparison with the standard audio track features.

[0013] A further improvement of the present invention is that Mouth shape score The acquisition step is to compare the lip continuous frames with the standard video continuous frames of the corresponding physiological age group after lip extraction frame by frame, and use the pHash algorithm to calculate the similarity of each frame and weight it.

[0014] A second aspect of the present invention provides a method for operating a human language auxiliary learning device, comprising the following steps: Based on the conversation scene in which the human language auxiliary learning device is located, the data acquisition module is used to acquire the speech of a single conversation of a speaker and the video data of the mouth shape corresponding to the speech; wherein the speaker is a preset hearing-impaired person or a normal person; Using a feature extraction module to extract features from the speech and the video data of the speech corresponding to the mouth shape, to obtain a speech and corresponding mouth shape video feature coding sequence; The data processing module uses a trained natural language recognition and standard scoring model adapted to the selected physiological age group to perform recognition processing on the speech and the corresponding mouth shape video feature coding sequence to obtain an initial language and text information sequence that can be expressed by the speaker's preset physiological age group, and synchronously converts the initial language and text information sequence into a final language and text information sequence that can be understood by the preset hearing-impaired person's physiological age group; wherein, if the speaker is a preset hearing-impaired person, the trained natural language recognition and standard scoring model adapted to the selected physiological age group is also used to perform scoring processing on the speech and the corresponding mouth shape video feature coding sequence to obtain a final scoring result; Through the feedback module, based on the final language and text information sequence, matching pre-stored standard mouth shape video clips are retrieved to guide the preset hearing-impaired person to imitate and make sounds through the vocal cords through feedback; wherein, if the speaker is the preset hearing-impaired person, then when the final scoring result is lower than the preset threshold, multiple video clips corresponding to the final scoring results that can be understood by the physiological age groups of the preset hearing-impaired people are pushed multiple times to guide the correction of the vocal cord pronunciation of the preset hearing-impaired person.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a human language auxiliary learning device. This technical solution can quickly convert the content of the conversation into text information by real-time capture and recognition of voice and mouth shape, reducing the problem of information omission or distortion and improving communication efficiency; by providing a standard mouth shape video clip as an imitation object, it helps the hearing-impaired learn the correct pronunciation and mouth shape, thereby promoting oral communication between them and normal people. In addition, the present invention can provide personalized language and text information conversion and feedback according to the speaker's physiological age and pronunciation level. For the hearing-impaired, the device can also push targeted video clips for correction and guidance according to their pronunciation scores, thereby realizing personalized teaching. In addition, traditional deaf-mute education institutions require years of investment, and the effects often vary depending on factors such as educational methods and individual differences in education. The technical solution of the present invention reduces the learning burden of educators and hearing-impaired people and improves the educational effect through automation and intelligent methods. Through the language auxiliary learning device of the present invention, the hearing-impaired will be able to better integrate into society and enjoy a richer and more fulfilling life experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 is a schematic diagram of a human language auxiliary learning device in an embodiment of the present invention; Figure 2 is a schematic diagram of a voice information processing flow in an embodiment of the present invention; Figure 3 is a flowchart of an application example of a human language auxiliary learning device in a specific embodiment of the present invention; Figure 4 It is a schematic diagram of the architecture of an application example of a human language assisted learning device in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments and technical solutions are only part of the embodiments of the present invention, not all of the embodiments.

[0019] All other embodiments obtained by those of ordinary skill in the art without creative work based on the technical solutions disclosed in the embodiments of the present invention belong to the scope of protection of the present invention. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] See also Figure 1 , an embodiment of the present invention provides a human language auxiliary learning device, comprising: A data acquisition module, used to acquire the speech of a speaker in a single conversation and the video data of the mouth shape corresponding to the speech based on the conversation scene in which the human language auxiliary learning device is located; wherein the speaker is a preset hearing-impaired person or a normal person; A feature extraction module is used to extract features from the speech and the video data of the speech corresponding to the mouth shape, and obtain a speech and corresponding mouth shape video feature coding sequence; A data processing module is used to use a trained natural language recognition and standard scoring model adapted to a selected physiological age group to perform recognition processing on the speech and the corresponding lip shape video feature coding sequence, obtain an initial language and text information sequence that can be expressed by the speaker's preset physiological age group (exemplarily, it can be a language and text information sequence including characters, words, etc.), and synchronously convert the initial language and text information sequence into a final language and text information sequence that can be understood by a preset hearing-impaired person's physiological age group; in a further preferred technical solution, if the speaker is a preset hearing-impaired person, the speech and the corresponding lip shape video feature coding sequence are also scored using the trained natural language recognition and standard scoring model adapted to the selected physiological age group to obtain a final scoring result; The feedback module is used to retrieve and match the pre-stored standard mouth shape video clips (for example, standard mouth shape animation clips or life scene animation clips, etc.) based on the final language and text information sequence, so as to provide feedback to guide the preset hearing-impaired person to imitate and make sounds through the vocal cords; in a further preferred technical solution, if the speaker is the preset hearing-impaired person, then when the final score result is lower than the preset threshold, multiple video clips corresponding to the final score result that can be understood by the physiological age group of the preset hearing-impaired person are pushed multiple times to guide the correction of the vocal cord pronunciation of the preset hearing-impaired person. In actual applications, it can be repeated multiple times until the preset hearing-impaired person can make sounds through the vocal cord standard.

[0021] The embodiment of the present invention provides a human language auxiliary learning device, which is intended to solve the problem of oral communication difficulties between hearing-impaired people and normal people in the prior art. In the technical solution of the embodiment of the present invention, data acquisition and feature extraction are realized through a data acquisition module and a feature extraction module, which can provide a basis for subsequent processing; natural language recognition and scoring are realized through a data processing module, and an initial language and text information sequence that can be expressed by the speaker's preset physiological age group can be obtained, and the initial language and text information sequence is converted into a final language and text information sequence that is easy for the hearing-impaired to understand; in the feedback module, according to the final language and text information sequence, a pre-stored standard mouth shape video clip is retrieved and matched, and the hearing-impaired person is guided to imitate and speak through the vocal cords. In addition, the voice and mouth shape of the hearing-impaired person when speaking can be scored to evaluate the accuracy and fluency of their pronunciation. If the pronunciation score of the hearing-impaired person is lower than the preset threshold, the device will push multiple video clips suitable for their physiological age group multiple times to help them correct their pronunciation. In summary, the human language auxiliary learning device disclosed in the embodiment of the present invention promotes oral learning of the hearing-impaired and reduces the educational burden by technical means such as real-time capture and recognition of voice and mouth shape, and provision of personalized language and text information conversion and feedback. It can effectively solve the remaining technical problems in oral communication between the hearing-impaired and normal people and achieve remarkable results.

[0022] Based on the above disclosed embodiments, in a specific embodiment of the present invention, a system for assisting hearing-impaired people in language learning is provided, including: a data acquisition module, an information exchange control center, a feature extraction module, a data processing module and a feedback module; wherein, The data acquisition module specifically adopts a voice and video information acquisition terminal, which is responsible for collecting the speaker's voice and the video data of the mouth shape corresponding to the voice; The information exchange control center is responsible for receiving information from the language information collection terminal and forwarding it to the feature extraction module; The feature extraction module and the data processing module recognize and process the received voice and the video data of the voice corresponding to the mouth shape, convert them into multimedia information such as the speaker's image, action and video of the standard mouth shape, and feed this information back to the information exchange control center; The information exchange control center sends the feedback information to the feedback module. The feedback module can specifically adopt a multimedia interactive terminal, which receives and synchronously displays the information. In addition, the speaker's daily behavior video information can be collected and transmitted back to the information exchange control center. The information exchange control center synchronously uploads and processes the daily behavior video information, and applies the processing results to the subsequent recognition of voice and mouth shape video information and the generation of video information.

[0023] The technical solution of the embodiment of the present invention realizes the collection, processing, conversion and display functions of the speaker's voice, mouth shape and daily behavior video information, and is particularly suitable for assisting language learning, especially for the language learning of the hearing-impaired.

[0024] In an embodiment of the present invention, a working method of a human language auxiliary learning device is provided, comprising the following steps: Step 1, based on the conversation scene in which the human language auxiliary learning device is located, obtain the voice of a single conversation of a speaker and the video data of the mouth shape corresponding to the voice; wherein the speaker is a preset hearing-impaired person or a normal person.

[0025] Step 2, extracting features from the speech and the video data of the speech corresponding to the mouth shape, and obtaining a speech and corresponding mouth shape video feature coding sequence; The audio data of the speech is segmented according to the time interval, and based on the segmentation result, the video data is segmented into video A according to the pronunciation of the word. Video A consists of n consecutive frames. Combined; for each continuous frame Each frame in Perform mouth area segmentation and divide each frame After alignment, edge detection is used to segment the lip edges of each frame to obtain continuous lip frames. ; In addition, a Gaussian filter is used to smooth the image, and the two-dimensional Gaussian distribution is: ; Gradient calculation: Use the Sobel operator to calculate the first-order derivatives (i.e., first-order partial derivatives) of the image in the horizontal and vertical directions. The Sobel operator is: and ; According to the obtained and The gradient map finds the gradient and direction of the boundary, which are expressed as: ; ; Use non-maximum suppression and double threshold detection to determine edge points, and then connect the edge points to segment the shape of the lip; Use the face detection model to detect consecutive frames Perform person detection, remove background, and obtain continuous frames of faces .

[0026] Step 3, based on the acquired lip continuous frames and continuous frames of faces , through a pre-trained natural language recognition and standard scoring model adapted to a specific physiological age group, the initial language and text information sequence that can be expressed by the current speaker's preset physiological age group is identified, and the language understanding model is used to concatenate words and synchronously convert them into a final language and text information sequence that can be understood by the preset hearing-impaired person's physiological age group; wherein, if the speaker is a preset hearing-impaired person, his or her voice effect is synchronously scored to form a final scoring result.

[0027] In the embodiment of the present invention, the natural language recognition and standard scoring model for a specific physiological age group is divided into: a physiological age group recognition model based on human face and a language recognition model based on continuous lip frames; wherein, The physiological age recognition model uses the MORPH dataset to pre-train face data, and the loss function used is as follows: ; In the formula, represents the predicted age; Represents the image label; represents the standard deviation of the label; The mean absolute error (MAE) is used as the evaluation index of the model. The MAE formula is as follows: ; That is, the average absolute error between the estimated age and the actual age. The smaller the MAE, the smaller the error range.

[0028] The language recognition model based on continuous lip frames is pre-trained using the lip reading video training set. Based on the model recognition results of physiological age groups, lip reading video data of different age groups are used to train multiple model parameter versions. Among them, spatial convolution is used to perform three-dimensional convolution on continuous frames to recognize lip reading pronunciation using time series and image information. The specific steps are as follows: The three-dimensional convolution formula is:

[0029] Among them, W is the convolution weight, X is the three-dimensional convolution result, c is the number of convolutions, kt, kw, kh are the length, width and height of the convolution kernel.

[0030] According to the word results obtained by the spatial convolution model, Bi-LSTM is used to convert the words into understandable language and text information sequences; wherein, the above formula adopts the CTC loss function to optimize the model, and when the system is used, it adopts parameter version models of different age groups for lip reading recognition based on the age results determined by the physiological age group recognition model.

[0031] The final lip reading recognition results and audio track sequences obtained based on the pre-trained model are compared and scored with the standard vocalization videos of the same physiological age group. The scoring model is composed of mouth shape and pronunciation. The audio track feature extractor E is used to extract the video audio track features. ,Will Same as standard audio track features Perform cosine similarity comparison, the formula is as follows: ; Get the track similarity Confidence of the results after the lip sync video continuous frame model recognition Together they form the score, and the total confidence score is .

[0032] The lip shape scoring model takes consecutive frames of lips Continuous frames of standard video for the corresponding physiological age group after lip extraction Perform frame-by-frame comparison, use pHash algorithm to calculate the similarity of each frame, and weightedly obtain the mouth shape similarity score.

[0033] The specific exemplary steps are as follows: (1) Extract the corresponding frame image and reduce its size to 8*8; (2) Calculate DCT, select the 8*8 matrix in the upper left corner, and the DCT two-dimensional transform formula is: ; Where A is an orthogonal matrix: , a is , b is , c is ; (3) Calculate the DCT mean; (4) Hash value calculation and Hamming distance calculation, the similarity is , the final video mouth shape score is expressed as: ; In summary, the final mouth shape score and audio score are weighted averaged to obtain the final score result, and the calculation expression is: ; in, and It is the weighted value of the audio score and the mouth shape score, which can be adjusted adaptively.

[0034] Step 4: Based on the final language and text information sequence obtained in step 3, pre-stored standard mouth shape animation clips and life scene animation clips are matched and retrieved to provide feedback to the preset hearing-impaired person, guiding him to imitate and make sounds through his vocal cords. If the speaker is a preset hearing-impaired person, based on the final scoring result formed in step 3, pre-stored scoring result animation clips that can be understood by the physiological age group of the preset hearing-impaired person are matched and retrieved to guide him to independently evaluate and improve his vocal cord sound effect.

[0035] See also Figure 2 The language understanding model is trained using language data of different age groups to train language understanding model parameters of different age groups. The system uses recognition parameters of different age groups to recognize words as language sentences that can be understood by the corresponding age groups. In a specific embodiment of the present invention, the processing flow after receiving the voice information includes the following steps: Step 1: Receive a language information communication data packet; explanatory, this step involves receiving a language information data packet transmitted via communication; illustratively, the data packet can be received via a wireless network or a wired network, and parsed and read using a suitable communication protocol.

[0036] Step 2: Parsing the language information communication data packet; Explanatoryally, this step parses the received data packet to extract useful language information; Step 3: Speech recognition and processing; Explanatoryally, this step recognizes the speech signal from the data packet and may perform preliminary processing. In an exemplary optional technical solution, speech recognition technologies such as hidden Markov model (HMM) and deep neural network (DNN) can be applied to convert the speech signal into text data. The processing stage may include noise removal, voice activity detection, etc.

[0037] Step 4: Image recognition and processing; Explanatoryally, this step analyzes and recognizes the image data and may perform necessary processing. In an exemplary optional technical solution, a convolutional neural network (CNN) can be used for image recognition, and the recognition results can be used for image classification, target detection, etc. The processing stage may include image enhancement, denoising, scaling, etc.

[0038] Step 5: Multimedia information generation and return; Explanatoryally, this step generates corresponding multimedia information based on the results of speech recognition and image recognition, and returns it. In an exemplary optional technical solution, a composite file containing multimedia information such as text, images, and videos is generated and returned to the user through a user interface or a communication interface.

[0039] Step 6: Language information data read and stored; Explanatory, this step stores the processed language information data in a storage medium. In an exemplary optional technical solution, a relational database or a NoSQL database is used to store language information data, including text data, voice data, etc., and an index is established to improve data retrieval efficiency.

[0040] Step 7: Language information data set analysis and processing; Explanatoryally, this step further analyzes and processes the stored language information data set, which may involve data mining, pattern recognition, etc.

[0041] In the embodiment of the present invention, through the above steps, the entire process implements the complete logic from receiving the language information data packet to high-level data analysis and processing. Each step provides a reference for a specific technical method, which helps to understand and implement the entire process.

[0042] See also Figure 3 and Figure 4 In a specific embodiment of the present invention, an application example of a human language auxiliary learning device is provided, which is explained in detail through the following steps: 1. Basic data collection: Equivalent to a data acquisition module, the device is in a conversation scene and collects the speaker's single conversation voice and video data of the corresponding mouth shape of the voice; wherein the speaker can be a preset hearing-impaired person or a normal person.

[0043] 2. Language recognition model training: Using the data acquisition module and feature extraction module, we collect and extract voice and video data features for hearing-impaired people and normal people at different physiological age stages, and then train a specific natural language recognition model to identify the language information and needs of hearing-impaired people.

[0044] 3. Normal person language recognition model training: Similarly, but for normal people, the corresponding natural language recognition model is trained to accurately recognize and understand the language expressions of normal people.

[0045] 4. Creation of multimedia library of life scene animation: Create a multimedia library containing life scene animations for subsequent generation of corresponding life scene animation clips to help the hearing-impaired understand language through a combination of vision and hearing.

[0046] 5. Identify and determine the physiological age and language comprehension level of the hearing-impaired: Using the data processing module, the physiological age and language comprehension level of the hearing-impaired person can be identified through the natural language recognition and standard scoring model trained to adapt to the selected physiological age group.

[0047] 6. Identify normal people’s speech information and generate corresponding life scene animation clips: The data processing module recognizes the speech information of normal people and generates corresponding life scene animation clips from the life scene animation multimedia library according to the recognition results.

[0048] 7. Identify the language information and needs of the hearing-impaired, provide correct feedback to strengthen their understanding and correct their pronunciation: The data processing module identifies the language information and needs of the hearing-impaired, and displays standard lip shape video clips through the feedback module to guide the hearing-impaired to imitate and make sounds through the vocal cords. If the score is lower than the preset threshold, relevant video clips are pushed multiple times to correct their pronunciation.

[0049] 8. Evaluate the learning effect and optimize the models in steps 2 and 3: Continuously evaluate the learning outcomes of the hearing-impaired and optimize and adjust the language recognition models for the hearing-impaired and normal people based on the feedback results to improve the accuracy and effectiveness of recognition.

[0050] 9. Repeat steps 4 to 8: Continuously cycle through the optimization steps to gradually improve the system and enhance the overall learning effect and user experience.

[0051] In the embodiment of the present invention, the human language auxiliary learning device can effectively help the hearing-impaired people to get enhanced training in language comprehension and expression through the above steps, and improve the learning effect through multimedia and animation; wherein, by collecting the voice, mouth shape and life scene video data of the hearing-impaired people and the normal people around them in the daily communication process, using big data analysis and machine learning algorithms, the physiological age, language comprehension level and corresponding demand information of the hearing-impaired people are identified, and the language, expression and behavior of the normal people are matched to generate multimedia video information that can be understood by them, guide and help the hearing-impaired people to make correct sounds through the vocal cords, and finally communicate with the normal people in daily life through language. The technical scheme of the present invention provides a way to replace the traditional educational institutions and families to realize the oral education of the hearing-impaired, which can effectively reduce the education cost of the whole society and family and improve the learning efficiency; in addition, the teaching model of the hearing-impaired people provided by the present invention can greatly improve the problem of poor education effect caused by the capital investment, the personal quality of teachers and parents and the individual intelligence differences of the hearing-impaired people in the traditional education model.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A human language auxiliary learning device, characterized in that: include: A data acquisition module, used to acquire the speech of a single conversation of a speaker and the video data of the mouth shape corresponding to the speech based on the conversation scene in which the human language auxiliary learning device is located; wherein the speaker is a preset hearing-impaired person or a normal person; A feature extraction module is used to extract features from the speech and the video data of the speech corresponding to the mouth shape, and obtain a speech and corresponding mouth shape video feature coding sequence; A data processing module, used to use the trained natural language recognition and standard scoring model adapted to the selected physiological age group to perform recognition processing on the speech and the corresponding lip shape video feature coding sequence, obtain an initial language and text information sequence that can be expressed by the speaker's preset physiological age group, and synchronously convert the initial language and text information sequence into a final language and text information sequence that can be understood by the preset hearing-impaired person's physiological age group; wherein, if the speaker is a preset hearing-impaired person, the trained natural language recognition and standard scoring model adapted to the selected physiological age group is also used to perform scoring processing on the speech and the corresponding lip shape video feature coding sequence to obtain a final scoring result; A feedback module is used to retrieve and match pre-stored standard mouth shape video clips based on the final language and text information sequence, so as to guide the preset hearing-impaired person to imitate and make sounds through the vocal cords through feedback; wherein, if the speaker is the preset hearing-impaired person, then when the final score result is lower than a preset threshold, multiple video clips corresponding to the final score results that can be understood by the physiological age group of the preset hearing-impaired person are pushed multiple times to guide the correction of the vocal cord pronunciation of the preset hearing-impaired person.

2. A human language auxiliary learning device according to claim 1, characterized in that: In the feature extraction module, the step of performing feature extraction on the speech and the video data of the speech corresponding to the mouth shape to obtain the speech and the corresponding mouth shape video feature coding sequence specifically includes: The audio data corresponding to the speech is segmented according to the time interval to obtain the speech segmentation result; based on the speech segmentation result, the video data is segmented into a video including multiple continuous frames according to the pronunciation of the word; the mouth area of ​​each frame in each continuous frame is segmented, and the segmented pictures of each frame are aligned, and then the lip edge segmentation of each frame segmented picture is performed through edge detection to obtain lip continuous frames; In addition, a human face detection model is used to perform human face detection on multiple continuous frames and remove the background to obtain continuous frames of human faces.

3. A human language auxiliary learning device according to claim 2, characterized in that: The step of performing lip edge segmentation on each frame segmentation picture through edge detection to obtain lip continuous frames comprises: Based on the segmented pictures of each frame, a Gaussian filter is used to smooth the image to obtain a two-dimensional Gaussian distribution; Based on the obtained two-dimensional Gaussian distribution, the Sobel operator is used to calculate the first-order derivatives of the image in the horizontal and vertical directions to obtain the horizontal gradient map And the vertical gradient map ; Among them, the Sobel operator is and ; According to the horizontal gradient map obtained And the vertical gradient map , calculate the gradient of the boundary and direction ; where the gradient of the boundary and direction The calculation expressions are: ； ； Boundary-based gradient and direction , use non-maximum suppression and double threshold detection to determine the edge points, connect the edge points and segment the shape of the lips to obtain continuous lip frames.

4. The human language auxiliary learning device according to claim 2, characterized in that: In the data processing module, the steps of using the trained natural language recognition and standard scoring model adapted to the selected physiological age group to perform recognition processing on the speech and the corresponding mouth shape video feature coding sequence, obtaining an initial language and text information sequence that can be expressed by the speaker's preset physiological age group, and synchronously converting the initial language and text information sequence into a final language and text information sequence that can be understood by the preset hearing-impaired physiological age group include: Based on the acquired continuous lip frames and continuous face frames, the initial language and text information sequence that can be expressed by the current speaker's preset physiological age group is identified through a natural language recognition and standard scoring model that has been trained to adapt to the selected physiological age group, and the words are concatenated and synchronously converted into a final language and text information sequence that can be understood by the preset hearing-impaired person's physiological age group using a language understanding model; The natural language recognition and standard scoring model adapted to the selected physiological age group includes: a physiological age group recognition model based on human face and a language recognition model based on continuous lip frames.

5. A human language auxiliary learning device according to claim 4, characterized in that: The face-based physiological age group recognition model adopts a convolutional neural network; During the training process of the face-based physiological age recognition model, The loss function used It is expressed as: ； In the formula, represents the predicted age; Represents the image label; represents the standard deviation of the label; The evaluation index used is the mean absolute error MAE, and the calculation expression of MAE is: ； In the formula, N Indicates the total quantity; , Estimated age and real age respectively.

6. The human language auxiliary learning device according to claim 4, characterized in that: During the training process of the language recognition model based on continuous lip frames, The training is performed using a lip reading video training set; wherein the recognition result of the face-based physiological age group recognition model is used as a basis, and lip reading video data of different age groups are used to train multiple model parameter versions; During the training process, spatial convolution is used to perform three-dimensional convolution on continuous frames to use time sequence and image information to recognize lip pronunciation; based on the word results obtained by spatial convolution, Bi-LSTM is used to convert the words into understandable language and text information sequences; the CTC loss function is used to optimize the model parameters; When in use, according to the age result determined by the physiological age group recognition model based on the face, a language recognition model based on lip continuous frames with different age group parameter versions is adopted to perform lip reading recognition.

7. The human language auxiliary learning device according to claim 4, characterized in that: In the step of using the trained natural language recognition and standard scoring model adapted to the selected physiological age group to score the speech and the corresponding mouth shape video feature coding sequence to obtain the final scoring result, Final rating results S The calculation expression is: ； In the formula, , They are audio score and mouth shape score respectively; , They are the weighted values ​​of audio score and mouth shape score respectively.

8. The human language auxiliary learning device according to claim 7, characterized in that: ； In the formula, is the recognition result confidence of the language recognition model based on continuous lip frames; is the track similarity; Among them, the track similarity The acquisition steps are: using the audio track feature extractor to extract the video audio track features, and performing cosine similarity comparison with the standard audio track features.

9. The human language auxiliary learning device according to claim 7, characterized in that: Mouth shape score The acquisition step is to compare the lip continuous frames with the standard video continuous frames of the corresponding physiological age group after lip extraction frame by frame, and use the pHash algorithm to calculate the similarity of each frame and weight it.

10. A method for operating the human language assisted learning device according to claim 1, characterized in that: The following steps are involved: Based on the conversation scene in which the human language auxiliary learning device is located, the data acquisition module is used to acquire the speech of a single conversation of a speaker and the video data of the mouth shape corresponding to the speech; wherein the speaker is a preset hearing-impaired person or a normal person; Using a feature extraction module to extract features from the speech and the video data of the speech corresponding to the mouth shape, to obtain a speech and corresponding mouth shape video feature coding sequence; The data processing module uses a trained natural language recognition and standard scoring model adapted to the selected physiological age group to perform recognition processing on the speech and the corresponding mouth shape video feature coding sequence to obtain an initial language and text information sequence that can be expressed by the speaker's preset physiological age group, and synchronously converts the initial language and text information sequence into a final language and text information sequence that can be understood by the preset hearing-impaired person's physiological age group; wherein, if the speaker is a preset hearing-impaired person, the trained natural language recognition and standard scoring model adapted to the selected physiological age group is also used to perform scoring processing on the speech and the corresponding mouth shape video feature coding sequence to obtain a final scoring result; Through the feedback module, based on the final language and text information sequence, matching pre-stored standard mouth shape video clips are retrieved to guide the preset hearing-impaired person to imitate and make sounds through the vocal cords through feedback; wherein, if the speaker is the preset hearing-impaired person, then when the final scoring result is lower than the preset threshold, multiple video clips corresponding to the final scoring results that can be understood by the physiological age groups of the preset hearing-impaired people are pushed multiple times to guide the correction of the vocal cord pronunciation of the preset hearing-impaired person.

Images