Children language rehabilitation training method and system based on voice and mouth shape recognition

Abstract

The invention relates to the technical field of children language rehabilitation training, and provides a children language rehabilitation training system based on voice and mouth shape recognition, which comprises a mouth shape simulation frame, a voice and mouth shape recognition module, a voice and mouth shape recognition module and a voice and mouth shape recognition module, an external camera is used for tracking and simulating the lips and the face of the training object in a model animation mode; and the tracking and capturing module is used for collecting face data of the training object and feeding back the face data to the model of the simulation mouth shape frame so as to fit the face of the training object and improve the training conformity. According to the invention, specific facial features and mouth shapes of each child can be individually fitted, customized rehabilitation training is provided, the mouth shapes of the children can be better fitted, more accurate pronunciation mouth shapes can be guided to be made, meanwhile, guide animations and roles which attract the attention of the children can be generated, the participation degree of the children is improved, the rehabilitation training is more interactive and interesting, and the interestingness of the children is improved. And the language learning interest and effect of the children can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of children's language rehabilitation training, and in particular to a children's language rehabilitation training method and system based on speech and lip shape recognition. Background Art

[0002] Hearing impairment is one of the more common diseases in any country. More and more families with hearing-impaired children send their children to specialized rehabilitation centers for hearing and speech therapy. At the same time, parents should also help hearing-impaired children with corresponding rehabilitation training and consolidation exercises at home, so that they can achieve the basic indicators of hearing and speech rehabilitation assessment before school age and acquire more complete listening and speaking abilities.

[0003] Existing rehabilitation training courses are relatively conservative in design and lack innovation. Repeated pronunciation exercises and repeated corrections of a sound or a word will make children feel bored and uninterested during training. In addition, facing the same incorrect pronunciation, children will feel a certain sense of frustration and dare not continue to pronounce, which affects the effect of hearing and speech training for hearing-impaired children and their enthusiasm and initiative to participate in the training. At present, auxiliary hearing and speech rehabilitation training products mainly use graphics, text, audio and other technologies to present teaching content in a straightforward manner. Although they can fully cover various rehabilitation training contents, they are still lacking in the fun and interactive areas of training, and rarely consider the experience of hearing-impaired children when using them. Therefore, there is a need for a children's language rehabilitation training method and system based on speech and lip recognition. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a method and system for children's language rehabilitation training based on speech and lip shape recognition, which solves the problem that the prior children's language rehabilitation training is too boring, affecting the effectiveness of auditory speech training for hearing-impaired children and their enthusiasm and initiative in participating in the training.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0006] A children's language rehabilitation training system based on speech and lip recognition, comprising:

[0007] The lip-simulating frame has a built-in model of lips and part of the face with a movable skeleton. It uses an external camera to track and imitate the lips and face of the training subject in the form of model animation.

[0008] The tracking and capturing module is used to collect the facial data of the training object and feed the facial data back to the model of the simulated lip frame to fit the face of the training object and improve the fit of the training;

[0009] The training animation generation module is used to generate guiding animations and guiding roles suitable for children in conjunction with the simulated lip-sync framework, so as to increase the interest of the children as training objects and avoid children feeling bored after long-term training;

[0010] The voice collection module is used to collect the sound of the training object during training. An external microphone is connected to collect the sound. The collected sound is compared with the training standard and fed back to the training process and the model of the simulated lip frame.

[0011] The training planning module is used to store the training plan and training process, and serves as the judgment standard for training and sends instructions to the voice acquisition module and the training animation generation module to execute the training plan.

[0012] Preferably, the tracking and capturing module comprises a facial scanning unit, a model adjustment unit, a grid anchoring unit and a feedback collection unit which are electrically connected to each other.

[0013] Preferably, the training animation generation module includes an animation model framework, a scene generation unit, a training object simulation image generation unit and a training wizard simulation image generation unit which are electrically connected to each other.

[0014] Preferably, the voice collection module includes a voice band comparison unit, a volume threshold prompt unit, and a sound feedback unit which are electrically connected to each other.

[0015] Preferably, the training planning module includes a training program storage unit, a training process generation unit, a training qualification standard determination unit, a training guidance instruction output unit and a training report generation unit which are electrically connected to each other.

[0016] A children's language rehabilitation training method based on speech and lip recognition comprises the following steps:

[0017] Step 1: The face and lips of the training object are captured by the camera, and the model mouth shape in the mouth shape frame is simulated and corrected by the tracking and capturing module to make it closer to the mouth shape of the training object;

[0018] Step 2: Generate a training object simulation image and a training guide simulation image in sequence using a training animation generation module according to the mouth shape data provided in step 1;

[0019] Step 3: Plan the training content according to the training planning module, set the phased training levels, and use the training wizard to simulate the image of the simulation animation to guide the training object to pronounce;

[0020] Step 4: During the training process, the voice collection module collects the voice of the training object, and the camera collects the mouth shape of the training object, which is fed back into the simulated image of the training object;

[0021] Step 5: The voice collection module performs accuracy matching detection based on the collected voice;

[0022] Step 6: If the test is passed, enter the next stage of training.

[0023] Preferably, the specific steps of correcting the model mouth shape in step 1 are:

[0024] S1. First, scan the face of the training object, and cover the grid layer with the scanned facial image;

[0025] S2, using a grid anchoring unit to set anchor points on the grid overlay layer of the facial image, and also setting anchor points on the model simulating the mouth frame according to the position of the human facial structure in the image;

[0026] S3, adjusting the model using a model adjustment unit according to the anchor points on the image grid layer until the anchor points coincide with each other;

[0027] S4. Perform secondary calibration using the feedback acquisition unit to capture the facial movements of the training subject, synchronize the movements of the model, and perform comparative correction.

[0028] Preferably, the specific steps of generating the simulated image of the training object and the simulated image of the training guide in step 2 are:

[0029] S1, importing the model corrected and calibrated in step 1 into the animation model framework, and the animation model framework calibrates the original model according to the model data;

[0030] S2, adding image elements to the original model through the training object simulation image generation unit to form a training object simulation image, and loading it into the animation model framework;

[0031] S3, adding image elements to the original model through the training wizard simulation image generation unit to form a training wizard simulation image, and loading it into the animation model framework;

[0032] S4. Select scene materials through the scene generation unit and load them into the animation model framework to form a simulated animation training environment.

[0033] Preferably, the specific steps of setting the phased training levels in step 3 are:

[0034] S1. extracting training content that matches the training object through the training program storage unit;

[0035] S2, generating training levels through a training process generating unit according to the training progress of the training object;

[0036] S3, the training checkpoint sets the training target, i.e., the clearance target, according to the training qualification standard determination unit;

[0037] S4. The training guide instruction output unit issues instructions to the training guide simulation image, instructing the training image to perform an action animation corresponding to the pronunciation.

[0038] Preferably, the specific steps of the accuracy matching detection in step 5 are:

[0039] S1, the voice received by the microphone is introduced into the voice band comparison unit and the volume threshold prompt unit for comparison and detection of the volume;

[0040] S2, set the band qualified rate X%, convert the speech into a band graph, overlap it with the qualified band to generate an overlap rate of X%, and it is considered qualified when X% < Y%;

[0041] S3, when the volume is too low or the voice band is unqualified, the voice feedback unit sends instructions to the training guide simulation image to remind the training subject that it is unqualified;

[0042] S4. The training wizard simulates and displays the correct pronunciation mouth shape.

[0043] The present invention provides a children's language rehabilitation training method and system based on speech and lip shape recognition, which has the following beneficial effects:

[0044] 1. The present invention can be personalized to fit the specific facial features and mouth shape of each child, and provide customized rehabilitation training to better fit the child's mouth shape to guide more accurate pronunciation of the mouth shape. At the same time, it can generate guiding animations and characters to attract children's attention, increase children's participation, make rehabilitation training more interactive and interesting, and help improve children's language learning interest and effect.

[0045] 2. The present invention can collect children's voice and lip shape data in real time, compare them with the training standards, provide instant feedback, assist children to correct pronunciation and lip shape errors in time, and dynamically adjust the training content and difficulty according to the children's progress to ensure the continuity and effectiveness of the training, thereby more effectively promoting the development of children's language ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a system diagram of the present invention;

[0047] Figure 2 It is a schematic diagram of the tracking and capturing module system of the present invention;

[0048] Figure 3 A schematic diagram of a training animation generation module system of the present invention;

[0049] Figure 4 It is a schematic diagram of the voice collection module system of the present invention;

[0050] Figure 5 It is a schematic diagram of the training planning module system of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Example:

[0053] As one aspect of this application, please see the attached Figure 1 -Attached Figure 5 The embodiment of the present invention provides a children's language rehabilitation training system based on speech and lip shape recognition, comprising:

[0054] The lip-simulating frame has a built-in model of lips and part of the face with a movable skeleton. It uses an external camera to track and imitate the lips and face of the training subject in the form of model animation.

[0055] The tracking and capturing module is composed of a facial scanning unit, a model adjustment unit, a grid anchoring unit and a feedback collection unit which are electrically connected to each other, and is used to collect facial data of the training object and feed the facial data back to the model of the simulated lip frame to fit the face of the training object and improve the fit of the training;

[0056] The training animation generation module is composed of an animation model framework, a scene generation unit, a training object simulation image generation unit and a training guide simulation image generation unit which are electrically connected to each other, and is used to cooperate with the simulation lip shape framework to generate a guiding animation and a guiding role suitable for children, so as to increase the interest of the children as training objects and prevent them from feeling bored after long-term training;

[0057] The voice collection module is composed of a voice band comparison unit, a volume threshold prompt unit, and a sound feedback unit which are electrically connected to each other, and is used to collect the sound emitted by the training object during training. An external microphone for collecting sound is connected, and the collected sound is compared with the training standard, and the sound is fed back to the training process and the model of the simulated lip shape frame;

[0058] The training planning module is composed of a training plan storage unit, a training process generation unit, a training qualification standard judgment unit, a training guidance instruction output unit and a training report generation unit which are electrically connected to each other. It is used to store training plans and training processes, and serve as judgment criteria for training, as well as to send instructions to the voice acquisition module and the training animation generation module to execute the training plan.

[0059] Based on the above-mentioned children's language rehabilitation training system based on speech and lip shape recognition, as another aspect of the present application, a children's language rehabilitation training method based on speech and lip shape recognition includes the following steps:

[0060] Step 1: The face and lips of the training object are captured by the camera, and the model mouth shape in the mouth shape frame is simulated by the tracking and capture module for correction to make it closer to the mouth shape of the training object. The specific steps for correcting the model mouth shape are as follows:

[0061] S1. First, scan the face of the training object, and cover the grid layer with the scanned facial image;

[0062] S2, using a grid anchoring unit to set anchor points on the grid overlay layer of the facial image, and also setting anchor points on the model simulating the mouth frame according to the position of the human facial structure in the image;

[0063] S3, adjusting the model using a model adjustment unit according to the anchor points on the image grid layer until the anchor points coincide with each other;

[0064] S4. Perform secondary calibration using the feedback acquisition unit to capture the facial movements of the training subject, synchronize the movements of the model, and perform comparative correction.

[0065] Step 2: Based on the mouth shape data provided in step 1, a training object simulation image and a training wizard simulation image are generated in sequence using a training animation generation module. The specific steps for generating the training object simulation image and the training wizard simulation image are as follows:

[0066] S1, importing the model corrected and calibrated in step 1 into the animation model framework, and the animation model framework calibrates the original model according to the model data;

[0067] S2, adding image elements to the original model through the training object simulation image generation unit to form a training object simulation image, and loading it into the animation model framework;

[0068] S3, adding image elements to the original model through the training wizard simulation image generation unit to form a training wizard simulation image, and loading it into the animation model framework;

[0069] S4. Select scene materials through the scene generation unit and load them into the animation model framework to form a simulated animation training environment.

[0070] Step 3: Plan the training content according to the training planning module, set up phased training levels, and use the training wizard to simulate the image of the simulation animation to guide the training object to pronounce. The specific steps for setting the phased training levels are as follows:

[0071] S1. extracting training content that matches the training object through the training program storage unit;

[0072] S2, generating training levels through a training process generating unit according to the training progress of the training object;

[0073] S3, the training checkpoint sets the training target, i.e., the clearance target, according to the training qualification standard determination unit;

[0074] S4. The training guide instruction output unit issues instructions to the training guide simulation image, instructing the training image to perform an action animation corresponding to the pronunciation.

[0075] Step 4: During the training process, the voice collection module collects the voice of the training object, and the camera collects the mouth shape of the training object, which is fed back into the simulated image of the training object;

[0076] Step 5: The voice collection module performs accuracy matching detection based on the collected voice. The specific steps of the accuracy matching detection are as follows:

[0077] S1, the voice received by the microphone is introduced into the voice band comparison unit and the volume threshold prompt unit for comparison and detection of the volume;

[0078] S2, set the band qualified rate X%, convert the speech into a band graph, overlap it with the qualified band to generate an overlap rate of X%, and it is considered qualified when X% < Y%;

[0079] S3, when the volume is too low or the voice band is unqualified, the voice feedback unit sends instructions to the training guide simulation image to remind the training subject that it is unqualified;

[0080] S4. The training wizard simulates and displays the correct pronunciation mouth shape.

[0081] Step 6: If the test is passed, enter the next stage of training.

[0082] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A children's language rehabilitation training system based on speech and lip recognition, characterized in that: include: The lip-simulating frame has a built-in model of lips and part of the face with a movable skeleton. It uses an external camera to track and imitate the lips and face of the training subject in the form of model animation. The tracking and capturing module is used to collect the facial data of the training object and feed the facial data back to the model of the simulated lip frame to fit the face of the training object and improve the fit of the training; The training animation generation module is used to generate guiding animations and guiding roles suitable for children in conjunction with the simulated lip-sync framework, so as to increase the interest of the children as training objects and avoid children feeling bored after long-term training; The voice collection module is used to collect the sound of the training object during training. An external microphone is connected to collect the sound. The collected sound is compared with the training standard and fed back to the training process and the model of the simulated lip frame. The training planning module is used to store the training plan and training process, and serves as the judgment standard for training and sends instructions to the voice acquisition module and the training animation generation module to execute the training plan.

2. A children's language rehabilitation training system based on speech and lip recognition according to claim 1, characterized in that: The tracking and capturing module comprises a facial scanning unit, a model adjustment unit, a grid anchoring unit and a feedback collection unit which are electrically connected to each other.

3. A children's language rehabilitation training system based on speech and lip recognition according to claim 1, characterized in that: The training animation generation module comprises an animation model framework, a scene generation unit, a training object simulation image generation unit and a training guide simulation image generation unit which are electrically connected to each other.

4. A children's language rehabilitation training system based on speech and lip recognition according to claim 1, characterized in that: The voice collection module includes a voice band comparison unit, a volume threshold prompt unit, and a sound feedback unit which are electrically connected to each other.

5. A children's language rehabilitation training system based on speech and lip recognition according to claim 1, characterized in that: The training planning module includes a training program storage unit, a training process generation unit, a training qualification standard determination unit, a training guidance instruction output unit and a training report generation unit which are electrically connected to each other.

6. A child language rehabilitation training method based on speech and lip shape recognition, using a child language rehabilitation training system based on speech and lip shape recognition as claimed in any one of claims 1 to 5, characterized in that: The following steps are involved: Step 1: The face and lips of the training object are captured by the camera, and the model mouth shape in the mouth shape frame is simulated and corrected by the tracking and capturing module to make it closer to the mouth shape of the training object; Step 2: Generate a training object simulation image and a training guide simulation image in sequence using a training animation generation module according to the mouth shape data provided in step 1; Step 3: Plan the training content according to the training planning module, set the phased training levels, and use the training wizard to simulate the image of the simulation animation to guide the training object to pronounce; Step 4: During the training process, the voice collection module collects the voice of the training object, and the camera collects the mouth shape of the training object, which is fed back into the simulated image of the training object; Step 5: The voice collection module performs accuracy matching detection based on the collected voice; Step 6: If the test is passed, enter the next stage of training.

7. A method for children's language rehabilitation training based on speech and lip recognition according to claim 6, characterized in that: The specific steps for correcting the model mouth shape in step 1 are: S1. First, scan the face of the training object, and cover the grid layer with the scanned facial image; S2, using a grid anchoring unit to set anchor points on the grid overlay layer of the facial image, and also setting anchor points on the model simulating the mouth frame according to the position of the human facial structure in the image; S3, adjusting the model using a model adjustment unit according to the anchor points on the image grid layer until the anchor points coincide with each other; S4. Perform secondary calibration using the feedback acquisition unit to capture the facial movements of the training subject, synchronize the movements of the model, and perform comparative correction.

8. A method for children's language rehabilitation training based on speech and lip recognition according to claim 6, characterized in that: The specific steps of generating the simulated image of the training object and the simulated image of the training guide in step 2 are: S1, importing the model corrected and calibrated in step 1 into the animation model framework, and the animation model framework calibrates the original model according to the model data; S2, adding image elements to the original model through the training object simulation image generation unit to form a training object simulation image, and loading it into the animation model framework; S3, adding image elements to the original model through the training wizard simulation image generation unit to form a training wizard simulation image, and loading it into the animation model framework; S4. Select scene materials through the scene generation unit and load them into the animation model framework to form a simulated animation training environment.

9. A method for children's language rehabilitation training based on speech and lip recognition according to claim 6, characterized in that: The specific steps for setting the phased training levels in step 3 are as follows: S1. extracting training content that matches the training object through the training program storage unit; S2, generating training levels through a training process generating unit according to the training progress of the training object; S3, the training checkpoint sets the training target, i.e., the clearance target, according to the training qualification standard determination unit; S4. The training guide instruction output unit issues instructions to the training guide simulation image, instructing the training image to perform an action animation corresponding to the pronunciation.

10. A method for children's language rehabilitation training based on speech and lip recognition according to claim 6, characterized in that: The specific steps of the accuracy matching detection in step 5 are: S1, the voice received by the microphone is introduced into the voice band comparison unit and the volume threshold prompt unit for comparison and detection of the volume; S2, set the band qualified rate X%, convert the speech into a band graph, overlap it with the qualified band to generate an overlap rate of X%, and it is considered qualified when X% < Y%; S3, when the volume is too low or the voice band is unqualified, the voice feedback unit sends instructions to the training guide simulation image to remind the training subject that it is unqualified; S4. The training wizard simulates and displays the correct pronunciation mouth shape.

Images