A hearing-impaired rehabilitation training system and method

Abstract

The application relates to the field of language rehabilitation of hearing-impaired children, and provides a hearing-impaired rehabilitation training system and method, which comprises a display module, a pronunciation module, a data acquisition module, a storage module and a control module; the display module can output images or videos according to the instruction of the control module; the pronunciation module can output different audio sounds according to the control module; the data acquisition module can acquire the sound, expression, action or physiological information of a patient and send the information to the control module; the control module judges whether the patient is interested in the current images or videos and corresponding sounds according to the information acquired by the data acquisition module, and increases the playing of the corresponding category videos or images according to the interest of the patient. Through multi-modal data acquisition and analysis, the application can automatically guide the training according to the interest and state of the patient, replaces artificial guidance, effectively grasps the golden window period of language development of infants and young children, and significantly improves the efficiency and effect of rehabilitation training.

Description

Technical Field

[0001] This application relates to the field of language rehabilitation for hearing-impaired children, and in particular to a hearing-impaired rehabilitation training system and method. Background Technology

[0002] Some children are born with hearing impairments, unable to perceive sound or with weak sound perception, making it difficult for them to learn language. They often require cochlear implants or hearing aids, especially cochlear implants. Cochlear implants do not directly repair damaged cochleas; instead, they establish a new sound pathway for the auditory nerve and brain through "sound-to-electricity conversion" and "electronic stimulation." However, the cochlear implant is not a physiological organ, especially for infants and young children. Even with a cochlear implant, language learning is challenging, requiring specialized auditory and speech training. Auditory training gradually builds the ability to perceive, distinguish, recognize, and understand sound, progressing from "hearing" to "understanding." Speech training focuses on learning to control vocalization, mastering correct pronunciation, accumulating vocabulary, learning grammar, and communicating in real-life situations. For hearing-impaired children, infancy (0-3 years) is the golden window for auditory and speech center development. Cochlear implantation and rehabilitation training during this stage typically yield the best results, potentially enabling them to "not be mute despite being deaf." If rehabilitation is delayed until childhood (after age 3), the brain's language plasticity decreases, and the difficulty of rehabilitation will increase significantly.

[0003] Chinese patent application CN220491471U, entitled "Language-Assisted Rehabilitation Training Device and System," discloses a language-assisted rehabilitation training device and system. The device includes an audio detection device and a first terminal device. The audio detection device comprises an audio detection module and a first communication module; the audio detection module is connected to the first communication module; the detection end of the audio detection module is used to contact the user's first sensing part, and the audio detection module is configured to detect the current sound vibration frequency signal of the user's first sensing part; the first terminal device includes a processor, a second communication module, and a display; the processor is connected to both the second communication module and the display; the second communication module is connected to the first communication module; the processor is configured to control the display to show text information corresponding to the received current sound vibration frequency signal, thereby enabling convenient rehabilitation training for hearing-impaired individuals who have recently regained their hearing. This device is not suitable for infants and young children.

[0004] Chinese patent application CN113593374A, entitled "A Multimodal Speech Rehabilitation Training System Integrating Oral Motor Training," discloses a multimodal speech rehabilitation training system integrating oral motor training. The system includes a human-computer interaction device comprising a data acquisition unit, a data analysis unit, a data comparison unit, a display unit, and a database. The data acquisition unit is electrically connected to the data analysis unit, the data analysis unit is electrically connected to the data comparison unit, and the data comparison unit is connected to the display unit and the database. This system directly performs vocalization training, making it unsuitable for infants and young children.

[0005] Chinese patent application No. CN107301863A, entitled "A Rehabilitation Method and Rehabilitation Training System for Speech Disorders in Deaf and Mute Children," discloses a rehabilitation method and system for speech disorders in deaf and mute children. It utilizes software analysis methods and a user-friendly human-computer interaction interface to guide and stimulate the learning interest of deaf children, solving the problem of complex operation. It is suitable for deaf children's independent learning. This technical solution is mainly aimed at older children and is difficult to apply to infants and toddlers. Summary of the Invention

[0006] This application provides a hearing-impaired rehabilitation training system and method to at least solve the problems of language rehabilitation training technology for hearing-impaired children in the prior art.

[0007] According to a first aspect of this application, a hearing-impaired rehabilitation training system is provided, including a display module, a sound module, a data acquisition module, a storage module, and a control module. The display module can output images or videos according to the instructions of the control module. The sound module can output sounds of different audio frequencies according to the control module. The data acquisition module can collect the child's voice, facial expressions, movements, or physiological information and send them to the control module. The control module determines whether the child is interested in the current image or video and the corresponding sound based on the information obtained by the data acquisition module, and adds playback of corresponding categories of videos or images according to the child's interest.

[0008] Compared with existing technologies, the hearing impairment rehabilitation training system of this application has the following beneficial effects: Early cochlear implantation is a crucial step in intervention when hearing impairment is detected in infancy. Following this, the hearing rehabilitation training system described in this application can more efficiently promote language acquisition and reduce learning delays. Infants and young children often struggle with language learning due to the lack of vocalization. Traditional training methods, relying on caregivers, require significant time investment and may yield inconsistent results due to individual differences. This hearing rehabilitation training system utilizes a highly sensitive data acquisition module to monitor the child's eye movements, facial expressions, and physiological responses in real time, promptly acquiring learning status and information. It can also analyze the child's interests and preferences using artificial intelligence algorithms to dynamically adjust training content. Furthermore, gamified and interactive learning scenarios can attract the child's active participation. This system effectively replaces repetitive guidance from caregivers, providing standardized and continuous training support. Even infants as young as a few months old can achieve initial training results through the system's progressive stimulation and feedback mechanisms, transitioning from vocalization to articulation, gradually establishing a connection between sound and meaning, and laying a solid foundation for the later development of clear language expression and social communication skills.

[0009] In one possible implementation, the data acquisition module includes at least an audio pickup module, a camera module, and a physiological data acquisition module. The audio pickup module collects the child's sound information in real time, such as cries, voice, or ambient sounds, using a high-sensitivity microphone. This audio data is compressed and stored in a designated partition of the storage module, and simultaneously transmitted to the control module. Audio processing algorithms are used for analysis to identify emotional fluctuations, pain expressions, or abnormal acoustic features. The camera module is equipped with an adjustable-focus high-definition camera that continuously captures the child's limb movements, gestures, or facial expressions. The video stream is encoded into a digital signal and stored in the storage module, while simultaneously being sent to the control module. Computer vision technology is used to analyze behavioral patterns, facial dynamics, and activity levels, thereby assisting in assessing the child's physiological comfort or psychological state. The physiological data acquisition module integrates multiple biosensors, such as a heart rate monitor, temperature probe, and pulse oximeter, to non-invasively collect key physiological indicators of the child, including heart rate, body temperature, respiratory rate, and blood oxygen saturation. Real-time data is encrypted and stored in the storage module, then instantly transmitted to the control module. This multi-source information is combined for comprehensive analysis and trend prediction, enabling early warning or health assessment. These modules work collaboratively to construct a multi-dimensional data acquisition network, providing a complete information foundation for comprehensive and accurate monitoring of the child's condition.

[0010] In one possible implementation, the physiological data includes at least one of heart rate, blood pressure, electroencephalogram (EEG), respiratory rate, or cerebral oxygenation. These key indicators are monitored and recorded in real time through an integrated sensor system. Simultaneously, a camera module is designed to capture the user's eye dynamics, accurately recording blink frequency and pupil diameter changes. Combined with data analysis algorithms, this supports continuous assessment and feedback of the physiological state, determining whether the child is focused during learning, and adjusting the learning content promptly if the child is not interested.

[0011] In one possible implementation, the storage module stores tagged image or video data designed to support relevant application scenarios, such as medical assistance, child development tracking, or behavioral analysis. The stored content covers at least the following categories: images or videos of the child's relatives for emotional connection and identification; images or videos of everyday items to help familiarize the child with their environment and routines; images or videos of food to aid in diet management and nutrition education; images or videos of toys to promote entertainment, interaction, and cognitive development; images or videos of landscapes to provide visual stimulation and relaxation; and images or videos of animals to enhance interest and nature awareness. Furthermore, the storage module may be expanded to other categories as needed, such as images or videos of learning tools, medical devices, or social scenarios, to ensure data diversity and usability, thereby optimizing system functionality.

[0012] In one implementation, the sound pickup module is positioned close to the child and equipped with directional sound pickup and noise reduction functions to effectively filter out interfering sounds from the sound module. The control module receives the sound signals collected by the sound pickup module in real time and analyzes them using a built-in speech recognition algorithm to determine whether the sound information contains a valid command. When the system confirms that the received information is a valid command, the control module will activate the corresponding output unit according to the command content, which can drive the display device to play the corresponding image or video content, thereby achieving accurate response to the child's interactive needs.

[0013] In one possible implementation, the control module first analyzes and identifies the acquired sound information. If it determines that the sound information does not belong to the preset instruction range, it further evaluates whether the sound is an imitation of the target pronunciation. When it is detected that the accuracy of the imitated sound does not meet the set standard, the pronunciation module will replay the accurate standard sound and simultaneously repeat the corresponding video or image material to enhance the learning effect.

[0014] In one embodiment, the display module can be a high-resolution display screen, a large-size projection screen, an immersive 3D screen, or virtual reality (VR) glasses to adapt to the visual needs of different scenarios; at the same time, the sound module adopts a high-quality stereo sound system that supports multi-channel surround sound effects, thereby enhancing the overall audiovisual experience.

[0015] According to a second aspect of this application, a method for using a hearing impairment rehabilitation training system is provided, which involves multiple training cycles starting from the implantation of a cochlear implant in a child: In the first training cycle (two to four months), multiple videos of family members speaking are recorded and played in a loop or randomly. Images of smiling, angry, praising, and helpless expressions are also created. If the child can make a sound while watching the video, a smiling image is given as feedback. If the child makes a sound after watching multiple videos, a praising image is given as feedback. If the child doesn't make a sound while watching a video, a helpless image is given as feedback. If the child watches multiple different videos without making any sound or facial expression, an angry image is given as feedback. This stage is the initial learning phase, primarily focusing on helping the child adapt to the training model.

[0016] In the second training cycle (three to eight months), when playing videos and pictures, only the same word sound is repeatedly uttered. When the child imitates the sound, a smiling emoticon image is given as feedback. When the child's pronunciation is relatively standard, a praising emoticon image is given as feedback. If the child sees a video without making a sound, a helpless emoticon image is given as feedback. If the child sees multiple different videos without making any sound or facial expression feedback, an angry emoticon image is given as feedback. In the early stages of this training cycle, videos or images in which the child makes sounds are played more often. Once the pronunciation is relatively standard, videos or images in which the child did not make sounds in the early stages are played. This stage mainly focuses on vocal training, guiding the patient to make sounds.

[0017] In the third training cycle (two to four months), videos related to the instructions are played. The video content can be adjusted according to the instructions given by the child. The child can give instructions based on sound or movement. Some instructions require the cooperation of relevant caregivers, while others are completed directly by the system. This stage focuses on learning instructions, allowing the child to understand simple instructions.

[0018] In the fourth training cycle (three to eight months), learning videos and word expansion videos are played. When the child voluntarily utters a word, the word is expanded into a sentence to see the child's feedback instructions. The system or relevant caregivers assist the child in completing the task. This stage focuses on expansion, turning simple words into sentences.

[0019] In the fifth training cycle (six to sixteen months), learning and interactive videos are played. When the child pronounces words or sentences, artificial intelligence engages in dialogue with the child. This stage primarily focuses on achieving interaction; after completing this training, the child's condition is essentially close to that of a normal child.

[0020] The present invention provides a method for using a hearing-impaired rehabilitation training system. Through a phased training approach, the system guides children step by step from sound perception to language comprehension and expression, gradually establishing auditory pathways. It effectively seizes the golden window of language development from 0 to 1 years old, significantly improving the efficiency and effectiveness of rehabilitation training and reducing the guidance burden on caregivers.

[0021] In one possible implementation, at least a portion of the videos played are taken from everyday life scenarios, such as family interactions, work settings, or leisure activities. These scenarios are generated through real recordings or simulations, aiming to enhance the relatability and practicality of the content, making it easier for children to empathize and engage with the playback context.

[0022] In one possible implementation, caregivers provide timely rewards based on the child's daily training performance, such as through verbal praise, small prizes, or progress charts, to enhance the child's enthusiasm and cooperation, thereby ensuring the effective implementation of the training plan and the gradual achievement of rehabilitation goals.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0024] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0025] Figure 1 A schematic diagram of a hearing impairment rehabilitation training system according to an embodiment of this application is shown.

[0026] Explanation of the labels in the diagram: 1. Display module; 2. Sound module; 3. Data acquisition module; 4. Sound pickup module; 5. Camera module; 6. Physiological data acquisition module; 7. Control module. Detailed Implementation

[0027] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] Example 1: like Figure 1 As shown, this embodiment provides a hearing impairment rehabilitation training system. This system is primarily used for early rehabilitation intervention in infants and young children after cochlear implantation. It can also be used for preschool children after cochlear implantation, although the rehabilitation effect will be somewhat less pronounced. Adults with acquired hearing loss can also use it, but the usage method needs appropriate adjustments. The system includes a display module 1, a sound generation module 2, a data acquisition module 3, a storage module, and a control module 7. The control module 7, as the core processing unit of the system, establishes communication connections with the display module 1, sound generation module 2, data acquisition module 3, and storage module to realize data transmission and command issuance.

[0029] Specifically, display module 1 is configured to output images or videos according to instructions from control module 7. In practical applications, display module 1 can be a screen placed in front of the child to play high-definition animations or educational videos to attract the child's visual attention. Sound module 2 is configured to output different audio frequencies according to the output of control module 7. Sound module 2 typically consists of high-fidelity stereo speakers, capable of clearly playing speech, music, or natural environmental sounds to provide auditory stimulation in conjunction with the images displayed on display module 1. The storage module is used to store various materials required for system operation, including but not limited to image files, video files, audio files, and system operation logs. It can use local hard drive storage or a combination of cloud storage and local caching to ensure real-time data retrieval and security.

[0030] One of the core innovations of this embodiment lies in the collaborative working logic between the data acquisition module 3 and the control module 7. The data acquisition module 3 can collect the child's voice, facial expressions, movements, or physiological information and send it to the control module 7. Since infants and young children do not yet possess mature language expression abilities, their psychological state and interests are often expressed through non-verbal signals. Therefore, the data acquisition module 3 needs to have multimodal perception capabilities. For example, a microphone for collecting sound information can capture the child's babbling or crying / laughing sounds; a camera for collecting facial expressions and movements can capture the child's gaze direction, facial smiles, or limb movements; sensors for collecting physiological information (such as a wearable wristband) can monitor the child's heart rate variability or skin conductance. When the data acquisition module 3 detects crying, it needs to measure the duration of the sound. If it exceeds one minute, it needs to immediately call for a caregiver. If a caregiver is present, The control module 7 determines whether the child is interested in the current image or video and its corresponding sound based on the information obtained from the data acquisition module 3, and increases the playback of videos or images of the corresponding category according to the child's interest. This process is not a simple conditioned reflex, but a comprehensive judgment logic based on multimodal data fusion. Specifically, the control module 7 has a preset interest determination model, which quantifies the collected multi-dimensional information. For example, when the combination of "pupil dilation (physiological information) + prolonged gaze fixed on the screen (motor information) + slightly increased heart rate (physiological information)" is detected, the control module 7 determines that the child is "highly interested" in the current content. Conversely, if the combination of "gaze wandering (motor information) + yawning (facial expression information) + stable heart rate (physiological information)" is detected, the child is determined to be "uninterested" or have "decreased attention" in the current content.

[0031] If the judgment result indicates interest, control module 7 will execute a positive feedback strategy, namely, increasing the playback frequency or extending the playback time of the corresponding category of videos or images. When playing images, the words related to the image content should be announced, such as announcing "elephant" when playing a picture of an elephant. Interest is the first teacher; children with disabilities are more focused on things they are interested in and have stronger learning abilities. For example, if the currently playing video is about "animals" and the child shows strong interest, the system will automatically retrieve more "animal" category materials from the storage module for playback, or repeat the current video to strengthen auditory memory. This adaptive adjustment mechanism based on interest feedback utilizes the physiological and psychological characteristics of infants and young children, whose attention is easily distracted but easily driven by interest. It can effectively solve the problem of fixed training content and difficulty in attracting infants and young children's attention for a long time in existing technologies, thereby achieving more efficient auditory perception and language enlightenment training during the golden window period. It should be understood that the specific logic of interest judgment described above is only an illustrative example. Those skilled in the art can train and optimize the judgment model through machine learning algorithms according to actual training needs to adapt to the individual differences of different children.

[0032] Example 2: This embodiment, based on embodiment 1, provides a detailed description of the specific composition of the data acquisition module 3 and the construction of the material library in the storage module. Specifically, the data acquisition module 3 includes at least a sound pickup module 4, a camera module 5, and a physiological data acquisition module 6. The sound pickup module 4 collects sound information, stores it in the storage module, and sends it to the control module 7 for analysis. The camera module 5 collects the child's movement or facial expression information and sends it to the control module 7 for analysis. The physiological data acquisition module 6 collects physiological information and sends it to the control module 7 for analysis.

[0033] The specific placement of the sound pickup module 4 is crucial to the quality of sound acquisition. Since infants' vocalizations are typically weak and discontinuous, the sound pickup module 4 is preferably a high-sensitivity microphone array, which can be placed on the child's headband, bib, or a nearby toy stand to ensure clear capture of subtle sound signals such as babbling and breathing, while effectively reducing interference from ambient background noise. The camera module 5 is usually placed around the display module 1 or directly in front of the child's care area. Its acquisition targets include not only the child's facial expressions (such as smiling, frowning, and yawning) and body movements (such as waving, kicking, and turning the head), but also subtle eye features. Specifically, the camera module 5 can capture blink frequency and changes in pupil diameter. Blink frequency is an important indicator of a child's fatigue or concentration, while changes in pupil diameter are closely related to an individual's cognitive processing load and emotional arousal. When a child shows interest in a certain type of video content, their pupils usually dilate involuntarily. This physiological characteristic provides the control module 7 with an objective and difficult-to-fake basis for judging "interest".

[0034] The physiological data acquisition module 6 further expands the system's perception dimensions. This module collects at least one of the following physiological data: heart rate, blood pressure, electroencephalogram (EEG), respiratory rate, or cerebral oxygen saturation. In practical applications, the physiological data acquisition module 6 can employ wearable sensors (such as smart bracelets or ankle bracelets) or non-contact sensors (such as radar wave monitoring devices). For example, heart rate variability (HRV) can reflect a child's emotional state of tension or relaxation; EEG data can directly reflect the activity level of the cerebral cortex; when playing content that interests the child, the EEG usually exhibits active waveforms in specific frequency bands. By introducing the physiological data acquisition module 6, the system can acquire the child's deeper physiological responses, thereby compensating for potential misjudgments that may arise from relying solely on visual observation. For example, a child may appear motionless and seemingly daydreaming, but their heart rate and EEG indicate a state of high concentration. If the system judges disinterest solely based on movement and switches content accordingly, it will interrupt the child's cognitive process. Therefore, the fusion analysis of multimodal data significantly improves the accuracy and robustness of interest judgment. Of course, the more data collected, the higher the cost. Data can be selected for testing based on market pricing. Alternatively, existing models can be used to directly test children with the condition. For example, the "iCatcher+" model determines gaze direction, the "PyAFAR" model detects facial expressions, and the "ST-MERN" model detects both facial expressions and body posture. Even if a child cannot speak, a large deep learning model can perceive their true thoughts based on body language, potentially even understanding them better than their parents.

[0035] In terms of material library construction, the storage module stores tagged images or videos, including at least one or more of the following: images or videos of the child's relatives, daily necessities, food, toys, landscapes, and animals. This classification method is designed based on the cognitive development patterns of infants and toddlers. Among them, images or videos of the child's relatives (such as videos of the mother calling the child's name) are most likely to evoke a sense of security and attention in infants and toddlers, making them suitable as core materials in the early stages of training; materials closely related to the child's life, such as daily necessities, food, and toys, help the child establish a connection between sound and objects, promoting language generalization; while landscape and animal materials are used to expand the child's cognitive horizons. The term "tagged" means that when storing materials, not only the image or video data itself is stored, but also its category tags (such as "animals - cat"), emotional attributes (such as "cheerful" and "soothing"), and recommendation weights, etc., are also stored. When the control module 7 calls up materials, it will perform precise matching based on these tags. For example, when the control module 7 determines that the child is interested in "food" videos based on multimodal data, it will prioritize retrieving and playing materials marked as "food" from the storage module, thereby achieving accurate personalized training recommendations.

[0036] To facilitate the evaluation of learning and training results, the sound pickup module 4 can evaluate whether the child's pronunciation is standard, and the camera module 5 uses a trackable pan-tilt camera to track and film the child's head, which can determine whether the child's mouth shape is standard and comprehensively evaluate the child's learning situation.

[0037] It should be understood that the above examples of physiological data and material classification are merely illustrative. Those skilled in the art can add other types of physiological indicators (such as skin conductance response) or material classifications (such as vehicle categories) according to actual training needs. As long as the function of multimodal data collection and classification recommendation can be achieved, they should be included within the scope of protection of this invention.

[0038] Example 3: This embodiment further optimizes the sound signal processing logic and the system's interactive feedback mechanism based on Embodiment 1 or Embodiment 2. Specifically, the sound pickup module 4 is positioned close to the child and filters the sound from the pronunciation module 2. The control module 7 analyzes whether the obtained sound information is an instruction. If it is confirmed to be an instruction, it controls the corresponding image or video output according to the instruction. The instructions include marked words and control words. For example, "Dad" and "Mom" are marked words. When the child makes the sound of "Dad," a video related to "Dad" appears. Control words include "want," "don't want," "good," and "bad."

[0039] Regarding the placement of the pickup module 4, considering the weak vocal intensity of infants and the fact that the training environment is usually accompanied by teaching audio played by the pronunciation module 2, using conventional far-field pickup devices would easily cause the child's faint babbling sounds to be drowned out by environmental noise or teaching audio. Therefore, in this embodiment, the pickup module 4 preferably adopts a near-field pickup method, specifically placed on the headband, bib, or specially designed collar clip worn by the child, with a distance of 5 cm to 20 cm from the child's mouth and nose. This close-range placement can significantly improve the pickup intensity of the child's sound signal from a physical perspective, increasing the signal-to-noise ratio. Simultaneously, to address the interference of the sound played by the pronunciation module 2 on the child's speech commands, the pickup module 4 needs to have the function of filtering the sound from the pronunciation module 2. This functionality can be achieved through a combination of hardware and software: at the hardware level, the pickup module 4 can use a directional microphone to sensitively pick up sounds only from the direction of the child's mouth and nose, physically suppressing sounds from the direction of the display module 1 or the sound module 2; at the software level, the control module 7 integrates an echo cancellation algorithm or adaptive filter. This algorithm uses the audio data currently played by the sound module 2 stored in the storage module as a reference signal to inversely cancel the playback audio components in the mixed signal collected by the pickup module 4, thereby extracting the pure sound signal of the child. This sound source separation technology ensures that the system can accurately identify the child's intentions and avoids system logic confusion caused by misidentifying the playback sound of the sound module 2.

[0040] After successfully extracting the child's voice signal, control module 7 executes a rigorous two-level judgment logic. First, control module 7 analyzes whether the obtained voice information is an instruction. Specifically, the storage module pre-stores an instruction vocabulary, such as keywords like "put down," "next," and "look." Control module 7 performs speech recognition (ASR) processing on the extracted voice signal, converting it into text or a phoneme sequence, and matches it with the instruction vocabulary. If the match is successful and the instruction information is confirmed, the system executes the corresponding operation. For example, when the system recognizes the child uttering "next," control module 7 immediately switches to the next video segment, thereby giving the child control over the training content and enhancing their sense of participation and accomplishment.

[0041] If the control module 7 determines that the received sound information is not a command, it then determines whether the sound is an accurate imitation. This logic is mainly applied in the language imitation training stage. Specifically, when the pronunciation module 2 plays a standard pronunciation (such as "mama"), the system enters a listening state. The control module 7 compares the sound subsequently produced by the child with the acoustic characteristics (such as pitch, duration, and formant frequency) of the standard pronunciation. If the similarity exceeds a preset threshold (e.g., 80%), it is determined to be an accurate imitation, and the system can provide positive feedback (such as a smiley face or reward sound effect); if the imitation is inaccurate, the pronunciation module 2 replays the accurate sound and repeats the video or image. This error correction mechanism simulates the "demonstration-imitation-correction" cycle in human teaching. For example, when the child tries to imitate the pronunciation of "apple" but pronounces it unclearly, the system will not skip it directly, but will automatically trigger the replay mechanism, displaying the image of an apple again and playing the standard pronunciation, guiding the child to make a second attempt. This closed-loop logic not only ensures the effectiveness of training but also solves the problem in existing technologies where parents or caregivers lack the professional knowledge to accurately judge pronunciation accuracy through real-time intelligent error correction, thus achieving standardization and intelligence in rehabilitation training. It should be understood that the above-mentioned similarity threshold setting is merely illustrative; those skilled in the art can dynamically adjust the threshold according to the child's age or training difficulty to adapt to the needs of different training stages. If a second attempt still fails to pronounce the pronunciation, it can be marked as higher difficulty and will not recur in the short term to avoid affecting the child's learning motivation.

[0042] Example 4: This embodiment further expands the description of the specific form of the hardware device in the hearing impairment rehabilitation training system based on the above embodiments. Specifically, the display module 1 is a display screen, projection screen, 3D screen, or VR glasses, and the sound module 2 is a stereo speaker.

[0043] The specific implementation of display module 1 should be adapted to the child's age, training goals, and training environment. As one of the most common implementation methods, display module 1 can be a display screen, such as a liquid crystal display (LCD) or an organic light-emitting diode display (OLED). Display screens offer advantages such as clear display, moderate cost, and ease of deployment in a home environment, making them suitable for most daily training scenarios. Especially for initial sound perception and basic vocabulary learning, display screens can provide stable and intuitive visual stimulation.

[0044] As an alternative implementation, display module 1 can also be a projection screen. Projection screens are typically used in conjunction with projectors to project large images. This configuration is particularly suitable for group training scenarios in rehabilitation institutions, or for younger children whose vision is not yet fully developed and who are more sensitive to large, dynamic images. Projecting a large image enhances visual impact, improves children's attention in a group setting, and also facilitates guidance from caregivers or teachers.

[0045] In addition, display module 1 can also be a 3D screen or VR glasses (virtual reality glasses). These two forms are primarily used to provide an immersive training experience. With a 3D screen, naked-eye 3D technology or 3D glasses can make training materials appear three-dimensional, enhancing the child's spatial awareness. VR glasses, on the other hand, can completely isolate external environmental interference, providing the child with an immersive audiovisual training environment. This form is particularly suitable for advanced training stages requiring high concentration, such as training to distinguish specific sounds in noisy environments or simulating real-life scenarios for interactive dialogue training. It should be understood that although VR glasses can provide excellent immersion, due to the special characteristics of infants' visual development, the wearing time must be strictly controlled, generally not exceeding 10 minutes at a time, no more than twice a day, and under the supervision of a caregiver.

[0046] The sound module 2 is specifically implemented as a stereo system. The stereo system can play sounds with spatial orientation, simulating a realistic sound field environment. For example, when training a child's ability to distinguish the direction of a sound source, the stereo system can simulate the effect of sound coming from the left or right by using the volume and time difference between the left and right channels. Combined with the image displayed on the display module 1 (such as a small dog running into the screen from the left), this guides the child to establish a correspondence between "sound-location-image." This multi-sensory synergistic stimulation is of great significance for rebuilding the auditory pathway in hearing-impaired children. It should be understood that the above-described hardware configurations are merely illustrative examples. Those skilled in the art can choose other display or sound devices with equivalent functions according to actual needs, all of which should be included within the scope of protection of this invention.

[0047] Example 5: This embodiment provides a method for using a hearing impairment rehabilitation training system. This method is based on the system described in the previous embodiment and involves multiple training cycles starting from the implantation of a cochlear implant in the child. The design of this method follows the physiological laws of auditory and speech development in infants and young children, dividing the rehabilitation process into five progressive training cycles. The aim is to achieve a leap from "sound perception" to "language comprehension and communication" through systematic intervention. This training method requires the use of numerous electronic devices; it is recommended that daily training time not exceed two hours.

[0048] Specifically, the first training cycle lasts two to four months, primarily corresponding to the developmental characteristics of infants and toddlers during their auditory awareness period. During this cycle, multiple videos of family members speaking are recorded and played in a loop or randomly. The voices of family members (such as mother and father) are the most familiar and reassuring auditory stimuli for the child, effectively stimulating their initial attention to sound. Simultaneously, the system creates feedback materials such as smiling, angry, praising, and helpless facial expressions. During training, when the child sees a video and makes a sound, a smiling emoticon is given as feedback. This positive feedback strengthens the child's psychological connection between "vocalization" and "pleasure." When the child watches multiple videos and makes sounds consecutively, a praising emoticon is given as feedback, further reinforcing their initiative to vocalize. If the child sees a video but makes no sound, a helpless emoticon is given, indicating that the child's current state has not been recognized as positive interaction by the system. If the child watches multiple different videos consecutively without making any sound or showing any facial expression or movement, an angry emoticon is given. This slightly negative feedback aims to induce emotional fluctuations in the child, thereby stimulating their attention. It should be understood that the two- to four-month duration is based on the average time required for infants to establish initial auditory habits. In practical application, it can be adaptively adjusted according to the child's implantation age and individual differences.

[0049] The second training cycle lasts three to eight months, corresponding to the infant's imitation and pronunciation period. During this cycle, when playing videos and pictures, only the sound of the same word is repeatedly pronounced to strengthen auditory memory through high-frequency repetition. Some videos and pictures include specific mouth shapes for pronunciation, making it easier for the child to imitate. The videos and pictures can be recorded during interactions between family members and the child, or other early childhood education videos. When the child imitates a sound, a smiling emoji image is given as feedback to encourage the imitation behavior; when the child's pronunciation is relatively standard, a praising emoji image is given as feedback to reinforce correct pronunciation; if the child sees a video without making a sound, a helpless emoji image is given as feedback; if the child sees multiple different videos without making any sound or facial expression feedback, an angry emoji image is given as feedback. In particular, in the early stages of this training cycle, videos or images in which the child makes sounds are played more frequently; once the pronunciation is more standard, videos or images in which the child did not make sounds in the early stages are then played. This "from easy to difficult" material delivery strategy aligns with the "zone of proximal development" theory in infants and toddlers learning language, avoiding frustration caused by excessive difficulty and maintaining the child's interest in training.

[0050] The third training cycle lasts two to four months, corresponding to the infant's instruction comprehension period. During this cycle, instruction-related videos are played, with content adjusted based on the child's commands. The child can issue commands based on sound or movement; for example, if the child says "I want to eat" or points, the system recognizes this and adjusts the video content to display food. Caregivers can then further confirm whether the child is hungry and whether feeding is necessary. Some commands require caregiver intervention; for example, if the child says "hug," the system prompts the caregiver to perform a hug. Other commands are executed directly by the system; for example, if the child says "change," the system automatically switches the video. By introducing caregiver involvement, a "human-machine collaborative" training environment is created, enhancing the realism and social aspects of the training.

[0051] The fourth training cycle lasts three to eight months, corresponding to the vocabulary expansion period for infants and toddlers. During this cycle, learning videos and vocabulary expansion videos are played. Learning videos mainly consist of early childhood cartoons or other short educational videos, with a duration of no more than half an hour. When the child utters a word, the system expands that word into a sentence to see the child's feedback. For example, when the child utters the sound "apple," the system automatically expands it into the sentence "I want to eat an apple," guiding the child to respond. The system or relevant caregivers assist the child in completing this interaction. This stage aims to help the child transition from single-word expression to phrase or sentence expression, improving the logical structure of their language organization.

[0052] The fifth training cycle lasts from six to sixteen months, corresponding to the language communication period of infants and toddlers. During this cycle, learning videos and interactive videos are played. When the child utters words or sentences, artificial intelligence engages in dialogue with the child. Specifically, the system's built-in natural language processing model can recognize the child's vocal intentions and generate context-appropriate response speech, which is played through pronunciation module 2, thus simulating real-life dialogue scenarios. This stage aims to consolidate the child's acquired language abilities through high-frequency interaction and train their coping abilities in complex contexts. The duration of the five cycles mentioned above is not fixed but rather a reference range set according to the general developmental patterns of the child's language center. In actual training, it can be dynamically adjusted based on the child's rehabilitation assessment results, reflecting the scientific rigor and flexibility of the invention.

[0053] Example 6: This embodiment, based on embodiment 5, further optimizes the source of training materials and the reward mechanism during the training process. Specifically, in any of the above training cycles, at least a portion of the videos played are taken from everyday life scenarios.

[0054] For hearing-impaired children, the ultimate goal of rehabilitation training is to generalize and apply language skills in real life. In existing technologies, many training materials are cartoons or standard instructional videos. While these can attract the child's attention, they often lead to "separate learning," where the child only recognizes objects on the screen but not real-life objects, or only speaks in specific training scenarios and remains silent in daily life. To address this issue, the storage module in this embodiment specifically stores video materials taken from everyday life scenarios. Specifically, these scenarios include at least high-frequency segments such as eating, bathing, dressing, playing, and going for walks. These video materials can be pre-recorded and uploaded to the storage module by caregivers, or captured in real-time by a home camera equipped with the system. For example, in the word imitation stage of the second training cycle, when the target vocabulary is "bathing," the system prioritizes retrieving and playing real video clips of the child bathing themselves. Because the video contains images of the child's familiar bathroom environment, bath toys, and their own movements, this strong sense of immersion can effectively activate the child's contextual memory, helping them quickly establish neural connections between "sound-scene-action," thereby greatly improving the efficiency of language generalization and shortening the conversion cycle from "learning" to "using."

[0055] Furthermore, caregivers provide timely rewards based on the child's daily training performance. This feature aims to construct a collaborative rehabilitation model of "systematic training + artificial reinforcement." Although the system of this invention can provide immediate feedback through facial expressions (such as smiles and praise), for infants and young children, physical contact and emotional interaction from close caregivers (such as parents) have an irreplaceable motivating effect. In specific implementation, the control module 7 generates a training report for the day after training and pushes reward suggestions to caregivers through a terminal (such as a mobile APP). For example, if the system detects that the child actively issued multiple commands in the third training cycle, the system will prompt "Baby did a great job today, we suggest giving a hug as a reward." After the caregiver gives the child a hug, kiss, or sticker reward in real life, they can confirm the reward has been given in the system. The system records this reward behavior and uses it as reference data for subsequent analysis of the child's behavioral motivation. This mechanism organically combines cold machine training with warm interpersonal interaction, using parent-child attachment to strengthen the child's positive behavior, which not only improves the rehabilitation effect but also promotes the healthy development of the parent-child relationship, and has important clinical application value. It should be understood that the reward mechanism is not limited to the end of daily training. It can also be set to trigger prompts immediately at specific training nodes (such as after completing a high-difficulty imitation task) to achieve more precise behavioral reinforcement.

[0056] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0057] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A hearing impairment rehabilitation training system, characterized in that, It includes a display module, a sound module, a data acquisition module, a storage module, and a control module. The display module can output images or videos according to the instructions of the control module. The sound module can output different audio sounds according to the control module. The data acquisition module can collect the child's voice, facial expressions, movements, or physiological information and send them to the control module. The control module determines whether the child is interested in the current image or video and the corresponding sound based on the information obtained by the data acquisition module, and adds the playback of corresponding categories of videos or images according to the child's interests.

2. The hearing impairment rehabilitation training system according to claim 1, characterized in that, The data acquisition module includes at least a sound pickup module, a camera module, and a physiological data acquisition module. The sound pickup module can collect sound information, store it in the storage module, and send it to the control module for analysis. The camera module can collect the child's movement or facial expression information and send it to the control module for analysis. The physiological data acquisition module can collect physiological information and send it to the control module for analysis.

3. The hearing impairment rehabilitation training system according to claim 2, characterized in that, Physiological data include at least one of heart rate, blood pressure, brain waves, respiratory rate, or cerebral blood oxygenation. The camera module can collect the number of blinks and changes in pupil diameter.

4. The hearing impairment rehabilitation training system according to claim 3, characterized in that, The storage module stores tagged images or videos, including at least one or more of the following: images or videos of the child's relatives, images or videos of daily necessities, images or videos of food, images or videos of toys, images or videos of landscapes, and images or videos of animals.

5. The hearing impairment rehabilitation training system according to claim 4, characterized in that, The sound pickup module is positioned close to the child and filters the sound from the speech module. The control module analyzes the obtained sound information to determine if it is an instruction. If it is an instruction, the corresponding image or video output is controlled according to the instruction.

6. The hearing impairment rehabilitation training system according to claim 5, characterized in that, If the control module determines that the obtained sound information is not a command, it will determine whether the sound is an accurate imitation. If the imitation is inaccurate, the sound module will replay the accurate sound and repeat the video or image.

7. The hearing impairment rehabilitation training system according to any one of claims 1 to 6, characterized in that, The display module can be a display screen, projection screen, 3D screen, or VR glasses, and the sound module can be a stereo speaker.

8. A method of using a hearing impairment rehabilitation training system, comprising the hearing impairment rehabilitation training system of claim 7, characterized in that, From the moment the child receives a cochlear implant, multiple training cycles are conducted: In the first training cycle, multiple videos of family members speaking were recorded and played in a loop or randomly. Images of smiling, angry, praising, and helpless expressions were also created. If the child could make a sound while watching the video, a smiling image was given as feedback. If the child made a sound while watching multiple videos, a praising image was given as feedback. If the child did not make a sound while watching the video, a helpless image was given as feedback. If the child watched multiple different videos without making any sound or making any facial expressions, an angry image was given as feedback. In the second training cycle, when playing videos and pictures, only the same word sound is repeatedly uttered. When the child imitates the sound, a smiling emoticon image is given as feedback. When the child pronounces the sound relatively clearly, a praising emoticon image is given as feedback. If the child sees a video without making a sound, a helpless emoticon image is given as feedback. If the child sees multiple different videos without making any sound or facial expression feedback, an angry emoticon image is given as feedback. In the early stages of this training cycle, more videos or images in which the child makes sounds are played. Once the child's pronunciation is relatively clear, videos or images in which the child did not make sounds in the early stages are played. In the third training cycle, videos related to the instructions are played. The video content can be adjusted according to the instructions given by the child. The child can give instructions based on sound or movement. Some instructions require the cooperation of relevant caregivers to complete, while some instructions are completed directly by the system. In the fourth training cycle, learning videos and word expansion videos are played. When the child voluntarily utters a word, the word is expanded into a sentence to see the child's feedback instructions. The system or relevant caregivers assist the child in completing the task. In the fifth training cycle, learning videos and interactive videos are played. When the child makes the sounds of words or sentences, artificial intelligence is used to have a dialogue with the child.

9. The method of using the hearing impairment rehabilitation training system according to claim 8, characterized in that, At least some of the videos played were taken from everyday life scenes.

10. The method of using the hearing impairment rehabilitation training system according to claim 8, characterized in that, The caregivers will reward the child promptly based on the child's daily training performance.

Images