Non-verbal man-machine interaction system and method for developmental reading disorder

Abstract

The invention discloses a non-verbal man-machine interaction system and a non-verbal man-machine interaction method for developmental reading disorder. The non-verbal man-machine interaction system comprises an evaluation unit, a primary task unit, an advanced task unit, a parameter configuration unit, an interaction unit and a feedback unit. Wherein the evaluation unit is used for collecting personal information of a user and an evaluation result of reading ability. And the primary task unit and the advanced task unit are connected with the evaluation unit so as to push primary or advanced audio-visual matching tasks with different difficulties to the user based on the evaluation result of the user. The parameter configuration unit is connected with the primary and advanced task units and has an audiovisual matching element library for configuring primary and advanced audiovisual matching tasks. And the interaction unit is connected with the primary task unit and the advanced task unit so as to perform man-machine interaction with a user based on the primary or advanced audio-visual matching task. And the feedback unit is connected with the interaction unit so as to perform result feedback on the user based on the man-machine interaction result.

Description

Technical Field

[0001] The invention relates to a non-verbal human-computer interaction system for developmental dyslexia, and also relates to a non-verbal human-computer interaction method realized by using the non-verbal human-computer interaction system, belonging to the technical field of human-computer interaction. Background Art

[0002] Developmental dyslexia is a common learning disorder characterized by a significant lag in reading ability among school-age children. According to the Diagnostic and Statistical Manual of Mental Disorders (DSM-5), this disorder involves difficulty reading words accurately or fluently, as well as poor decoding and spelling skills. In China, with the increase in the number of patients and the growing demand for diagnosis and treatment, developmental dyslexia has become an important public health issue. Therefore, the development of effective interventions is crucial to improving patients' reading ability and quality of life.

[0003] Research has shown that children with developmental dyslexia have deficits in audiovisual integration, which directly affects their reading fluency. Audiovisual matching ability, the ability to effectively integrate visual and auditory information, is associated with speech skill deficits, which may independently affect the reading fluency of Chinese children. Therefore, improving audiovisual integration ability is the key to improving the reading ability of children with developmental dyslexia.

[0004] In a Chinese patent application with publication number CN113499525A, a training system for children with developmental dyslexia is disclosed. The training system is designed to improve children's abilities in multiple cognitive areas. The system includes a central executive system module, which trains the refreshing, conversion and inhibition abilities in working memory through a series of tasks such as n back and digital active memory. The visual-spatial working memory module improves the visual and spatial information processing abilities through visual matrix tasks, etc. The buffer module trains the processing abilities of spatial verbal information through spatial verbal integration and separation tasks. The verbal working memory module enhances verbal working memory abilities through Chinese character memory and rhyme judgment tasks.

[0005] However, although this training system targets high-level cognitive abilities, it does not involve underlying perceptual training, which is insufficient for comprehensively improving the abilities of children with dyslexia. In addition, this training system may require children to have a certain foundation of language knowledge, which may be difficult for children with weaker language abilities, limiting its general applicability. Therefore, future interventions need to consider more comprehensive strategies, including underlying perceptual training, to improve applicability to children with different language ability levels. Summary of the invention

[0006] The primary technical problem to be solved by the present invention is to provide a non-verbal human-computer interaction system for developmental dyslexia.

[0007] Another technical problem to be solved by the present invention is to provide a non-verbal human-computer interaction method for developmental dyslexia.

[0008] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0009] According to a first aspect of an embodiment of the present invention, a non-verbal human-computer interaction system for developmental dyslexia is provided, comprising:

[0010] The evaluation unit includes an information input module and a cognitive evaluation module; wherein the information input module is used to collect the user's personal information, and the cognitive evaluation module is used to evaluate the user's reading ability and obtain the evaluation results;

[0011] A primary task unit is connected to the evaluation unit to push primary audio-visual matching tasks of different difficulty levels to the user based on the evaluation result; wherein the primary audio-visual matching task is configured to select a matching primary visual sequence from two primary visual sequences according to a primary auditory sequence; the primary visual sequence is composed of a plurality of visual elements arranged in a preset manner to form primary visual sequences of different difficulty levels; the primary auditory sequence is matched with one of the primary visual sequences;

[0012] A high-level task unit is connected to the evaluation unit to push high-level audio-visual matching tasks of different difficulties to the user based on the evaluation result; wherein the high-level audio-visual matching task is configured to identify a visual element that matches the last auditory element of a high-level auditory sequence from a high-level visual sequence according to a high-level auditory sequence; and the number of auditory elements in the high-level auditory sequence is greater than the number of visual elements in the high-level visual sequence;

[0013] a parameter configuration unit connected to the primary task unit and the advanced task unit, and having an audiovisual matching element library, so as to configure the primary audiovisual matching task and the advanced audiovisual matching task based on the audiovisual matching element library; wherein the audiovisual matching element library includes a plurality of visual elements, a plurality of auditory elements, and a plurality of element arrangement modes, and the plurality of visual elements correspond one to one with the plurality of auditory elements;

[0014] An interaction unit connected to the primary task unit and the advanced task unit to perform human-computer interaction with a user based on the primary audio-visual matching task or the advanced audio-visual matching task;

[0015] The feedback unit is connected to the interaction unit to obtain the human-computer interaction result of the user, show the accuracy rate to the user, and explain the purpose and significance of this human-computer interaction.

[0016] Preferably, the primary audio-visual matching task is configured in the following manner:

[0017] Randomly select two groups of visual elements of the same number from the audio-visual matching element library;

[0018] Determine the task difficulty according to the evaluation result, and retrieve the corresponding element arrangement method based on the task difficulty;

[0019] Based on the retrieved arrangement of elements, two groups of visual elements of equal number are arranged respectively, so as to form two primary visual sequences of equal length but inconsistent for visual presentation;

[0020] The same number of auditory elements are retrieved from the audio-visual matching element library, and are arranged in accordance with the retrieved elements to form a primary auditory sequence, so that the primary auditory sequence matches one of the primary visual sequences; and, after the two primary visual sequences are presented, the primary auditory sequence is played.

[0021] Preferably, the advanced audio-visual matching task is configured in the following manner:

[0022] Randomly retrieve a first number of visual elements from the audio-visual matching element library;

[0023] Determine the task difficulty according to the evaluation result, and retrieve the corresponding element arrangement method based on the task difficulty;

[0024] Arranging the first number of visual elements based on the retrieved element arrangement mode to form a high-level visual sequence for visual presentation;

[0025] Randomly retrieve a second number of auditory elements from the audiovisual matching element library, where the second number is greater than the first number;

[0026] The second number of auditory elements are arranged based on the retrieved element arrangement to form a high-level auditory sequence, and a part of the high-level auditory sequence is matched with the high-level visual sequence; and after the high-level visual sequence is presented, the high-level auditory sequence is played.

[0027] Preferably, the visual element is a preset graphic, and the auditory element is a stimulating sound;

[0028] Among them, the higher the height of the preset graphic, the higher the pitch of the stimulation sound; the longer the length of the preset graphic, the longer the duration of the stimulation sound; the wider the width of the preset graphic, the stronger the intensity of the stimulation sound.

[0029] Preferably, the plurality of element arrangements at least include:

[0030] Simple Gestalt, which consists of a monotonous ascending or descending Gestalt structure;

[0031] Gestalt subgroups, including multiple subgroups, and each subgroup has a common Gestalt structure;

[0032] Subgroups separated by pauses, including multiple subgroups, with different subgroups marked by sound pauses, and avoiding stress at the beginning of the subgroup;

[0033] The subgroups have the same block but different order, including multiple subgroups, each subgroup contains the same elements, and the order of elements between subgroups is different.

[0034] Preferably, the parameter configuration unit further includes a user preference module for allowing the user to adjust the duration and playback interval of the auditory element;

[0035] The standard duration of an auditory element is t 1 , the user can adjust the standard duration to T within the first time range 1 ; The time interval between the presentation of two auditory elements is t 2 , the user can adjust the presentation time interval to T within the second time range 2 .

[0036] Preferably, the non-verbal human-computer interaction system further comprises:

[0037] A difficulty adjustment unit, connected to the interaction unit, to receive the human-computer interaction result of the user and accumulate the correct number and the wrong number of the user;

[0038] Among them, when the cumulative number of correct answers reaches 3, the task difficulty increases by one level, and the cumulative number of correct answers is reset and recalculated; when the cumulative number of errors reaches 2, the task difficulty decreases by one level, and the cumulative number of errors is reset and recalculated.

[0039] According to a second aspect of an embodiment of the present invention, a non-verbal human-computer interaction method for developmental dyslexia is provided, comprising the following steps:

[0040] Obtain the user's personal information and reading ability assessment results;

[0041] Based on the evaluation result of the user, configuring a primary audio-visual matching task and an advanced audio-visual matching task for the user based on a parameter configuration unit;

[0042] Pushing the primary audio-visual matching task to the user for human-computer interaction, so as to obtain a primary human-computer interaction result of the user completing the primary audio-visual matching task;

[0043] Determine whether the primary human-computer interaction result meets the standard; if so, start the human-computer interaction of the advanced audio-visual matching task; if not, reconfigure the primary audio-visual matching task, and perform human-computer interaction with the user again based on the reconfigured primary audio-visual matching task until the primary human-computer interaction result meets the standard;

[0044] Pushing the advanced audio-visual matching task to the user for human-computer interaction, so as to obtain an advanced human-computer interaction result of the user completing the advanced audio-visual matching task;

[0045] The high-level audio-visual matching task is updated based on the high-level human-computer interaction result to perform the next round of human-computer interaction until the human-computer interaction process is completed.

[0046] Preferably, during the process of the user performing human-computer interaction, the correct number and the wrong number of the user are accumulated based on the human-computer interaction result of the user;

[0047] When the cumulative number of correct answers reaches 3, the difficulty level increases by one level, and the cumulative number of correct answers is reset and recalculated;

[0048] When the cumulative number of errors reaches 2, the difficulty is reduced by one level, and the cumulative number of errors is reset and recalculated.

[0049] Preferably, if the user is an initial user, the user's audio-visual ability is calculated based on the user's personal information and the evaluation result, so as to determine the user's initial task level based on the audio-visual ability;

[0050] If the user is a historical user, the final difficulty of the previous human-computer interaction is retrieved and reduced by 2 levels to serve as the initial task level of this human-computer interaction.

[0051] Compared with the prior art, the present invention has the following technical effects:

[0052] 1. Different from the existing speech training such as dictation, reading and writing on the market, the audio-visual matching ability training of the embodiment of the present invention starts from the basic structure of reading ability, fills the gap of the underlying perceptual defect training intervention products, and divides the core perceptual modules that are highly related to dyslexia, including auditory range recognition and auditory induction duration. Therefore, through perceptual training, the brain function of children with dyslexia is improved, and their basic abilities such as speech awareness, auditory attention, and visual perception are improved, so that the user's audio-visual integration ability is greatly improved, and the symptoms of dyslexia are alleviated.

[0053] 2. The training aims to improve children's more general visual perception impairment rather than their auditory processing ability for specific texts. The training uses non-verbal symbols and hearing to make direct comparisons of training results across language systems. It is suitable for children of all ages, literacy foundations, and reading levels, and avoids the impact of literacy problems on the training of children with dyslexia.

[0054] 3. According to the correctness of the user input options, the training parameters are adjusted according to the parameter setting rules, the corresponding training conditions are changed, and the training parameters closest to the user's zone of proximal development are provided to the user, and the exercises that are most suitable and timely meet the user's needs are provided. In this way, the training is targeted and adaptive, and the user can also train repeatedly according to his or her own situation, making full use of cognitive plasticity, improving basic cognitive ability, and promoting reading-related cognitive development. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A structural diagram of a non-verbal human-computer interaction system for developmental dyslexia provided by the first embodiment of the present invention;

[0056] Figure 2 An overall flow chart of a non-verbal human-computer interaction method for developmental dyslexia provided by the second embodiment of the present invention;

[0057] Figure 3 This is a detailed flow chart of a non-verbal human-computer interaction method for developmental dyslexia provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0058] The technical content of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] The technical concept of the embodiment of the present invention is to focus on improving the visual and auditory matching ability of children with dyslexia, rather than simply targeting specific text pronunciation processing skills. By using non-verbal symbols to train the visual and auditory matching ability, the visual and auditory processing ability of children with dyslexia can be targeted to enhance the cognitive ability related to reading from a more basic level. This method not only expands the scope of applicable populations and avoids the requirements for children's reading knowledge and literacy, but also effectively reduces the impact of individual differences in literacy ability on training effectiveness, and is suitable for all children with developmental dyslexia.

[0060] The embodiment of the present invention targets the visual-auditory matching ability deficiency of children with dyslexia, and based on the principle of cognitive plasticity, adopts a computerized training model to improve the visual-auditory matching ability of children. By matching auditory information with visual symbols, it helps individuals better understand and process the acquired information, improves phonetic awareness, and enhances reading fluency and attention, thereby achieving the purpose of improving their reading ability.

[0061] First embodiment

[0062] like Figure 1 As shown, a non-verbal human-computer interaction system for developmental dyslexia provided by the first embodiment of the present invention includes an evaluation unit 1, a primary task unit 2, a high-level task unit 3, a parameter configuration unit 4, an interaction unit 5 and a feedback unit 6. Among them, the evaluation unit 1 is used to collect the user's personal information and the evaluation results of the reading ability. The primary task unit 2 is connected to the evaluation unit 1 to push primary audio-visual matching tasks of different difficulties to the user based on the user's evaluation results. The high-level task unit 3 is connected to the evaluation unit 1 to push high-level audio-visual matching tasks of different difficulties to the user based on the user's evaluation results. The parameter configuration unit 4 is connected to the primary task unit 2 and the high-level task unit 3, and has an audio-visual matching element library for configuring primary audio-visual matching tasks and high-level audio-visual matching tasks. The interaction unit 5 is connected to the primary task unit 2 and the high-level task unit 3 to perform human-computer interaction with the user based on the primary audio-visual matching task or the high-level audio-visual matching task. The feedback unit 6 is connected to the interaction unit 5 to obtain the user's human-computer interaction results, and to show the correctness to the user, and explain the purpose and significance of this human-computer interaction.

[0063] The following is a detailed description of each functional unit of the non-verbal human-computer interaction system:

[0064] 1. Evaluation Unit

[0065] In this embodiment, the evaluation unit 1 includes an information entry module 11 and a cognitive evaluation module 12. Specifically, the information entry module 11 is used to collect the user's personal information, such as age, gender, education level, etc., and store it in the user database. The cognitive evaluation module 12 is used to evaluate the user's reading ability (for example, through a built-in scale or a built-in evaluation task) and obtain the evaluation results.

[0066] 2. Primary Task Unit

[0067] In this embodiment, the primary task unit 2 is connected to the evaluation unit 1. When the user completes the cognitive evaluation, the primary task unit 2 will push primary audio-visual matching tasks of different difficulties to the user according to the user portrait of the user.

[0068] Specifically, the primary audio-visual matching task is configured to select a matching primary visual sequence from two primary visual sequences according to a primary auditory sequence (the specific configuration process is described in detail below). The primary visual sequence is composed of multiple visual elements arranged in a preset manner to form primary visual sequences of different difficulties. And the primary auditory sequence is matched with one of the primary visual sequences.

[0069] In the rule description stage, the first stage presents two primary visual sequences. Clicking on the visual element of any primary visual sequence will play the corresponding auditory element, which is convenient for users to perceive the audio-visual matching. The second stage presents two columns of primary visual sequences and plays a primary auditory sequence, which corresponds to one of the primary visual sequences. When playing the auditory sequence, the corresponding visual sequence elements will light up one after another, indicating the progress of the current audio-visual sequence. After this animation is played once, it will be played repeatedly before the user confirms. Finally, a confirmation button is presented at the bottom of the screen, which is required for the user to click to confirm and start training.

[0070] 3. Advanced Task Unit

[0071] In this embodiment, the advanced task unit 3 is connected to the evaluation unit 1. When the user completes the cognitive evaluation, the advanced task unit 3 will push advanced audio-visual matching tasks of different difficulties to the user according to the user portrait of the user.

[0072] Specifically, the high-level audio-visual matching task is configured to identify, from a high-level visual sequence, a visual element that matches the last auditory element of a high-level auditory sequence based on a high-level auditory sequence (the specific configuration process is described in detail below).

[0073] In the rule description stage, the first stage includes n visual elements of the high-level visual sequence and the corresponding auditory elements. Clicking on the visual sequence element will play the corresponding auditory element, which is convenient for users to perceive the audio-visual matching. The second stage presents a regular visual sequence and plays a regular auditory sequence. This regular auditory sequence corresponds to the regular visual sequence. When playing the regular auditory sequence, the visual elements in the corresponding regular visual sequence will light up one after another, prompting the current audio-visual progress. When the auditory element of the last auditory sequence is played, the gesture picture on the screen will change, resembling a downward touch, prompting the user to initiate the behavior of clicking the screen at this time. After this animation is played once, it is played repeatedly before the user confirms. Finally, a confirmation button is presented at the bottom of the screen, which is required for the user to click to confirm and start training.

[0074] It should be understood that in the rule description stage, the number of elements in the rule visual sequence and the rule auditory sequence is one-to-one corresponding to facilitate the user to understand the training rules. However, during formal training, the number of auditory elements in the advanced auditory sequence is greater than the number of visual elements in the advanced visual sequence. The user needs to pay attention to which visual element the current auditory element corresponds to, so as to click the screen when the auditory element corresponding to the last visual element is played.

[0075] 4. Parameter Configuration Unit

[0076] In this embodiment, the parameter configuration unit 4 is connected to the primary task unit 2 and the advanced task unit 3, and has an audio-visual matching element library 41, so as to configure the primary audio-visual matching task and the advanced audio-visual matching task based on the audio-visual matching element library. Specifically, the audio-visual matching element library includes multiple visual elements, multiple auditory elements, and multiple element arrangements, and the multiple visual elements correspond one to one with the multiple auditory elements.

[0077] Specifically, in this embodiment, the visual element is a preset graphic (e.g., a cuboid), and the auditory element is a stimulating sound. The higher the height of the preset graphic, the higher the pitch of the stimulating sound; the longer the length of the preset graphic, the longer the duration of the stimulating sound; and the wider the width of the preset graphic, the stronger the intensity of the stimulating sound.

[0078] Furthermore, the various element arrangement modes include at least: (1) simple Gestalt, including a Gestalt structure of monotonous increasing or decreasing scale; (2) Gestalt subgroups, including multiple subgroups, and each subgroup has a common Gestalt structure; (3) subgroups separated by pauses, including multiple subgroups, different subgroups are marked by sound pauses, and stress is avoided at the starting point of the subgroup; (4) subgroups with the same block but different order, including multiple subgroups, each subgroup contains the same elements, and the order of elements arrangement is different between subgroups.

[0079] The following describes the configuration process of the primary audio-visual matching task and the advanced audio-visual matching task in detail:

[0080] (1) Configuration of the primary audio-visual matching task

[0081] ① Randomly select two groups of visual elements of the same number from the audio-visual matching element library, for example: each group is 10 cuboids of different sizes;

[0082] ② Determine the task difficulty based on the evaluation results, and select the corresponding element arrangement method based on the task difficulty;

[0083] ③ Based on the retrieved arrangement of elements, two groups of equal number of visual elements are arranged respectively, so as to form two equal but inconsistent primary visual sequences (i.e., two columns of cuboids, each column has 10 cuboids) for visual presentation;

[0084] ④ Retrieving the same number of auditory elements from the audio-visual matching element library, and arranging them according to the arrangement of the retrieved elements to form a primary auditory sequence (i.e., 10 stimulus sounds), so that the primary auditory sequence matches one of the primary visual sequences;

[0085] ⑤ After the two primary visual sequences are presented, play the primary auditory sequence.

[0086] (2) Configuration of advanced audio-visual matching tasks

[0087] ① randomly selecting a first number of visual elements from the audio-visual matching element library;

[0088] ② Determine the task difficulty based on the evaluation results, and select the corresponding element arrangement method based on the task difficulty;

[0089] ③ Arranging the first number of visual elements based on the retrieved element arrangement mode to form a high-level visual sequence for visual presentation;

[0090] ④ Randomly selecting a second number of auditory elements from the audio-visual matching element library, where the second number is greater than the first number;

[0091] ⑤ Arranging the second number of auditory elements based on the retrieved element arrangement mode to form a high-level auditory sequence, and making a part of the high-level auditory sequence match the high-level visual sequence;

[0092] ⑥After the advanced visual sequence is presented, play the advanced auditory sequence.

[0093] In addition, the parameter configuration unit 4 also includes a user preference module 42 for allowing the user to adjust the duration and playback interval of the auditory element. Specifically, in this embodiment, the standard duration of an auditory element is t 1 , the user can adjust the standard duration to T in the first time range 1The time interval between the presentation of two auditory elements is t 2 , the user can adjust the presentation time interval to T in the second time range 2 .

[0094] 5. Interaction Unit

[0095] In this embodiment, the interaction unit 5 is connected to the primary task unit 2 and the advanced task unit 3 to perform human-computer interaction with the user based on the primary audio-visual matching task or the advanced audio-visual matching task.

[0096] Specifically, the interaction unit 5 will first perform human-computer interaction with the user based on the primary audio-visual matching task, thereby obtaining the user's primary human-computer interaction result. Then, it is determined whether to start the training of the advanced task based on whether the primary human-computer interaction result meets the standard. Among them, if the user's primary human-computer interaction result does not meet the standard, it is necessary to continuously perform human-computer interaction with the user based on different primary audio-visual matching tasks until the user's primary human-computer interaction result meets the standard, and then start the training of the advanced task.

[0097] In addition, preferably, the non-verbal human-computer interaction system further includes a difficulty adjustment unit 7. The difficulty adjustment unit 7 is connected to the interaction unit 5 to receive the human-computer interaction result of the user and accumulate the correct number and the wrong number of the user. Wherein, each time the user makes a correct round, the accumulated correct number is +1, and when the accumulated correct number reaches 3 consecutively, the difficulty is increased by one level, and the accumulated correct number counter is cleared and recalculated. Each time the user makes a wrong round, the accumulated error number is +1, and when the accumulated error number reaches 2, the difficulty is reduced by one level, and the accumulated error number counter is cleared and recalculated.

[0098] 6. Feedback Unit

[0099] Specifically, the feedback unit 6 presents training feedback according to the user's human-computer interaction results (e.g., the duration, number of rounds, score, final difficulty level, etc. of this training), shows the user the accuracy rate, and explains the purpose and significance of this human-computer interaction. At the same time, the user's final score, training duration and difficulty level are stored in the user database.

[0100] The next time the user starts training, the final difficulty of the previous training is retrieved from the user database and downgraded by 2 levels before starting a new training session. When presenting the user's weekly and monthly training reports, the user's cumulative training time, accuracy, and level are displayed.

[0101] Second embodiment

[0102] like Figure 2 and Figure 3 As shown, based on the above first embodiment, the second embodiment of the present invention provides a non-verbal human-computer interaction method for developmental dyslexia, which specifically includes steps S1-S7:

[0103] S1: Obtain the user's personal information and reading ability assessment results.

[0104] Specifically, when a user logs into the non-verbal human-computer interaction system, he or she needs to first enter personal information and then take an assessment of reading ability to form a user profile for the user.

[0105] S2: Based on the user's evaluation results, a primary audio-visual matching task and an advanced audio-visual matching task are configured for the user based on a parameter configuration unit.

[0106] Specifically, in this step, the parameter configuration unit 4 will retrieve the data in the user database, and then the system will perform algorithm calculation and deployment and generate a non-verbal audio-visual matching task suitable for the user level. The specific task configuration process refers to the description of the parameter configuration unit 4 in the first embodiment, which will not be repeated here.

[0107] When the algorithm is being calculated, the system will determine whether the user is an initial user. If the user is an initial training user, the system will retrieve the user's age, education level, current speech awareness assessment score, and audio-visual ability task-based assessment score to calculate the user's audio-visual ability, and then determine the initial task level based on the user's audio-visual ability.

[0108] If the user is a historical user, the final difficulty of the previous human-computer interaction is retrieved and reduced by 2 levels as the initial task level of this human-computer interaction.

[0109] S3: Pushing the primary audio-visual matching task to the user for human-computer interaction, so as to obtain the primary human-computer interaction result of the user completing the primary audio-visual matching task.

[0110] After completing the configuration of the primary audio-visual matching task and the advanced audio-visual matching task, the user clicks OK to start human-computer interaction. The following are the specific steps for conventional human-computer interaction:

[0111] S31: The user clicks OK to start human-computer interaction.

[0112] Specifically, a background image is retrieved from the background image library. From the element library, the following are retrieved: (1) a preset visual sequence pattern; (2) the length, width, and relative height of a rectangle within the visual sequence pattern; (3) an auditory sequence element; and (4) an activity button. Then, the visual sequence is presented according to the current difficulty.

[0113] In addition, some interactive design buttons are in long-term continuous presentation mode, including:

[0114] (1) Activity button, which is displayed in the center of the left side of the screen. The button is a specific activity icon extracted from the element library.

[0115] (2) The replay button is located at the upper left corner of the screen and has the words replay and a replay symbol written on it.

[0116] (3) Adjustment button, located in the upper right corner of the screen, with the word Adjustment / Settings and a gear icon written on it.

[0117] S32: The user responds to the audio-visual sequence integration.

[0118] In the primary audio-visual matching task, the user needs to judge which primary visual sequence matches the played primary auditory sequence. After the user makes a response, the following feedback is given according to the correctness of the user's response:

[0119] When the user selects the correct option, a smiley face appears on the screen. Conversely, when the user selects the wrong option, the entire screen flashes and the auditory sequence is replayed, and the corresponding visual sequence elements light up one after another, indicating the progress of the current audio-visual sequence.

[0120] S33: The correctness of the user's selection and the reaction time are recorded in the user database, and the difficulty of the next round is adjusted according to the user's accuracy and reaction time, and this round ends.

[0121] Specifically, each time the user gets a round right, the cumulative correct number is +1. When the cumulative correct number reaches 3, the difficulty increases by one level, and the cumulative correct number counter is reset and recalculated. Each time the user gets a round wrong, the cumulative error number is +1. When the cumulative error number reaches 2, the difficulty decreases by one level, and the cumulative error number counter is reset and recalculated.

[0122] S34: According to the user's accuracy rate, adjust and present a new round of primary audio-visual matching tasks.

[0123] Specifically, the score result of the user's final computing task is used as the training result and evaluation of the user's computing unit, and the task difficulty that occurs during the user's next human-computer interaction is updated accordingly.

[0124] S4: Determine whether the primary human-computer interaction results meet the standards.

[0125] When the user meets the training standard in the primary non-verbal audio-visual matching task, the advanced non-verbal audio-visual matching task training module is started. If the standard is not met, the primary audio-visual matching task is reconfigured (i.e., step S3 is repeated), and human-computer interaction is performed with the user again based on the reconfigured primary audio-visual matching task until the primary human-computer interaction result meets the standard.

[0126] S5: Pushing the advanced audio-visual matching task to the user for human-computer interaction, so as to obtain the advanced human-computer interaction result of the user completing the advanced audio-visual matching task.

[0127] It can be understood that step S5 is substantially the same as step S3, except that:

[0128] (1) The number of visual and auditory elements retrieved is different;

[0129] (2) In the advanced audio-visual matching task, users are required to click an active button on the screen when the auditory sequence plays to the last visual element.

[0130] S6: Based on the high-level human-computer interaction results, the high-level audio-visual matching task is updated to carry out the next round of human-computer interaction until the human-computer interaction process is completed.

[0131] S7: Result feedback

[0132] Specifically, when a user completes a primary human-computer interaction or an advanced human-computer interaction, the system will provide feedback to the user on the interaction results based on the interaction duration, number of rounds, score, and final difficulty level of this human-computer interaction to help the user better understand their own situation.

[0133] In addition, after every week or month, the system will show the user the cumulative duration, accuracy and current level of human-computer interaction in the form of weekly or monthly reports.

[0134] In summary, the non-verbal human-computer interaction system and method for developmental dyslexia provided by the embodiments of the present invention have the following beneficial effects:

[0135] 1. Different from the existing speech training such as dictation, reading and writing on the market, the audio-visual matching ability training of the embodiment of the present invention starts from the basic structure of reading ability, fills the gap of the underlying perceptual defect training intervention products, and divides the core perceptual modules that are highly related to dyslexia, including auditory range recognition and auditory induction duration. Therefore, through perceptual training, the brain function of children with dyslexia is improved, and their basic abilities such as speech awareness, auditory attention, and visual perception are improved, so that the user's audio-visual integration ability is greatly improved, and the symptoms of dyslexia are alleviated.

[0136] 2. The training aims to improve children's more general visual perception impairment rather than their auditory processing ability for specific texts. The training uses non-verbal symbols and hearing to make direct comparisons of training results across language systems. It is suitable for children of all ages, literacy foundations, and reading levels, and avoids the impact of literacy problems on the training of children with dyslexia.

[0137] 3. According to the correctness of the user input options, the training parameters are adjusted according to the parameter setting rules, the corresponding training conditions are changed, and the training parameters closest to the user's zone of proximal development are provided to the user, and the exercises that are most suitable and timely meet the user's needs are provided. In this way, the training is targeted and adaptive, and the user can also train repeatedly according to his or her own situation, making full use of cognitive plasticity, improving basic cognitive ability, and promoting reading-related cognitive development.

[0138] It should be noted that the above embodiments are only examples, and the technical solutions of the various embodiments can be combined, all within the protection scope of the present invention.

[0139] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0140] The above is a detailed description of a non-verbal human-computer interaction system and method for developmental dyslexia provided by the present invention. For those skilled in the art, any obvious changes made to the present invention without departing from the essence of the present invention will constitute an infringement of the patent right of the present invention and will bear corresponding legal responsibilities.

Claims

1. A non-verbal human-computer interaction system for developmental dyslexia, characterized in that include: The evaluation unit includes an information input module and a cognitive evaluation module; wherein the information input module is used to collect the user's personal information, and the cognitive evaluation module is used to evaluate the user's reading ability and obtain the evaluation results; A primary task unit is connected to the evaluation unit to push primary audio-visual matching tasks of different difficulty levels to the user based on the evaluation result; wherein the primary audio-visual matching task is configured to select a matching primary visual sequence from two primary visual sequences according to a primary auditory sequence; the primary visual sequence is composed of a plurality of visual elements arranged in a preset manner to form primary visual sequences of different difficulty levels; the primary auditory sequence is matched with one of the primary visual sequences; A high-level task unit is connected to the evaluation unit to push high-level audio-visual matching tasks of different difficulties to the user based on the evaluation result; wherein the high-level audio-visual matching task is configured to identify a visual element that matches the last auditory element of a high-level auditory sequence from a high-level visual sequence according to a high-level auditory sequence; and the number of auditory elements in the high-level auditory sequence is greater than the number of visual elements in the high-level visual sequence; a parameter configuration unit connected to the primary task unit and the advanced task unit, and having an audiovisual matching element library, so as to configure the primary audiovisual matching task and the advanced audiovisual matching task based on the audiovisual matching element library; wherein the audiovisual matching element library includes a plurality of visual elements, a plurality of auditory elements, and a plurality of element arrangement modes, and the plurality of visual elements correspond one to one with the plurality of auditory elements; An interaction unit connected to the primary task unit and the advanced task unit to perform human-computer interaction with a user based on the primary audio-visual matching task or the advanced audio-visual matching task; The feedback unit is connected to the interaction unit to obtain the human-computer interaction result of the user, show the accuracy rate to the user, and explain the purpose and significance of this human-computer interaction.

2. The non-verbal human-computer interaction system according to claim 1, characterized in that The primary audio-visual matching task is configured in the following way: Randomly select two groups of visual elements of the same number from the audio-visual matching element library; Determine the task difficulty according to the evaluation result, and retrieve the corresponding element arrangement method based on the task difficulty; Based on the retrieved arrangement of elements, two groups of visual elements of equal number are arranged respectively, so as to form two primary visual sequences of equal length but inconsistent for visual presentation; Retrieving the same number of auditory elements from the audio-visual matching element library, and arranging the retrieved elements in an arrangement manner to form a primary auditory sequence, so that the primary auditory sequence matches one of the primary visual sequences; And, after the two primary visual sequences are presented, the primary auditory sequence is played.

3. The non-verbal human-computer interaction system according to claim 1, characterized in that The high-level audio-visual matching task is configured in the following way: Randomly retrieve a first number of visual elements from the audio-visual matching element library; Determine the task difficulty according to the evaluation result, and retrieve the corresponding element arrangement method based on the task difficulty; Arranging the first number of visual elements based on the retrieved element arrangement mode to form a high-level visual sequence for visual presentation; Randomly retrieve a second number of auditory elements from the audiovisual matching element library, where the second number is greater than the first number; Arranging the second number of auditory elements based on the retrieved element arrangement mode to form a high-level auditory sequence, and making a part of the high-level auditory sequence match the high-level visual sequence; And, after the high-level visual sequence is presented, the high-level auditory sequence is played.

4. The non-verbal human-computer interaction system according to claim 1, wherein: The visual element is a preset graphic, and the auditory element is a stimulating sound; Among them, the higher the height of the preset graphic, the higher the pitch of the stimulation sound; the longer the length of the preset graphic, the longer the duration of the stimulation sound; the wider the width of the preset graphic, the stronger the intensity of the stimulation sound.

5. The non-verbal human-computer interaction system as claimed in claim 4, characterized in that The various element arrangements include at least: Simple Gestalt, which consists of a monotonous ascending or descending Gestalt structure; Gestalt subgroups, including multiple subgroups, and each subgroup has a common Gestalt structure; Subgroups separated by pauses, including multiple subgroups, with different subgroups marked by sound pauses, and avoiding stress at the beginning of the subgroup; The subgroups have the same block but different order, including multiple subgroups, each subgroup contains the same elements, and the order of elements between subgroups is different.

6. The non-verbal human-computer interaction system according to claim 4, characterized in that: The parameter configuration unit further includes a user preference module for allowing the user to adjust the duration and playing interval of the auditory element; The standard duration of an auditory element is t1, and the user can adjust the standard duration to T1 within a first time range; the presentation time interval between two auditory elements is t2, and the user can adjust the presentation time interval to T2 within a second time range.

7. The non-verbal human-computer interaction system according to claim 1, characterized in that Also includes: A difficulty adjustment unit, connected to the interaction unit, to receive the human-computer interaction result of the user and accumulate the correct number and the wrong number of the user; Among them, when the cumulative number of correct answers reaches 3, the task difficulty increases by one level, and the cumulative number of correct answers is reset and recalculated; when the cumulative number of errors reaches 2, the task difficulty decreases by one level, and the cumulative number of errors is reset and recalculated.

8. A non-verbal human-computer interaction method for developmental dyslexia, implemented based on the non-verbal human-computer interaction system according to any one of claims 1 to 7, characterized in that The steps include: Obtain the user's personal information and reading ability assessment results; Based on the evaluation result of the user, configuring a primary audio-visual matching task and an advanced audio-visual matching task for the user based on a parameter configuration unit; Pushing the primary audio-visual matching task to the user for human-computer interaction, so as to obtain a primary human-computer interaction result of the user completing the primary audio-visual matching task; Determine whether the primary human-computer interaction result meets the standard; if so, start the human-computer interaction of the advanced audio-visual matching task; If the standard is not met, reconfigure the primary audio-visual matching task, and perform human-computer interaction with the user again based on the reconfigured primary audio-visual matching task until the primary human-computer interaction result meets the standard; Pushing the advanced audio-visual matching task to the user for human-computer interaction, so as to obtain an advanced human-computer interaction result of the user completing the advanced audio-visual matching task; The high-level audio-visual matching task is updated based on the high-level human-computer interaction result to perform the next round of human-computer interaction until the human-computer interaction process is completed.

9. The non-verbal human-computer interaction method according to claim 8, characterized in that: During the process of the user performing human-computer interaction, the correct number and the wrong number of the user are accumulated based on the human-computer interaction result of the user; When the cumulative number of correct answers reaches 3, the difficulty level increases by one level, and the cumulative number of correct answers is reset and recalculated; When the cumulative number of errors reaches 2, the difficulty level drops by one level, and the cumulative number of errors is reset and recalculated.

10. The non-verbal human-computer interaction method according to claim 9, characterized in that: If the user is an initial user, the user's audio-visual ability is calculated based on the user's personal information and the evaluation result, thereby determining the user's initial task level based on the audio-visual ability; If the user is a historical user, the final difficulty of the previous human-computer interaction is retrieved and reduced by 2 levels to serve as the initial task level of this human-computer interaction.

Images