Autism intervention and evaluation system based on virtual reality

Abstract

The invention relates to an infantile autism intervention and evaluation system based on virtual reality, belongs to the field of auxiliary rehabilitation training, and is used for solving the problem that a personalized simulation controllable environment cannot be provided for an infantile autism patient in the prior art. The system comprises a virtual reality module and an intervention evaluation feedback module. The intervention evaluation feedback module is configured to be used for obtaining a preliminary defect screening result of the patient, or generating evaluation results of improvement or reduction of recent linguistic competence and improvement degree of behavior competence of the patient based on voice data and behavior data collected by the virtual reality module; the virtual reality module is configured to be capable of customizing a virtual reality plot and sending voice data and behavior data collected in an intervention training process to the intervention evaluation feedback module; when the system is used, targeted intervention training and evaluation are carried out on the autism patient by utilizing a self-defined virtual reality plot according to a preliminary defect screening result or an evaluation result generated by the intervention evaluation feedback module.

Description

Technical Field

[0001] The invention relates to the field of auxiliary rehabilitation training, and in particular to an autism intervention and evaluation system based on virtual reality. Background Art

[0002] Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder that affects social interaction, communication ability, and behavioral performance. People with autism often face problems such as lack of social skills, difficulty in language communication, and stereotyped and repetitive behaviors. With the development of virtual reality (VR) technology, people have begun to explore its application in medical intervention, especially its potential in autism treatment and assessment.

[0003] Existing virtual reality systems provide autistic patients with a simulated, controllable environment so that they can conduct intervention and assessment under conditions that are closer to real-life situations. However, most existing virtual reality systems are divorced from real-life social scenarios, and the assessment of the development of patients' language abilities is relatively one-sided. There are also large differences among individual autistic patients, making existing virtual reality systems not suitable for every autistic patient. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the purpose of the present invention is to provide an autism intervention and assessment system based on virtual reality, which can provide personalized intervention and assessment for each autistic patient. The specific technical solution is as follows.

[0005] On the first aspect, this case proposes an autism intervention and assessment system based on virtual reality, the system comprising a virtual reality module and an intervention assessment feedback module; wherein: the intervention assessment feedback module is configured to obtain the patient's preliminary defect screening results, or to generate an assessment result of the patient's recent improvement or decline in language ability and the degree of improvement in behavioral ability based on the voice data and behavioral data collected by the virtual reality module; the virtual reality module is configured to be able to customize virtual reality plots, and send the voice data and behavioral data collected during the intervention training process to the intervention assessment feedback module; when using the system, based on the preliminary defect screening results or the assessment results generated by the intervention assessment feedback module, the customized virtual reality plot is used to conduct targeted intervention training and assessment on autistic patients.

[0006] In one embodiment of the above technical solution, the intervention assessment feedback module includes a speech ability assessment unit and a behavioral ability assessment unit; the speech assessment unit is configured to obtain the patient's speech features based on the speech data collected when the patient simulates the scene plot, and evaluate the patient's recent improvement or decline in language ability based on the speech features; the action assessment unit is configured to obtain the patient's behavioral features based on the action data recognized by the patient when the patient simulates the scene plot, and evaluate the degree of improvement in the patient's recent behavioral ability based on the behavioral features.

[0007] In one implementation of the above technical solution, the speech features include accuracy, fluency and formant.

[0008] In one embodiment of the above technical solution, the degree of improvement of the patient's recent behavioral ability is evaluated based on behavioral characteristics, including: judging the correctness of the imitation of the three-dimensional trajectory of the patient's head and hand movements and the head and hand movement trajectory of the character's animation behavior in the actual virtual reality animation based on the behavioral characteristics, and analyzing whether there are stereotyped movements and the degree of stereotyped movements, and evaluating the degree of improvement of the patient's recent behavioral ability based on the correctness and the degree of stereotyped movements.

[0009] In one implementation of the above technical solution, the data used by the speech evaluation unit and the action evaluation unit in the evaluation are data collected at least 5 times.

[0010] In one embodiment of the above technical solution, the virtual reality module includes a display device, a virtual scene module, a motion recognition device, an audio playback device and a voice recording device; the display device is configured to display the virtual reality scene; the virtual scene module is configured to provide a virtual reality plot for intervention training; the motion recognition device is configured to identify and collect the patient's head and hand position information, head and hand direction and motion information; the voice recording device is configured to identify and collect the patient's voice data when the patient simulates the character's voice in the virtual reality scene; the audio playback device is configured to play audio files related to the virtual reality scene.

[0011] In one implementation of the above technical solution, the virtual scenario module includes a custom plot unit; the custom plot unit is configured to set and play a custom virtual reality plot.

[0012] In one implementation of the above technical solution, the custom plot unit includes a custom plot editing subunit and a custom plot playing subunit; the custom plot editing subunit is configured to customize the plot, including editing the actions and lines of the virtual characters in the scene plot; the custom plot playing subunit is configured to play the edited custom plot.

[0013] On the second aspect, the present case proposes a computer-readable storage medium storing a computer program that can be loaded by a processor and executed by any of the above-mentioned systems.

[0014] Beneficial technical effects: The system proposed in the present disclosure can provide a personalized plot definition function. There are great differences between individuals with autism, and there are large differences in perception, cognition, information processing and other fields. At the same time, there are also diversity in the changes in their voice characteristics. Their particularity determines that they should be provided with more personalized intervention content. This system designs and generates plots suitable for their development needs based on the individual differences and intervention needs of autistic patients through the plot customization function. On the one hand, it can meet the intervention training needs of different autistic patients for social skills in different situations. On the other hand, during the intervention training process, data is collected, and after data processing and evaluation, a comprehensive evaluation result can be obtained to improve the reliability and accuracy of language ability evaluation, so as to better grasp the effect of intervention training. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 , one Schematic diagram of system structure block in an implementation manner.

[0016] Figure 2 , one A schematic diagram of a virtual reality module structure frame in an implementation manner.

[0017] Figure 3 , one Schematic diagram of the intervention assessment feedback module structure frame in this implementation mode.

[0018] In the figure, 110-virtual reality module, 120-intervention evaluation feedback module, 111-display device, 112-control device, 113-virtual scenario module, 114-action recognition device, 115-audio playback device, 116-voice recording device, 113a-preset plot unit, 113b-customized plot unit, 113b1-customized plot editing subunit, 113b2-customized plot playback subunit, 121-defect screening module, 122-data processing module, 123-ability assessment module, 123a-language ability assessment unit, 123b-behavioral ability assessment unit. DETAILED DESCRIPTION

[0019] The following will be combined with the attached Figure 1 , clearly and completely describing how to implement the technical solution of this case. Obviously, the described implementation method is only a part of the implementation method of this case, not all of the implementation methods. Based on the implementation method in this case, all other implementation methods obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] Reference Figure 1 As shown, in one embodiment of the autism intervention and evaluation system based on virtual reality, a virtual reality module (110) and an intervention evaluation feedback module (120) are included. Wherein, the intervention evaluation feedback module is used to perform preliminary defect screening on the patient. Based on the preliminary defect screening results, a virtual reality plot can be used for intervention training. The virtual reality module is responsible for the display of the virtual reality plot, guides the autistic patient to interact with the virtual reality scene, identifies and records the movements and voices of the autistic patient, and transmits the voice information and movement information to the intervention evaluation feedback module, and the intervention evaluation feedback module performs further data evaluation work based on the patient's movements and voice data. Then, based on the evaluation results, in the virtual reality module, not only can the preset virtual reality plot be selected, but also the virtual reality plot can be customized as needed, so as to carry out individualized intervention training for individual patients. When customizing the virtual reality plot, the plot is usually customized according to the preliminary defect screening results or the evaluation results generated by the intervention evaluation feedback module to improve the pertinence of individual intervention training.

[0021] See also Figure 2 In a specific embodiment of the virtual reality module, it includes a display device (111), a control device (112), a virtual scene module (113), a motion recognition device (114), an audio playback device (115) and a voice recording device (116). Specifically, the display device is responsible for displaying the virtual reality scene. The audio playback device is used to play the virtual character plot audio file related to the virtual reality scene. The motion recognition device is used to identify and collect the head and hand position information and head and hand direction and motion information of the autistic patient when the patient simulates the character action in the virtual reality scene. The voice recording device is responsible for identifying and collecting the voice data of the autistic patient when the patient simulates the character voice in the virtual reality scene. The virtual scene module is responsible for playing the animation of the entire virtual reality scene. The virtual scene module may include a preset plot unit (113a) and a custom plot unit (113b), the former is used to play the basic animation scene defined and stored by the software system, and the latter is responsible for processing the setting and playing of the custom plot and other operations, so that the patient can carry out targeted intervention training. Finally, the control device is responsible for inputting the user's interactive operations on the software system, including controlling the display device to display the picture to be played by the virtual scenario module, controlling the audio playback device to play the voice matching the picture, and at the same time, when the virtual scenario is playing, controlling the action recognition device to collect patient behavior data and controlling the voice recording device to record patient voice data.

[0022] The above-mentioned custom plot unit may include a custom plot editing subunit (113b1) and a custom plot playing subunit (113b2). Specifically, the custom plot editing subunit is used for assessing personnel to customize the plot according to needs, including editing the actions and lines of virtual characters in the scene plot, and collecting and evaluating the voice data of the diversity of autistic patients by diversifying the plot according to the specific evaluation results of the subsequent voice. The custom plot playing subunit is combined with an audio playback device to play the plot set by the custom plot editing subunit for autistic patients, and at the same time, the behavior data and voice data of autistic patients are collected through a motion recognition device and a voice recording device.

[0023] See also Figure 3 The intervention assessment feedback module includes a defect screening module (121), a data processing module (122), and an ability assessment module (123). Specifically, the defect screening module is used to analyze the development status of autistic patients, obtain basic information on the ability defects of autistic patients by adopting a questionnaire-like survey model, and generate intervention training target information. The data processing module is used to collect data on social skills and behavior patterns. The ability assessment module is used to evaluate the collected data on social skills and generate an assessment result. The ability assessment module may include a language ability assessment unit (123a) and a behavior ability assessment unit (123b). Each unit is introduced as follows.

[0024] The above-mentioned language ability assessment unit is used to extract speech features and conduct assessments. The specific implementation is briefly described as follows: through deep learning technology, a speech data set is used to train a speech assessment model, and then the speech assessment model is used to perform feature extraction and other operations on the collected patient speech data. After obtaining the feature data such as the accuracy, fluency and resonance peak of the patient's speech, firstly, the accuracy of the language content and the character lines in the virtual reality plot are compared according to the speech recognition results, and then the development and changes of the patient's various sound features are evaluated. The speech feature data of normal people in the same situation is used as a control group. Through at least 5 intervention processes, the data obtained can be used to obtain the evaluation results of the recent improvement or decline in language ability, and according to the changes in the evaluation results, the specific changes in language ability are expressed in percentage.

[0025] The above-mentioned behavioral ability evaluation unit is used to obtain the patient's behavioral characteristics through a deep learning method, and judge the correctness of the imitation of the three-dimensional trajectory of the head and hand movements of the autistic patient and the head and hand movement trajectory of the animated behavior of the character in the actual virtual reality animation based on the behavioral characteristics, and analyze whether there are stereotyped movements and the degree of stereotyped movements, and analyze the behavioral performance collected during at least 5 interventions. The degree of stereotyped movements is obtained by comparing with the behavioral data of normal people as a control group. Based on the correctness and the degree of stereotyped movements, an assessment of recent behavioral abilities is performed, and according to the changes in the evaluation results, the specific changes in behavioral abilities are expressed in percentage.

[0026] In addition, those skilled in the art can clearly understand that the above system also includes a software or hardware storage device.

[0027] Through the description of the above implementation methods, those skilled in the art can clearly understand that the disclosed system can be implemented by means of software plus necessary general hardware, and of course can also be implemented by dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits, etc. However, for the present disclosure, software program implementation is a better implementation method in most cases.

[0028] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. A virtual reality-based autism intervention and assessment system, characterized by: The system includes a virtual reality module and an intervention assessment feedback module; wherein: The intervention assessment feedback module is configured to obtain the preliminary defect screening results of the patient, or generate the assessment results of the patient's recent improvement or decline in language ability and the degree of improvement in behavioral ability based on the voice data and behavioral data collected by the virtual reality module; The virtual reality module is configured to customize the virtual reality plot and send the voice data and behavior data collected during the intervention training process to the intervention evaluation feedback module; When using the system, based on the preliminary defect screening results or the evaluation results generated by the intervention evaluation feedback module, autistic patients are given targeted intervention training and evaluation using a customized virtual reality scenario.

2. The system according to claim 1, characterized in that: The intervention assessment feedback module includes a speech ability assessment unit and a behavior ability assessment unit; The speech evaluation unit is configured to obtain speech features of the patient based on speech data collected by the patient when simulating the scenario, and to evaluate the improvement or decline of the patient's recent language ability based on the speech features; The action evaluation unit is configured to obtain the patient's behavioral characteristics based on the action data recognized by the patient when simulating the scene plot, and evaluate the patient's recent improvement in behavioral ability based on the behavioral characteristics.

3. The system according to claim 2, characterized in that The speech features include accuracy, fluency and formant.

4. The system according to claim 2, characterized in that The degree of improvement of the patient's recent behavioral ability is assessed based on behavioral characteristics, including: judging the correctness of the imitation of the three-dimensional trajectory of the patient's head and hand movements and the head and hand movement trajectory of the character's animation behavior in the actual virtual reality animation based on the behavioral characteristics, and analyzing whether there are stereotyped movements and the degree of stereotyped movements, and assessing the degree of improvement of the patient's recent behavioral ability based on the correctness and the degree of stereotyped movements.

5. The system according to claim 2, characterized in that The data used by the speech evaluation unit and the action evaluation unit in the evaluation are data collected at least 5 times.

6. The system according to claim 1, characterized in that: The virtual reality module includes a display device, a virtual scene module, a motion recognition device, an audio playback device, and a voice recording device; The display device is configured to display the virtual reality scene; The virtual scenario module is configured to provide a virtual reality scenario for intervention training; The motion recognition device is configured to recognize and collect the patient's head and hand position information, head and hand direction, and motion information; The voice recording device is configured to recognize and collect the voice data of the patient when the patient simulates the voice of the character in the virtual reality scene; The audio playback device is configured to play audio files related to the virtual reality scene.

7. The system according to claim 6, characterized in that: The virtual scenario module includes a custom plot unit; The custom plot unit is configured to set and play a custom virtual reality plot.

8. The system according to claim 7, characterized in that: The custom plot unit includes a custom plot editing subunit and a custom plot playing subunit; The custom plot editing subunit is configured to customize the plot, including editing the actions and lines of the virtual characters in the scene plot; The custom plot playing subunit is configured to play the edited custom plot.

9. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the system as claimed in any one of claims 1 to 8.

Images