A virtual digital image autism rehabilitation intervention system

Abstract

A virtual digital image autism rehabilitation intervention system is disclosed. The system includes: a multimodal data acquisition device for acquiring at least one input modal data of eye movement, facial expression, body movement and sound of autistic children; a multimodal output device for outputting at least one output modal information of image, sound, touch and smell; a virtual digital image presentation device for presenting a virtual digital image that can interact with autistic children; a multimodal fusion module for controlling the virtual digital image to perform corresponding actions through the virtual digital image presentation device based on the input modal data, and generating output modal information through the multimodal output device.

Description

Technical Field

[0001] The present disclosure relates to the technical field of autism rehabilitation intervention, and more specifically, to a virtual digital image autism rehabilitation intervention system. Background Art

[0002] Autism Spectrum Disorder (ASD), also known as autism, is a type of neurodevelopmental disorder that occurs in early childhood and has an unclear cause. Generally speaking, ASD begins before the age of 3. The main manifestations of ASD are social interaction disorders, communication disorders, and psychological developmental disorders such as limited, stereotyped, and repetitive behaviors.

[0003] At present, behavioral intervention is a proven effective method for developmental disabilities such as autism. Its theoretical basis is based on ABA (Applied Behavior Analysis). ABA is a behavioral science that aims to study the functional relationship between human behavior and the environment. By arranging different antecedents and consequences, or by direct teaching, people's specific behaviors (such as social skills, communication, reading, academics, adaptive behavior, fine motor skills, hygiene, appearance, housework skills, punctuality, work ability, etc.) are changed. The reason why behavioral intervention is effective is that it uses ABA as a theoretical guide to help children with autism establish developmental milestone behaviors so that children can have the skills to adapt to society and survive independently.

[0004] In the prior art, interactive game contents such as scene images, animations, and audio are presented through two-dimensional images on the display screen to achieve rehabilitation training that is cyclically progressive, including learning, acquisition, generalization, and consolidation. Digital games are effective for rehabilitation intervention for children with these conditions, but in digital game intervention scenarios, there is a lack of family-style companionship and social interaction in terms of sensory perception. The effect evaluation based on the operational data of digital game intervention training for children with these conditions is slightly single-dimensional and not comprehensive enough. Summary of the invention

[0005] The disclosed embodiment provides a new virtual digital image autism rehabilitation intervention system.

[0006] In a first aspect, a virtual digital image autism rehabilitation intervention system is provided, comprising: a multimodal data acquisition device for acquiring at least one input modal data of eye movement, facial expression, body movement and sound of an autistic child; a multimodal output device for outputting at least one output modal information of image, sound, touch and smell; a virtual digital image presentation device for presenting a virtual digital image that can interact with the autistic child; a multimodal fusion module for controlling the virtual digital image to perform corresponding actions through the virtual digital image presentation device based on the input modal data, and generating output modal information through the multimodal output device.

[0007] Through the virtual digital image autism rehabilitation intervention system in some embodiments of the present disclosure, it is possible to improve the training effect by providing real-time feedback through the virtual digital image based on the autistic child's reactions during intervention training, such as eye movements, facial expressions, body movements, sounds and other data.

[0008] In addition, the virtual digital image autism rehabilitation intervention system of some embodiments of the present invention presents the corresponding actions in the intervention process through the virtual digital image, accompanies and interacts with the autistic children, reduces the resistance of the autistic children, enhances the enthusiasm and compliance of the autistic children in participating in digital intervention training, and improves the effect of rehabilitation intervention.

[0009] Other features and advantages of the embodiments of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the embodiments of the present disclosure.

[0011] Figure 1 A schematic block diagram of a virtual digital image autism rehabilitation intervention system based on visual assistance according to an embodiment of the present disclosure is shown.

[0012] Figure 2 A schematic diagram of a virtual digital image autism rehabilitation intervention system according to an embodiment of the present disclosure is shown.

[0013] Figure 3 A schematic block diagram of a virtual digital image autism rehabilitation intervention system based on visual assistance according to another embodiment of the present disclosure is shown.

[0014] Figure 4 A schematic computing device that can be used to implement a virtual digital image autism rehabilitation intervention system according to an embodiment of the present disclosure is shown.

[0015] Figure 5A schematic network of an autism rehabilitation intervention system in which a virtual digital image can be deployed according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0016] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0017] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0018] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0019] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0020] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0021] Figure 1 A schematic block diagram of a virtual digital image autism rehabilitation intervention system according to an embodiment of the present disclosure is shown.

[0022] Figure 1 The autism rehabilitation intervention system shown includes: a multimodal data acquisition device 1, a multimodal output device 2, a virtual digital image presentation device 3, and a multimodal fusion module 4.

[0023] The multimodal data acquisition device 1 is used to acquire at least one input modality data of eye movement, facial expression, body movement and sound of an autistic child.

[0024] Conventional autism rehabilitation intervention systems directly input pre-set output items and images without paying attention to the current state of autistic children. Here, by setting up a multimodal data acquisition device, the current state of autistic children can be acquired in real time, so that the output items and / or output images can be adjusted in real time. In practice, because autistic children are not good at communicating with the outside world, but are very sensitive to environmental changes, existing autism rehabilitation intervention systems often cannot achieve the desired intervention effects. Here, multimodal data acquisition equipment can help the autism rehabilitation intervention system determine the current state of autistic children in real time, thereby adjusting the output items and / or output images to improve the intervention effect.

[0025] In one embodiment, the multimodal data acquisition device includes a first camera, a second camera, and a microphone. The first camera is used to collect the movements and facial expressions of the autistic child. The second camera is used to collect the eye movements of the autistic child. The microphone is used to collect the voice of the autistic child.

[0026] A variety of data can be collected through multimodal data collection equipment. For example, the language feedback, audio feedback, movement feedback, and related emotional information of the patient during the game are of great value in evaluating the intervention effect of the patient. Therefore, the use of multimodal data collection equipment can improve the effect of digital game-style rehabilitation training.

[0027] For example, Figure 2 As shown, the multimodal data acquisition device includes a first camera 11 for capturing the movements and facial expressions of autistic children. In this example, it can be set above the virtual digital image presentation device to capture all the movements and expressions of autistic children as much as possible. The multimodal data acquisition device also includes a second camera 12 for capturing the eye movements of autistic children. Since the second camera 12 needs to capture the eye movements of autistic children during the intervention process, its setting position can be set between the multimodal output device 2 and the virtual digital image presentation device 3. In addition, the multimodal data acquisition device can also include microphones, and the number and position of the microphones can be set according to actual conditions. As shown in FIG. Figure 2 The microphone can be arranged near the multimodal output device 2 and the autistic child, so as to detect the sound made by the child as much as possible. In this example, the microphone can be a microphone integrated in the multimodal output device 2 and the virtual digital image presentation device 3, or an additional head-mounted microphone.

[0028] In one embodiment, a relative position of the autistic child relative to at least one of a multimodal output device and a virtual digital image presentation device is determined based on at least one of a first camera and a second camera.

[0029] In this embodiment, at least one of the first camera and the second camera can also be used to determine the relative position of the autistic child relative to one or all of the multimodal output device or the virtual digital image presentation device. Specifically, the relative distance of the autistic child can be calculated by a preset algorithm such as a binocular stereo algorithm. It can be used to prompt the autistic child that the position is too close, too far or out of the device during the intervention process, so as to avoid the above problems affecting the intervention rehabilitation effect. In the current digital human application, the digital human is pre-set and cannot reflect the communication status with the autistic child in real time. Therefore, in this embodiment, by detecting the relative position of the autistic child, the output items of the multimodal output device or the virtual digital image output by the virtual digital image presentation device can be adjusted in real time, so that the output result can be adjusted to face the autistic child. In this way, when intervening in the autistic child, the multimodal output device or the virtual digital image can always surround the autistic child, so that the autistic child can always feel the kind and concerned eyes of the virtual digital image, thereby achieving the desired intervention effect. Those skilled in the art should understand that "real time" here refers to the time range that the autistic child can accept, and the adjustment within this time range will not have a significant impact on the autistic child.

[0030] In one embodiment, the multimodal data acquisition device further includes a third camera, which collects the position and behavior state of the virtual digital image. In some emergencies, due to errors in device settings, the output virtual digital image deviates. This deviation often causes a larger image to autistic children and may even cause the intervention effect to deteriorate. Therefore, in this embodiment, the virtual digital image can be monitored from the perspective of a bystander through the third camera. When a deviation in the virtual digital image is detected, the virtual digital image presentation device can be adjusted in time to avoid the negative impact of the intervention.

[0031] Here, cameras are divided into a first camera, a second camera, a third camera, etc. according to their functions. However, those skilled in the art should understand that with the development of technology, one or more of these functions can be implemented by one camera. Two or more of the first camera, the second camera, and the third camera here can also be implemented in one physical camera. In addition, one of these functions can also be implemented by multiple cameras. In this case, any one of the first camera, the second camera, and the third camera can also include two or more physical cameras.

[0032] In this example, the multimodal data acquisition device also includes a camera for acquiring the position and behavior status of the virtual digital image ( Figure 2(not shown). The camera can be set opposite to the virtual digital image presentation device. For example, it can be worn on the autistic child together with the microphone. During the intervention process, the position of the virtual digital image is monitored in real time to see if it is wrong, or whether the behavior state is displayed incorrectly, so as to facilitate timely discovery and correction when problems arise, and prevent the virtual digital image from causing trouble to the autistic child and affecting the intervention effect.

[0033] The multimodal output device 2 is used to output at least one output modality information of images, sounds, touch, and smells.

[0034] For example, Figure 2 As shown, the multimodal output device can be a device with image display capabilities, such as a tablet computer or a display screen, etc. The multimodal output device can also have touch or other interactive capabilities at the same time, and can display the following images, options, questions, etc. during the intervention process. At the same time, the multimodal output device can also include a speaker for playing audio such as intervention project instructions, voice prompts, interactive feedback, etc. to enhance the interactive effect. The multimodal output device can also output information through touch, for example, it can vibrate or generate some nerve stimulation signals that cause children to have tactile illusions. In addition, the multimodal output device can also have the function of odor output. For example, various preset odor sources can be pre-stored in the device and output during the intervention process.

[0035] The virtual digital image presentation device 3 is used to present a virtual digital image that can interact with the autistic child.

[0036] Autism rehabilitation intervention is usually one-on-one, high-frequency, repetitive, and effective for a certain period of time. However, autistic children generally have difficulty staying in their seats, their attention is easily distracted, and they often leave their seats and wander around. On the other hand, autistic children are highly dependent on family affection. In practice, digital game-based rehabilitation intervention methods that lack family-style companionship and interaction are difficult to attract autistic children to conduct long-term intervention training, their compliance is limited, and the rehabilitation intervention effect is lower than expected. Here, virtual digital images can be customized for individual family affection based on the characteristics of autistic children's preference for caregivers. During the intervention process, professional "rehabilitation therapist"-style family companionship, guidance, interaction, and encouragement can improve compliance, ensure effective intervention training time, and achieve better rehabilitation intervention effects. Especially for social communication rehabilitation intervention projects, the expressive ability of two-dimensional game scene design is limited, and the roles and interactions are restricted, but the virtual digital image design can solve this problem well. For example, the rehabilitation intervention training of "making requests with eye contact" requires lonely children to have eye contact with the requested party while making requests with words; and the rehabilitation intervention training of "children can make requests with their eyes (for example: the child looks at the adult, and the adult gives)".

[0037] In one example, the virtual digital image includes at least one of a virtual digital human displayed on a display, a three-dimensional virtual digital human, a holographic virtual digital human, and a robot.

[0038] For example, the virtual digital image presentation device 3 may be a display device, a holographic projection device or a robot device. For example, it may be set according to the virtual digital image in the system. For example, when the virtual digital image is a virtual digital person displayed on a display, Figure 2 As shown, the virtual digital image presentation device may be a display device. When the virtual digital image is a three-dimensional virtual digital human, the corresponding virtual digital image presentation device may be a three-dimensional imaging device. The same is true for holographic virtual digital humans and robot virtual digital images.

[0039] In one example, the virtual digital image presentation device further includes a virtual digital image presentation unit. The virtual digital image presentation unit performs the following processing: obtaining an actual face image of a foster parent of an autistic child; obtaining a personalized texture image of the autistic child, wherein the personalized texture image is selected for the autistic child; the virtual digital image presentation unit obtains input modal data from a multimodal data acquisition device, wherein the virtual digital image presentation unit combines the personalized texture image with the actual face image based on at least one of the orientation and the gaze direction. The personalized texture image and the actual face image are combined to generate a personalized virtual digital image. Here, the input modal data collected by the multimodal data acquisition device can be passed to the virtual digital image presentation unit through multiple preprocessing and / or via multiple other units or modules. The input modal data obtained by the virtual digital image presentation unit can be the original input modal data collected by the multimodal data acquisition device, or it can be processed input modal data.

[0040] In this example, the actual facial image of the foster person of the autistic child can be pre-collected and stored by a multimodal data acquisition device. The virtual digital image presentation unit can obtain the actual facial image of the foster person of the autistic child. At the same time, the personalized texture image selected by the autistic child can also be obtained. The personalized texture image can be, for example, personalized eyebrows, beards, facial lines or cartoon images, etc. The personalized texture image can be presented by the multimodal output device 2 or the virtual digital image presentation device 3 for the autistic child to choose. After that, modeling can be performed based on the actual facial image of the foster person to establish a model of a virtual digital person, and then the personalized texture image is combined with the facial image to generate a personalized virtual digital image. The reaction state of the autistic child to different personalized texture images can be detected by a multimodal data acquisition device, and the personalized texture image can be adjusted based on the reaction state. For example, when an autistic child shows an anxious mood in the face of a personalized virtual digital image based on a personalized texture image, the personalized texture image can be replaced with another personalized texture image.

[0041] In one example, the virtual digital image presentation unit can obtain at least one of the orientation or gaze direction of the autistic child input from the multimodal data acquisition device, and combine the personalized texture image with the face image according to the orientation or gaze position. For example, after the autistic child selects the personalized texture image, the relative position, facing angle, and visual focus direction of the autistic child and the virtual digital image can be determined based on the orientation of the autistic child to determine the position or direction in which the autistic child wants to combine the personalized texture image with the face image. Similarly, the position or direction in which the personalized texture image is combined with the face image can also be determined based on the gaze direction.

[0042] In one example, a virtual digital image presentation unit combines a personalized texture image with an actual face image by the following processing: obtaining an actual face feature point array based on the actual face image; obtaining a personalized feature point array based on the personalized texture image; adjusting the directionality of the actual face feature point array and the personalized feature point array in real time based on at least one of the orientation and gaze direction of the autistic child; and generating a personalized virtual digital image using the adjusted actual face feature point array and the personalized feature point array.

[0043] For example, after obtaining an actual face image, a basic 3D model of the face image can be established, and the face in the image can be parameterized. The parameters of the model can be estimated from the input image or video through an algorithm, so that the obtained model is as close as possible to the person in the photo. In this example, the 3DMM parameter estimation based on deep learning can be adopted, and an unsupervised learning method can be used to avoid the need for a large amount of labeled data to train the network, and a deep neural network can be trained to predict the 3DMM parameters, that is, to obtain the feature point array of the actual face. Similarly, the feature point array of the personalized texture can be obtained in advance through the algorithm and stored in the system, or it can be obtained in the above manner. After obtaining the feature point array of the actual face and the feature point array of the personalized texture, the directionality of the feature focus array can be adjusted based on one of the orientation and gaze direction of the autistic child, and finally modeling can be performed based on these feature point arrays to construct the image of a virtual digital person.

[0044] In one example, the virtual digital image presentation unit also obtains the bone feature data of the foster parent of the autistic child based on the actual face image, and adjusts the personalized texture image of the autistic child based on the bone feature data.

[0045] In this example, in order to restore the actual image of the foster parent as much as possible and avoid distortion caused by the personalized texture image, the virtual digital image presentation unit can also determine the bone feature data of the child foster parent based on the actual face image and the pre-trained algorithm or model, which can be the facial bone feature function data, and adjust the personalized texture image based on the data, such as adjusting the parameters of the feature point array of the personalized texture image, so that the virtual digital image finally generated is similar to the foster parent.

[0046] In one embodiment, the multimodal data acquisition device also acquires voice information of the foster parent of the autistic child. The multimodal fusion module generates basic voice feature data based on the voice information. The multimodal fusion module determines the current state data of the autistic child in real time based on the input modal data of the autistic child acquired by the multimodal data acquisition device. The multimodal fusion module adjusts the basic voice feature data based on the current state data to generate output voice. The virtual digital image presentation unit adjusts the personalized virtual digital image based on the current state data.

[0047] For example, the virtual digital image presentation unit adjusts the personalized virtual digital image based on the current state data by the following processing: determining the facial muscle change characteristics of the personalized virtual digital image based on the adjusted basic voice feature data; and generating a dynamic personalized virtual digital image based on the facial muscle change characteristics.

[0048] In one example, the virtual digital image presentation unit determines the current emotion of the autistic child based on the current state data, obtains a personalized texture image corresponding to the current emotion, and combines the personalized texture image corresponding to the current emotion with the adjusted actual face image to generate a personalized virtual digital image.

[0049] In one embodiment, the virtual digital image rendering unit divides the actual face image and the personalized texture image into multiple blocks respectively. The multimodal data acquisition device also obtains ambient light parameters, and the ambient light parameters include the position and brightness of the ambient light source. The virtual digital image rendering unit adjusts the brightness of the multiple blocks based on the ambient light parameters. The virtual digital image rendering unit combines the adjusted personalized texture image with the adjusted actual face image to generate a personalized virtual digital image.

[0050] In one embodiment, the multimodal data acquisition device obtains input information about the virtual digital image. The virtual digital image presentation unit generates a personalized texture image based on the input information about the virtual digital image.

[0051] The multimodal fusion module 4 is used to control the virtual digital image to perform corresponding actions based on the input modal data through the virtual digital image presentation device, and generate output modal information through the multimodal output device.

[0052] In this example, the multimodal fusion module can be the central processing module of the system, which can analyze and process based on the input modal data and control each device or module to perform corresponding tasks.

[0053] In an example, Figure 3 As shown, the system also includes an initial data collection module 5 for collecting relevant information of autistic children; a digital assessment module 6 for determining pre-assessment data of the ability level of autistic children based on the relevant information, so as to determine the ability baseline of autistic children in different fields, thereby determining the individualized ability characteristics of autistic children in different fields; a digital rehabilitation plan generation module 7 for generating a phased hierarchical progressive rehabilitation intervention plan suitable for the individual ability development of the autistic child based on the ability baseline and the individualized ability characteristics; a digital rehabilitation training module 8 for pushing digital rehabilitation projects that meet the common characteristics of similar autistic children according to the phased hierarchical progressive rehabilitation intervention plan; wherein the multimodal fusion module controls the multimodal output device to generate output modal information corresponding to the digital rehabilitation project and / or controls the virtual digital image of the virtual digital image presentation device to perform actions corresponding to the digital rehabilitation project based on the digital rehabilitation project.

[0054] In this example, the system also includes an initial data collection module 5, which can be in the form of a software application or client for collecting information about autistic children, such as name, gender, date of birth, place of birth, behavioral characteristics of autism, etc. At the same time, necessary information about the parents and foster parents of autistic children can also be collected. In this example, some information can be collected in the form of a questionnaire, and corresponding questions are asked through the questionnaire to obtain it. For example, the question can be whether the child can understand the functions, categories, and characteristics of at least 30 items. Or whether the child can sort according to the rules. Based on the collected information, the basic situation of the autistic child can be obtained so that the intervention training program suitable for the child's ability can be matched later.

[0055] In this example, the system also includes a data evaluation module 6, which can determine the data related to the child's ability level in various aspects based on the collected relevant information of the autistic child. For example, after collecting the basic relevant information of the autistic child, the child's initial ability is preliminarily determined, and the evaluation items in various fields are pushed based on the initial ability. The evaluation items can be provided to the child in the form of digital games. According to the completion of the autistic child, the ability data of the autistic child in various skill areas are determined. Such as vocabulary distinction, sensory response, number concept, Chinese characters and pinyin, etc. The ability baseline of the autistic child in different fields is determined based on the ability data. The baseline can be a preset ability baseline in the system, that is, the ability level of each aspect, such as the ability level of vocabulary distinction is level three, the sensory response is level four, etc. Ability characteristics, such as which aspects are strong, which aspects are weak, and the characteristics of the weak aspects, such as slow response, high error rate, etc.

[0056] The system also includes a digital rehabilitation plan generation module 7, which can formulate a corresponding training plan based on the information and ability baseline of the autistic children provided by the above modules, and the individual ability characteristics. The formulation of this plan can refer to the age, gender, behavioral characteristics, evaluation results, and rehabilitation plan content of similar autistic children in history, covering vocabulary discrimination, perceptual response, academic ability, social rules, social skills, life skills, life skills, ability baselines and characteristics, etc. Based on these characteristics, specific projects, learning cycles, learning intensity, etc. of intervention training are formulated.

[0057] The system also includes a digital rehabilitation training module 8, which can push corresponding digital rehabilitation projects according to the phased hierarchical progressive rehabilitation intervention plan. The digital rehabilitation project can be an intervention training project for a certain ability of autistic children, such as covering vocabulary discrimination, sensory response, academic ability, social rules, social skills, life skills, life skills, etc. These training projects can be embodied in the form of digital games.

[0058] In this example, the multimodal fusion module controls the multimodal output device to generate output modality information corresponding to the digital rehabilitation project and / or controls the virtual digital image of the virtual digital image presentation device to perform actions corresponding to the digital rehabilitation project based on the digital rehabilitation project.

[0059] For example, when the digital rehabilitation project is a quiz game, the multimodal fusion module can control the multimodal output device to generate information corresponding to the quiz game, such as question information or option information, etc., which is displayed in the multimodal output device, and can also control the virtual digital image to perform corresponding actions, such as reading the question aloud, indicating the options, etc. Or in some action demonstration games, the virtual digital image can also be controlled to perform corresponding actions so that the autistic child can imitate or answer the corresponding questions.

[0060] In one embodiment, the rehabilitation plan generation module updates the staged hierarchical progressive rehabilitation intervention plan in real time based on the input modal data from the multimodal data acquisition device. Here, the input modal data collected by the multimodal data acquisition device can be passed to the rehabilitation plan generation module through multiple preprocessing and / or via multiple other units or modules. The input modal data based on the rehabilitation plan generation module can be the original input modal data collected by the multimodal data acquisition device, or it can be processed input modal data. The digital rehabilitation training module pushes the updated digital rehabilitation project based on the updated staged hierarchical progressive rehabilitation intervention plan. Based on the updated digital rehabilitation project, the multimodal fusion module controls the multimodal output device to generate output modal information corresponding to the updated digital rehabilitation project and / or controls the virtual digital image of the virtual digital image presentation device to perform actions corresponding to the updated digital rehabilitation project.

[0061] In one embodiment, the time interval between completing the digital rehabilitation project and starting the updated digital rehabilitation project is greater than 3 seconds and less than 2 minutes. Since the feelings and reactions of autistic children are different from those of ordinary children, the inventors found in actual rehabilitation training that for the intervention system using virtual digital images, the interval between projects may affect autistic children and reduce the intervention effect. Therefore, according to practical experience, setting the time interval in the range of 3 seconds to 2 minutes can not only allow autistic children to feel the division between projects, but also will not have a greater impact on the patience of autistic children, thereby not affecting the intervention effect.

[0062] In this example, during the intervention training, the multimodal data acquisition device can continue to collect the input of autistic children during the intervention process, such as facial expressions, emotions or gazes, so as to obtain feedback on the intervention training. The rehabilitation training module can adjust the training plan based on the feedback from autistic children. For example, when the feedback is that autistic children think the training is difficult, the difficulty of subsequent training projects can be reduced. When the feedback is that the training is simple, the difficulty of subsequent training projects can be increased. Then push the updated digital rehabilitation project. The multimodal fusion module can control the multimodal output device to generate or the virtual digital image presentation device to present accordingly based on the updated project.

[0063] In this process, the digital evaluation module can determine the difficulty level of autistic children when performing digital rehabilitation projects based on the input modal data collected by the multimodal data acquisition device. The multimodal fusion module controls the virtual digital image of the virtual digital image presentation device to perform demonstration actions corresponding to the digital rehabilitation project based on the difficulty level. In other words, the digital evaluation module can determine whether it is difficult for autistic children to perform the project so that the rehabilitation plan generation module can update the rehabilitation plan. At the same time, the multimodal fusion module can also control the virtual digital image to perform demonstration actions according to the difficulty level. For example, when autistic children think that training is more difficult, the delicate digital image can be controlled to make prompt actions, demonstration actions, or audio related to the rehabilitation project.

[0064] In one embodiment, the multimodal fusion module outputs multiple objects corresponding to the digital rehabilitation project through a multimodal output device, and instructs the autistic child to select one of the multiple objects through the multimodal output device, wherein the multimodal fusion module controls the multimodal output device to highlight the object corresponding to the visual attention area of ​​the autistic child.

[0065] For example, in some digital rehabilitation projects and some question-and-answer games, the multimodal fusion module can control the multimodal output device to display multiple options. At the same time, it can also determine which object the autistic child has selected based on the collected visual attention area of ​​the autistic child. After the determination, it can also be highlighted through the multimodal output device to give feedback on the autistic child's choice.

[0066] In one embodiment, a multimodal data acquisition device acquires an autistic child's visual attention area, wherein a multimodal fusion module controls the eyes of a virtual digital image of a virtual digital image presentation device to make eye contact with the autistic child and follow the gaze movement of the autistic child based on the autistic child's visual attention area.

[0067] During the intervention training, such as some eye contact training items, the multimodal data acquisition equipment collects the visual attention area of ​​the autistic children. At the same time, the multimodal fusion module can adjust the eyes of the virtual digital image according to the gaze position of the autistic children, so that it moves with the gaze position of the autistic children and makes eye contact with the autistic children, etc. This makes the virtual image more vivid and enhances the sense of reality, thereby improving the training effect.

[0068] Figure 4 FIG. 1 shows a schematic computing device that can be used to implement a virtual digital image autism rehabilitation intervention system according to an embodiment. In the virtual digital image autism rehabilitation intervention system, one or more such Figure 4 The computing device shown.

[0069] like Figure 4 As shown, computing device 300 includes a processor 302 and a readable storage medium 304 .

[0070] The computing device 300 may further include a display screen 310, a user interface 312, a camera 314, an audio / video interface 316, a sensor 318, a communication component 320, etc. In addition, the computing device 300 may further include a power management chip 306 and a battery 308, etc. The computing device 300 may be various smart devices, etc.

[0071] The processor 302 may be any of a variety of processors. The readable storage medium 304 may store underlying software, system software, application software, data, etc. required for the operation of the computing device 300. The readable storage medium 304 may include various forms of memory, such as ROM, RAM, Flash, etc.

[0072] The display screen 310 may be a liquid crystal display screen, an OLED display screen, etc. In one example, the display screen 310 may be a touch screen. The user may perform input operations through the display screen 310. In addition, the user may also perform fingerprint recognition through the touch screen.

[0073] The user interface 312 may include a USB interface, a lightning interface, a keyboard, and the like.

[0074] Camera 314 can be a single camera or multiple cameras. In addition, camera 314 can be used for user's face recognition.

[0075] The audio / video interface 316 may include, for example, a speaker interface, a microphone interface, a video transmission interface such as HDMI, and the like.

[0076] The sensor 318 may include, for example, a gyroscope, an accelerometer, a temperature sensor, a humidity sensor, a pressure sensor, etc. For example, the environment around the computing device may be determined by the sensor.

[0077] The communication component 320 may include, for example, a WiFi communication component, a Bluetooth communication component, a 3G, 4G and 5G communication component, etc. Through the communication component 320, the computing device 300 may be arranged in a network.

[0078] The power management chip 306 can be used to manage the power input to the computing device 300, and can also manage the battery 308 to ensure greater utilization efficiency. The battery 308 is, for example, a lithium-ion battery.

[0079] Figure 4 The computing device shown is merely illustrative and is in no way intended to limit the embodiments herein, their application, or uses.

[0080] Figure 5 FIG. 1 shows a schematic network in which a virtual digital image autism rehabilitation intervention system can be arranged according to an embodiment. The virtual digital image autism rehabilitation intervention system can be arranged in Figure 4 In the network shown.

[0081] Figure 5 4 shows a plurality of terminal devices 31, 32, 33 and a communication network 40. A plurality of servers 41, 42 may be provided in the network 40. Each of the terminal devices 31, 32, 33 and the servers 41, 42 may be, for example, Figure 4 The computing device shown. A virtual digital image autism rehabilitation intervention system according to one embodiment includes at least one processor and at least one readable storage medium. The at least one processor and at least one readable storage medium may be distributed in terminal devices 31, 32, 33 and servers 41, 42. The readable storage medium stores executable instructions. When the processor executes the executable instructions, the executable instructions enable the processor to implement the steps in the virtual digital image autism rehabilitation intervention system according to the embodiment.

[0082] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0083] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0084] The embodiments of the present disclosure may be systems, methods and / or computer program products. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the embodiments of the present disclosure.

[0085] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0086] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0087] The computer program instructions for performing the operation of the embodiments of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of a computer-readable program instruction, and the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the embodiments of the present disclosure.

[0088] Various aspects of the embodiments of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0089] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0090] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0091] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of an instruction, and a part of a module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0092] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A virtual digital image autism rehabilitation intervention system, comprising: A multimodal data acquisition device for acquiring at least one input modality data of eye movement, facial expression, body movement and sound of an autistic child; A multimodal output device, used to output at least one output modality information of images, sounds, touch, and smells; A virtual digital image presentation device, used to present a virtual digital image that can interact with an autistic child, wherein the virtual digital image presentation device is a display device, a holographic projection device, or a robotic device; A multimodal fusion module, for controlling the virtual digital image to perform corresponding actions through a virtual digital image presentation device based on input modal data, and generating output modal information through a multimodal output device; The initial data collection module is used to collect relevant information about children with autism; A digital assessment module is used to determine the pre-assessment data of the ability level of the autistic child based on the relevant information, and based on the pre-assessment data of the ability level, push an adapted subset of streamlined assessment game items through a deep learning model to determine the ability baseline of the autistic child in different fields, thereby determining the individualized ability characteristics of the autistic child in different fields; A digital rehabilitation plan generation module is used to generate a phased hierarchical progressive rehabilitation intervention plan suitable for the individual ability development of the autistic child based on the ability baseline and the individualized ability characteristics; The digital rehabilitation training module is used to push digital rehabilitation projects that meet the common characteristics of similar autistic children according to the staged hierarchical progressive rehabilitation intervention plan. The method comprises: determining a relative position of the autistic child relative to at least one of the multimodal output device and the virtual digital image presentation device based on at least one of the first camera and the second camera, adjusting the output item of the multimodal output device or the virtual digital image output by the virtual digital image presentation device in real time, so that the output result can be adjusted to face the autistic child. The multimodal fusion module controls the multimodal output device to generate output modal information corresponding to the digital rehabilitation project and / or controls the virtual digital image of the virtual digital image presentation device to perform actions corresponding to the digital rehabilitation project based on the digital rehabilitation project.

2. The virtual digital image autism rehabilitation intervention system according to claim 1, wherein: The virtual digital image includes at least one of a virtual digital human displayed on a display, a three-dimensional virtual digital human, a holographic virtual digital human, and a robot.

3. The virtual digital image autism rehabilitation intervention system according to claim 1, wherein: The virtual digital image presentation device also includes a virtual digital image presentation unit. The virtual digital image presentation unit performs the following processing: Obtaining actual facial images of foster parents of autistic children; Acquire a personalized texture image of an autistic child, wherein the personalized texture image is selected for the autistic child; Combine personalized texture images with actual face images to generate personalized virtual digital images. The virtual digital image presentation unit obtains input modality data from the multimodal data acquisition device. The virtual digital image presentation unit determines at least one of the position and the gaze direction of the autistic child based on the input modality data. The virtual digital image presentation unit combines the personalized texture image with the actual face image based on at least one of the orientation and the gaze direction to generate a personalized virtual digital image.

4. The virtual digital image autism rehabilitation intervention system according to claim 3, wherein: The virtual digital image rendering unit combines the personalized texture image with the actual face image through the following processing: Acquire an array of actual facial feature points based on an actual facial image; Acquire a personalized feature point array based on the personalized texture image; Based on at least one of the orientation and gaze direction of the autistic child, adjusting the directionality of the actual facial feature point array and the personalized feature point array in real time; A personalized virtual digital image is generated using the adjusted actual facial feature point array and the personalized feature point array.

5. The virtual digital image autism rehabilitation intervention system according to claim 1, wherein: The rehabilitation plan generation module updates the phased hierarchical progressive rehabilitation intervention plan in real time based on the input modal data from the multimodal data acquisition device. Among them, the digital rehabilitation training module pushes updated digital rehabilitation projects based on the updated phased hierarchical progressive rehabilitation intervention plan. Among them, the multimodal fusion module controls the multimodal output device to generate output modal information corresponding to the updated digital rehabilitation project and / or controls the virtual digital image of the virtual digital image presentation device to perform actions corresponding to the updated digital rehabilitation project based on the updated digital rehabilitation project.

6. The virtual digital image autism rehabilitation intervention system according to claim 1, wherein: The multimodal data acquisition device includes a first camera, a second camera and a microphone. The first camera is used to collect the movements and facial expressions of autistic children, the second camera is used to collect the eye movements of autistic children, and the microphone is used to collect the voices of autistic children. Wherein, the first camera is arranged above the virtual digital image presentation device, Wherein, the second camera is arranged between the multimodal output device and the virtual digital image presentation device.

7. The virtual digital image autism rehabilitation intervention system according to claim 1, wherein: The multimodal data acquisition device also includes a third camera, which acquires the position and behavior status of the virtual digital image.

8. The virtual digital image autism rehabilitation intervention system according to claim 1, wherein: The multimodal fusion module outputs multiple objects corresponding to the digital rehabilitation items through a multimodal output device, and instructs the autistic child to select one of the multiple objects through the multimodal output device. Among them, the multimodal fusion module controls the multimodal output device to highlight the objects corresponding to the visual attention area of ​​the autistic child.

Images