Progressive training method and device pre-adaptive to real scene

Abstract

The invention relates to the technical field of electroencephalogram signal monitoring and data processing, in particular to a progressive training method and device pre-adaptive to a real scene, and the method comprises the steps: obtaining scene information of the real scene; performing element hierarchy splitting on the scene information, performing progressive hierarchy modeling according to elements of different hierarchies after splitting, and constructing a multi-hierarchy virtual scene simulating real environments of different hierarchies; and obtaining heterogeneous feature expression when the user performs pre-adaptation in the multi-level virtual scene, dynamically switching the levels of the multi-level virtual scene according to the heterogeneous feature expression, and performing progressive virtual sensory stimulation training pre-adaptive to the real scene in the multi-level virtual scene of the switched levels. By constructing the multi-level virtual scene strongly associated with the real scene, the problem that existing training is disjointed with the real scene is solved, and different training stages can be adapted through the multi-level progressive scene version design, so that the training of the user is more flexible.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electroencephalogram signal monitoring and data processing, and particularly relates to a progressive training method and device pre-adapted to a real scene. BACKGROUND

[0002] Some children with autism spectrum disorder (ASD) have abnormal sensory processing, and their sensory stimulation tolerance threshold is significantly lower than that of normal users. They are abnormally sensitive to sensory stimulation such as sound and light. An environment that is easily adapted to by normal users, such as light flashing and sudden noise increase, may cause stress loss behavior of ASD children, affecting their safety and social adaptability when going out.

[0003] At present, there are still problems such as scene disconnection from real environment, lack of targeted stress source design, and lagging risk warning in the intervention means for the scene adaptation of some ASD children. SUMMARY

[0004] The purpose of the present application is to provide a progressive training method and device pre-adapted to a real scene. The progressive multi-level virtual scene constructed is generated based on real environment data, and the stress source is strongly associated with the scene, so that the reality and adaptability of the virtual scene are significantly improved, solving the problem that the existing training is disconnected from the real scene, and through the multi-level progressive scene version design, different training stages can also be adapted, making the user's training more flexible.

[0005] In some embodiments, the present application provides a progressive training method pre-adapted to a real scene, which comprises: acquiring scene information of a real scene; performing element level splitting on the scene information, respectively performing progressive level modeling according to the elements of different levels after splitting, and constructing a multi-level virtual scene simulating different levels of real environment; acquiring a heterogeneous feature expression of a user when pre-adapting in the multi-level virtual scene, dynamically switching the level of the multi-level virtual scene according to the heterogeneous feature expression, and performing progressive virtual sensory stimulation training pre-adapted to the real scene in the multi-level virtual scene of the switched level.

[0006] In some embodiments, the acquisition of the heterogeneous feature expression of the user when pre-adapting in the multi-level virtual scene and the dynamic switching of the level of the multi-level virtual scene according to the heterogeneous feature expression comprise: acquiring a heterogeneous feature expression of a user when pre-adapting in a current level virtual scene; determining the adaptability of the user in the current level virtual scene according to the sensing parameters and the interaction parameters in the heterogeneous feature expression, and dynamically switching the level of the virtual environment in which the user is located according to the adaptability.

[0007] In some embodiments, the determining the adaptability of the user in the current hierarchical virtual scene according to the heterogeneous feature expression of the sensing parameters and the interaction parameters comprises: determining an environment adaptation degree corresponding to the physiological stability of the user according to the sensing parameters and / or the interaction parameters of the user; determining a stress reaction degree corresponding to the behavior performance of the user according to the sensing parameters and / or the interaction parameters of the user; determining a task completion degree corresponding to the task completion quality of the user according to the sensing parameters and / or the interaction parameters of the user; determining an index stability degree corresponding to the stability verification of the user according to the sensing parameters and / or the interaction parameters of the user; determining a care evaluation degree corresponding to the caregiver cooperation of the user according to the sensing parameters and / or the interaction parameters of the user; and determining the adaptability of the user in the current hierarchical virtual scene according to at least one of the environment adaptation degree, the stress reaction degree, the task completion degree, the index stability degree, and the care evaluation degree.

[0008] In some embodiments, the determining the adaptability of the user in the current hierarchical virtual scene according to the heterogeneous feature expression of the sensing parameters and the interaction parameters, and the determining the hierarchical adaptability of the user according to the adaptability comprises: if the adaptability of the user in the current hierarchical virtual scene determined according to the sensing parameters and / or the interaction parameters meets a switching condition, performing switching from the current hierarchical virtual scene to a next hierarchical virtual scene, so that the user performs the virtual sensory stimulation training pre-adapted to the real scene in the next hierarchical virtual scene, and iteratively adjusting the hierarchy of the virtual scene according to the adaptability of the user until the user completes the virtual sensory stimulation training pre-adapted to the real scene in a highest hierarchical virtual scene or terminates the training; if the adaptability of the user in the current hierarchical virtual scene determined according to the sensing parameters and / or the interaction parameters does not meet the switching condition, continuing to determine whether the user has a rollback record that has reached a preset upper limit of a number of times in the current hierarchical virtual scene, if yes, performing a risk warning or terminating the training, if not, making the user continue to perform the virtual sensory stimulation training pre-adapted to the real scene in the current hierarchical virtual scene, and continuing to iteratively adjust the hierarchy of the virtual scene according to the adaptability of the user until the user completes the virtual sensory stimulation training pre-adapted to the real scene in the highest hierarchical virtual scene or terminates the training.

[0009] In some embodiments, the element-level splitting of the scene information, the progressive hierarchical modeling of different levels of elements, and the construction of a multi-level virtual scene simulating different levels of real environment, comprise: element splitting of environment elements and social context elements in the scene information according to levels to obtain environment elements and social context elements of different levels; and construction of virtual scenes of corresponding levels by using the environment elements and the social context elements of different levels respectively; wherein the virtual scenes of different levels are used to simulate the real scene progressively, and the virtual scene continuously approaches the real scene as the progressive training proceeds.

[0010] In some embodiments, the construction of virtual scenes of corresponding levels by using the environment elements and the social context elements of different levels respectively, comprises: when a user first enters a virtual scene of a key level for virtual sensory stimulation training pre-adapted to the real scene, no sensory hypersensitivity item of the user is set in the environment elements of the key level for virtual scene construction of the key level; when the user is not first time in the virtual scene of the key level for virtual sensory stimulation training pre-adapted to the real scene, the sensory hypersensitivity item of the user is dynamically set in the environment elements of the key level for virtual scene construction of the key level, wherein the key level is a next highest level.

[0011] In some embodiments, the calculation of the physiological stability feature interval in the environmental adaptation degree comprises: obtaining a first parameter of the user in the current scene according to a safe environment standard, the first parameter comprising an environmental comfort reference value, obtaining a scene risk correction weight corresponding to the current scene category, adjusting the first parameter according to a scene risk correction strategy, a scene risk correction weight, and a sensitivity correction strategy to obtain an environmental critical condition; obtaining a physiological stability feature of the user in the current scene category and the current activity state, correcting the physiological stability feature by using the scene risk correction strategy to obtain a physiological stability feature interval of the user; collecting an electroencephalogram signal of the user, calculating an adaptability correction value of the user to the training device according to the electroencephalogram signal, and correcting the physiological stability feature interval according to the adaptability correction value.

[0012] In some embodiments, the collection of the electroencephalogram signal of the user, the calculation of the adaptability correction value of the user to the training device according to the electroencephalogram signal, and the correction of the physiological stability feature interval according to the adaptability correction value, comprise: collecting a resting-state electroencephalogram signal of the user without wearing the training device to calculate a first feature parameter group; collecting a resting-state electroencephalogram signal of the user after wearing the training device to calculate a second feature parameter group; determining the adaptability correction value of the user to the training device according to the difference between the first feature parameter group and the second feature parameter group; determining a scaling factor according to the adaptability correction value, and correcting the physiological stability feature interval according to the scaling factor.

[0013] In some embodiments, the method further comprises: if at least one of the physiological stability feature interval, the stress reaction degree, the task completion degree, the index stability degree, and the care evaluation degree triggers a high-risk threshold of the corresponding level virtual environment, terminating the virtual sensory stimulation training pre-adapted to the real scene.

[0014] In some embodiments, the present application also provides a progressive training device pre-adapted to a real scene, comprising: a modeling module configured to acquire scene information of a real scene, perform element level splitting on the scene information, perform progressive level modeling on different levels of elements after splitting, and construct a multi-level virtual scene simulating different levels of real environments; a multi-modal data synchronous acquisition module configured to acquire heterogeneous feature expressions of a user when pre-adapting in the multi-level virtual scene while wearing a training device; a multi-modal feature fusion analysis module configured to calculate sensory parameters and / or interaction parameters of the user according to the heterogeneous feature expressions; and a training interaction module configured to dynamically switch levels of the multi-level virtual scene according to the sensory parameters and / or interaction parameters of the user, and perform pre-adaptation training in the multi-level virtual scene of the switched level.

[0015] In the above embodiments, a progressive training method and device pre-adapted to a real scene are provided, which is a modeling method based on a real scene to perform virtual scene modeling, so that the user can perform virtual sensory stimulation training in the constructed virtual scene simulating a real scene. Different from the existing virtual training based on a preset virtual scene, the dynamic virtual modeling based on real environment information provided by the present application dynamically constructs a highly realistic virtual training environment according to real-time or near real-time environment information of the destination, and introduces common dynamic stressors that may cause user discomfort in the environment category, so that the user can perform progressive virtual sensory stimulation training pre-adapted to a real scene in the constructed virtual scene. And the multi-level progressive scene version design can also adapt to different training stages, so that the user's training is more flexible. BRIEF DESCRIPTION OF DRAWINGS

[0016] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings in which:

[0017] Figure 1 is a flowchart of a progressive training pre-adapted to a real scene provided by one of the embodiments of the present application;

[0018] Figure 2 is a schematic diagram of different virtual scene stage levels and switching between stage levels provided by one of the embodiments of the present application;

[0019] Figure 3is a personalized threshold adaptation and calibration flowchart provided by one of the embodiments of the present application;

[0020] Figure 4 is a personalized threshold adaptation and calibration flowchart provided by another embodiment of the present application;

[0021] Figure 5 is a flowchart of virtual scene different level scene switching provided by one of the embodiments of the present application;

[0022] Figure 6 is a flowchart based on hierarchical modeling and progressive training provided by one of the embodiments of the present application;

[0023] Figure 7 is a progressive training device module schematic diagram pre-adapted to a real scene provided by one of the embodiments of the present application;

[0024] Figure 8 is a structural schematic diagram of a computer device provided by one of the embodiments of the present application. DETAILED DESCRIPTION

[0025] In order to make the purposes, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0026] The technical solutions of various embodiments of the present application can be combined with each other, but must be based on the fact that a person having ordinary skill in the art can realize it, when the combination of technical solutions appears to be contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor is it within the scope of protection claimed by the present application.

[0027] The present solution is not directly aimed at obtaining disease diagnosis results or health conditions, but only a method for processing scene information and user sensor data and the like, and all steps are information processing methods implemented by computers and the like.

[0028] It should be fully understood that the user information (including but not limited to user physiological information, user personal information, etc.) involved in the present application is information and data authorized by the user or fully authorized by all parties, and the use of user information should follow the privacy policies and practices that are generally considered to meet or exceed the industry for maintaining user privacy, the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards, and provide corresponding operation portal for user to choose authorization or refusal.

[0029] In order to make the above purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below in combination with the drawings.

[0030] See Figure 1 This application provides a progressive training method for pre-adapting to real-world scenarios, comprising:

[0031] Step S101: Obtain scene information of the real scene.

[0032] Step S102: The scene information is split into elements at different levels, and progressive hierarchical modeling is performed according to the elements at different levels after splitting to construct a multi-level virtual scene that simulates different levels of real environment.

[0033] Step S103: Obtain the heterogeneous feature expression of the user during pre-adaptation in the multi-level virtual scene, dynamically switch the level of the multi-level virtual scene according to the heterogeneous feature expression, and conduct progressive virtual sensory stimulation training for pre-adaptation to the real scene in the multi-level virtual scene after the switch.

[0034] In this context, the real-world scene can be the user's destination. Scene information within the real-world scene can include environmental elements and social context elements. Environmental elements can include plants, buildings, animals, etc., while social context elements include the people interacting with the user, including the number and identities of these individuals. In some embodiments, the scene elements used to construct different levels of virtual scenes may differ, specifically including variations in environmental elements and social context elements.

[0035] In some embodiments, the scene information is hierarchically split into elements, and progressive hierarchical modeling is performed on the elements at different levels after splitting to construct multi-level virtual scenes that simulate real environments at different levels. This includes: splitting the environmental elements and social context elements in the scene information into elements at different levels according to their levels to obtain environmental elements and social context elements at different levels; constructing virtual scenes at corresponding levels using the environmental elements and social context elements at different levels; wherein, the virtual scenes at different levels are used to progressively simulate real scenes, and as progressive training progresses, the virtual scenes continuously approach the real scenes.

[0036] In some embodiments, virtual modeling can specifically be VR modeling, XR modeling, and MR modeling, etc., and a multi-level virtual scene can include four levels. The following embodiments use VR modeling and the construction of a four-level virtual scene as an example for illustration. However, it is understood that other embodiments may also include virtual scenes with other numbers of levels, and this application is not limited to a four-level virtual scene.

[0037] The application includes a VR modeling module, which constructs an immersive VR scene containing dynamic stressors and social contexts according to real-time environmental information of the destination in the real scene. And by splitting the scene elements in the real scene to obtain multi-level scene elements, and constructing corresponding multi-level progressive virtual scenes according to the multi-level scene elements. Specifically, the multi-level virtual scenes can include a basic version virtual scene, a simplified version virtual scene, a close-to-real version virtual scene, and a completely real version virtual scene, which are generated through four progressive multi-level virtual scenes to meet the user's demand for progressive training scenes.

[0038] The application also includes a VR training interaction module, which can guide the user to experience and try to cope with the preset stimulus and social interaction in the corresponding simulation environment of the multi-level virtual scene, and record the user's training performance data in the corresponding level virtual scene. Specifically, it can be divided into four progressive multi-level training stages. In the corresponding multi-level virtual scene, the user's performance is evaluated based on physiological, behavioral, and task quantitative indicators, and the training difficulty is switched in combination with the stability of the user's performance and the guardian feedback, to realize dynamic switching of levels in the dynamic multi-level virtual scene. Through the integration of the progressive training mode management subsystem, the training difficulty is dynamically adjusted. It should be noted that the four levels in the above embodiments are only one embodiment, and in other embodiments, there can be other levels.

[0039] In some embodiments, different levels of virtual scenes are constructed based on different environmental elements and social context elements. Take constructing four different levels of virtual scenes as an example for illustration.

[0040] In the first level virtual scene (basic version), the core goal is to eliminate the strangeness of the VR environment and establish initial safety, and the training task is to follow the virtual caregiver along a fixed route to familiarize with the environment. The scene features of the environmental elements only retain environmental elements, for example, including plants, buildings, animals, etc. The social context elements do not include strangers, and the virtual caregiver can be generated only according to the child's familiar guardian, and the virtual caregiver is visible at all times. Moreover, the environmental factors are all within the safety threshold.

[0041] In the second level of virtual scene (simplified version), the core goal is to adapt to slight burst stimulus, autonomously complete simple tasks, try passive social response, training tasks are to autonomously explore in virtual environment, complete simple tasks such as finding specified targets, etc. in park scene, find pink slide, supermarket scene, find potato chip shelf, respond to low-intensity single environmental stimulus, respond to virtual pedestrians. Environmental elements and social scenario elements include randomly increasing a preset number of (for example, three to five) virtual pedestrians, virtual pedestrians will actively initiate friendly interaction with the child, and single environmental disturbance will be randomly introduced at a specific time point, but the environmental signal is still within the safety threshold. The system will have a prompt before the stimulus is introduced, and the virtual caregiver is visible at all times.

[0042] In the third level of virtual scene (close to real version), the core goal is to adapt to complex stimulus, complete simple active social interaction, training tasks are to autonomously explore in virtual environment, complete simple tasks such as finding specified targets, etc. in park scene, find pink slide, supermarket scene, find potato chip shelf, respond to complex stimulus, respond to common social situations in the scene, complete active social tasks. Environmental elements and social scenario elements include fully realistic restoration of real people flow, randomly introduce complex environmental disturbance at a specific time point, and the disturbance item environmental factor is close to but does not exceed the safety threshold. The system has no prompt when the stimulus is introduced, sets typical and common social situations in the scene, and the virtual caregiver is visible at all times.

[0043] In the fourth level of virtual scene (fully real version), the core goal is to adapt to real scene, respond to potential overload stimulus, training tasks are to autonomously explore in virtual environment, complete simple tasks such as finding specified targets, etc. in park scene, find pink slide, supermarket scene, find potato chip shelf, respond to real environmental stimulus, including unpredictable potential overload stimulus, respond to common social situations in the scene, complete active social tasks. Environmental elements and social scenario elements include fully realistic restoration of real people flow, fully realistic restoration of real environmental factors, if it is detected that an environmental factor exceeds the safety threshold, the system automatically records, sets typical and common social situations in the scene, and the virtual caregiver is not visible, but will inform the child caregiver before the training begins.

[0044] The virtual scenes of different levels are shown in the following Table 1.

[0045] Table 1 Virtual scenes of different levels

[0046]

[0047] It should be noted that the basic version is the starting point of virtual training pre-adapted to real scenes, the simplified version is the preliminary complication of the basic version, the fully real version is the end point of pre-adaptation training, and the close to real version is a simplification of the fully real version.

[0048] Referring to Figure 2 , the arrow indicates the stage switching process, the dashed arrow corresponds to the version switching, the difficulty level of which is related to the real environment complexity of the destination, the version switching of the solid arrow, the difficulty level of which is controlled by the experience of experts, and the difficulty span is low. And there may be a rollback iteration when the version is switched.

[0049] In some embodiments, the present application also sets up single environmental interference and composite environmental interference. For example, suddenly loud music, billboards flashing, each of which is a single environmental interference (such as sound, light flashing, etc.), and the composite environmental interference is a random combination of single environmental interference. Different levels of virtual scenes are constructed using different environmental interferences, and the construction of progressive virtual scenes is realized.

[0050] In the above embodiment, the environmental interference is introduced at a specific time point in stage two and stage three, including setting the shortest time interval between different environmental interferences as t, and setting the number of times of introducing environmental interference in the training stage as n. Then, in the training stage, n time points are randomly sampled, and environmental interference is introduced according to the interval between time points greater than t.

[0051] In the above embodiment, the modeling method of virtual scene based on real scene, the multi-level progressive virtual scene is generated based on real environment data, the stressor is strongly associated with the scene, so that the reality and adaptability of the virtual scene are significantly improved, solving the problem that the existing training is disconnected with the real scene, and through the multi-level progressive scene version design, different training stages can also be adapted, so that the user's training is more flexible.

[0052] In the above embodiment, a progressive training method pre-adapted to a real scene is provided. This virtual scene modeling method based on real scene enables users to perform virtual sensory stimulation training in the virtual scene constructed by simulating the real scene. Through the dynamic virtual modeling based on real environment information provided by the present application, which is different from the existing virtual training based on pre-set virtual scene, the present application dynamically constructs a highly realistic virtual training environment using real-time or near real-time environment information of the destination, such as images, videos, audio, etc., and introduces dynamic stressors (such as sudden high-decibel noise, strong flashing light, etc.) and typical social situations (such as shopping, queuing, etc.) that are common in this environment category and may cause user discomfort, so that users can perform progressive virtual sensory stimulation training pre-adapted to real scenes in the constructed virtual scene.

[0053] In some embodiments, the heterogeneous feature expression of the user in the pre-adaptation in the multi-level virtual scene is obtained, and the level of the multi-level virtual scene is dynamically switched according to the heterogeneous feature expression, including: obtaining the heterogeneous feature expression of the user in the pre-adaptation in the current level virtual scene; determining the adaptability of the user in the current level virtual scene according to the sensing parameters and the interaction parameters in the heterogeneous feature expression, and dynamically switching the level of the virtual environment in which the user is located according to the adaptability.

[0054] The heterogeneous feature expression refers to a plurality of different dimension and different type of data features, which comprehensively evaluate the state of the user through a plurality of dimensions, and correspondingly switch the level of the virtual world in which the user is located.

[0055] Specifically, the present application includes a multi-modal data synchronous acquisition module, which acquires physiological signals, behavior data and training performance data of the user in real time by using multi-modal sensors. The physiological signals can include one or more of electroencephalogram (EEG), electrooculogram (EOG), galvanic skin response (GSR), heart rate variability (HRV), etc. The behavior data can include one or more of voice, tone, camera shooting behavior state, etc.

[0056] The present application further includes a multi-modal feature fusion analysis module, which extracts physiological signal features and behavior features through the multi-modal feature fusion analysis module, obtains task-related training data from the training system, monitors the state of the user, evaluates the training performance and adaptability of the user, and feeds back the evaluation results to the VR training module for controlling the switching of the training stage, so that the user can switch to a virtual scene level that matches his performance.

[0057] The present application further includes an evaluation result output and early warning module, which generates an adaptability evaluation report through the evaluation result output and early warning module, which can include a stage progress table, and provides prevention suggestions when high risk inadaptation is detected. Specifically, the output content can include progress visualization, such as stage completion progress bar, e.g. physiological completion 100%, task completion 80%, behavior abnormality 0. Further, the corresponding output content can be synchronized to the caregiver background. If the evaluation user cannot successfully complete all training stages, the guardian can be further pushed a destination risk warning.

[0058] In some embodiments, the determining the adaptability of the user in the current level virtual scene according to the sensing parameters and the interaction parameters in the heterogeneous feature expression comprises: determining an environment adaptation degree corresponding to the physiological stability of the user according to the sensing parameters and / or the interaction parameters of the user; determining a stress reaction degree corresponding to the behavior performance of the user according to the sensing parameters and / or the interaction parameters of the user; determining a task completion degree corresponding to the task completion quality of the user according to the sensing parameters and / or the interaction parameters of the user; determining an index stability degree corresponding to the stability verification of the user according to the sensing parameters and / or the interaction parameters of the user; determining a care evaluation degree corresponding to the caregiver cooperation of the user according to the sensing parameters and / or the interaction parameters of the user; and determining the adaptability of the user in the current level virtual scene according to at least one of the environment adaptation degree, the stress reaction degree, the task completion degree, the index stability degree and the care evaluation degree.

[0059] In some embodiments, the heterogeneous feature expression comprises the sensing parameters and the interaction parameters, and further, the sensing parameters can be used to evaluate the physiological stability and the behavior performance, and the interaction parameters comprise the task completion quality, the stability verification and the caregiver cooperation confirmation.

[0060] In some embodiments, the environment adaptation degree of the user can be determined according to the target frequency band proportion in the brain electrical data of the user and the relationship between the target physiological signal and the physiological stability feature interval. The target frequency band in the brain electrical data can be an alpha wave in the collected EEG, and the target physiological signal can be a physiological signal such as GSR, HRV, etc. Specifically, the environment adaptation degree of the user can be determined according to the alpha wave proportion in the EEG and the relationship between the physiological signal such as GSR, HRV, etc. and the physiological stability feature interval.

[0061] In some embodiments, the stress reaction degree of the user can be determined according to the avoidance behavior frequency and the stress reaction behavior frequency of the user. Specifically, the avoidance behavior frequency and the stress reaction behavior frequency of the user can be determined by the data collected by the camera and the sensor during the training process of the user, and then the stress reaction degree of the user can be determined.

[0062] In some embodiments, the task completion degree of the user can be determined according to the task completion rate, the task interruption times and the task completion time of the user. Specifically, the task completion rate, the task interruption times and the task completion time of the user can be determined by the data collected by the VR training interaction module during the training process of the user, and then the task completion degree of the user can be determined.

[0063] In some embodiments, the index stability of the user can be determined according to the number of times the user continuously meets the environmental adaptation degree, the stress reaction degree, and the task completion degree. Specifically, the index stability of the user can be determined by the multi-modal fusion analysis module during the training of the user.

[0064] In some embodiments, the care evaluation degree of the user can be determined according to the feedback of the caregiver.

[0065] Further, the level adaptation of the user is determined according to at least one of the environmental adaptation degree, the stress reaction degree, the task completion degree, the index stability, and the care evaluation degree, and the level of the virtual environment in which the user is located is dynamically switched according to the level adaptation. In some embodiments, the adaptation degree of the user in the current level virtual scene is determined according to the sensing parameters and the interaction parameters in the heterogeneous feature expression, the level adaptation of the user is determined according to the adaptation degree, including: if the adaptation degree of the user in the current level virtual scene determined according to the sensing parameters and / or the interaction parameters meets the switching condition, switching from the current level virtual scene to the next level virtual scene is performed, so that the user performs the virtual sensory stimulation training pre-adapted to the real scene in the next level virtual scene, and the level of the virtual scene is iteratively adjusted according to the adaptation degree of the user until the user completes the virtual sensory stimulation training pre-adapted to the real scene in the highest level virtual scene or terminates the training; if the adaptation degree of the user in the current level virtual scene determined according to the sensing parameters and / or the interaction parameters does not meet the switching condition, whether the user has a rollback record that has reached a preset upper limit of times in the current level virtual scene is determined, if yes, a risk warning is performed or the training is terminated, if not, the user continues to perform the virtual sensory stimulation training pre-adapted to the real scene in the current level virtual scene, and the level of the virtual scene is iteratively adjusted according to the adaptation degree of the user until the user completes the virtual sensory stimulation training pre-adapted to the real scene in the highest level virtual scene or terminates the training.

[0066] Referring to Figure 5A flowchart of a different level scene switching process is provided for the present application. First, it is determined whether the stability verification number requirement of the corresponding level is reached. If not, continue training in the current level. If the stability verification number has been reached, continue to determine whether the condition for switching to stage n+1 is met. If it is met, switch to level n+1 for training. If it is not met, continue to determine whether there is a rollback record in the current stage n. If there is no rollback record, continue training in the current level. Otherwise, end the training and push the destination risk warning to the guardian. In some embodiments, the upper limit of the rollback number of the user in the current stage is 1. It can be understood that in other embodiments, the upper limit of the rollback number of the user can also be other quantities, not limited to 1, and can be flexibly set according to actual needs. In other embodiments, the upper limit of the rollback number of the user in different levels of virtual environment can be different. For example, as the difficulty of virtual training becomes greater, the upper limit of the rollback number of the user is also increased accordingly.

[0067] In some embodiments, the switching conditions for different levels of virtual scenes are set in advance. Specifically, to ensure the objectivity, stability and individual adaptability of the training stage switching, the switching conditions can be determined according to multi-modal quantitative data, which can include physiological signals, behavior performance, task completion quality, continuous training performance stability, and caregiver coordination confirmation, etc. to form a three-layer decision logic of multi-dimensional index standard and stability verification and safety bottom line. Among them, multi-dimensional standard refers to the need to meet physiological stability, behavior performance, and task completion quality three-dimensional indicators at the same time to avoid single indicator misjudgment. Stability verification refers to the need to meet the standard for continuous multiple times, such as two or three consecutive training, rather than single accidental standard, to ensure the sustainability of the user's ability. Caregiver coordination confirmation refers to the caregiver confirming through background observation that the user's emotions remain calm during training and there is no resistance expression, such as expressing that he does not want to play or is afraid, etc.

[0068] In the following embodiments, a virtual scene including four levels is taken as an example for illustration.

[0069] Switch from stage 1 to stage 2 (first level virtual scene switching to second level virtual scene). The core goal is to confirm that the user has adapted to the VR environment, no basic stress reaction, and can complete simple instruction tasks. Among them, the judgment dimensions can include physiological stability, behavior performance, task completion quality, stability verification, and caregiver collaborative confirmation. Among them, the physiological stability dimension includes obtaining the physiological signals such as EEG, GSR, HRV of the user according to the multi-modal data acquisition module, and further evaluating whether the collected signals meet the index requirements, for example, whether the proportion of alpha waves of EEG is greater than or equal to r1, where alpha waves reflect a relaxed state, the higher the proportion, the more stable, whether the physiological signals such as GSR, HRV are within the physiological stability characteristic interval. Among them, the behavior performance includes obtaining the user's avoidance behavior data or stress behavior performance data according to the camera or sensor behavior recognition, and further evaluating whether the collected data meets the index requirements, for example, whether the avoidance behavior frequency is zero times per training, avoidance behavior includes covering eyes, retreating, crying and other avoidance resistance behaviors, whether the frequency of strong stress reaction behavior is zero times per training, strong stress reaction behavior includes screaming, curling and other strong stress behaviors. Among them, the task completion quality includes whether the user's task completion rate determined by the VR training interaction module reaches 100%, whether the user's task interruption times are 0 times, whether the user's average completion time is less than the preset time (such as 5 minutes), and whether the user has obvious procrastination behavior. Among them, the stability verification includes whether the user's continuous two times of training determined by the multi-modal fusion analysis module meet all the above indexes. Among them, the caregiver collaborative confirmation includes confirming whether the user's emotion is calm and whether there is no resistance expression (such as resistance expression of not wanting to play or being afraid) during the training process according to the multi-caregiver interaction background observation.

[0070] The switching condition from stage 1 to stage 2, please refer to the following table 2.

[0071] Table 2 Switching condition from stage 1 to stage 2

[0072]

[0073] Switch from stage 2 to stage 3 (second level virtual scene switching to third level virtual scene). The core goal is to confirm that the user can tolerate low intensity stressors and can complete passive social responses. Among them, the judgment dimensions can include physiological stability, behavior performance, task completion quality, stability verification, and caregiver collaborative confirmation. Among them, the physiological stability dimension includes obtaining the physiological signals such as EEG, GSR, and HRV of the user according to the multi-modal data acquisition module, and further evaluating whether the collected signals meet the index requirements, for example, whether the proportion of alpha waves of EEG is greater than or equal to r1, where alpha waves reflect a relaxed state, the higher the proportion, the more stable, and slight fluctuations are allowed, reflecting tolerance to low stimulation, and even if the physiological signals such as GSR and HRV fluctuate, they are still within the physiological stability characteristic interval. Among them, the behavior performance includes obtaining the user's avoidance behavior data or stress behavior performance data according to the camera or sensor behavior recognition and VR interaction data, and further evaluating whether the collected data meets the index requirements, for example, whether the avoidance behavior frequency is zero times in each training, avoidance behavior can include covering eyes, retreating, crying and other avoidance resistance behaviors, and special attention is paid to stimulus preview compliance, from seeing the stimulus preview, the current action can be maintained without escaping. Whether the frequency of strong stress response behavior meets zero times in each training, strong stress response behavior can include screaming, curling and other strong stress behaviors. Among them, the task completion quality includes whether the user's task completion rate determined by the camera or sensor behavior recognition and voice collection of the VR training interaction module reaches 100%, completing simple tasks such as finding specified targets, passive social responses such as virtual pedestrians waving hands, response methods including nodding, waving hands, and whispering "Hi". Whether the user's task interruption times is 0 times, whether the user's average completion time is less than the preset time (such as 8 minutes), and whether the user has no obvious procrastination behavior. Among them, the stability verification includes whether the user meets all the above indexes in two consecutive trainings determined by the multi-modal fusion analysis module. Among them, the caregiver collaborative confirmation includes confirming whether the user's emotions are calm and whether there is no resistance expression (such as resistance expression of not wanting to play or being afraid) during the training process according to the multi-caregiver interaction background observation.

[0074] The switching condition from stage 2 to stage 3, see Table 3 below.

[0075] Table 3 Switching condition from stage 2 to stage 3

[0076]

[0077] Switch from stage 3 to stage 4 (third level virtual scene to fourth level virtual scene). The core goal is to confirm that the patient can adapt to the complex stimulus and complete the designated task in a common social situation. Among them, the judgment dimensions can include physiological stability, behavior performance, task completion quality, stability verification, and caregiver collaborative confirmation. Among them, the physiological stability dimension includes obtaining the physiological signals such as EEG, GSR, and HRV of the user according to the multi-modal data acquisition module, and further evaluating whether the collected signals meet the index requirements, for example, whether the proportion of alpha waves in EEG is greater than or equal to r2, where alpha waves reflect a relaxed state, and the higher the proportion, the more stable, allowing moderate fluctuations, reflecting complex stimulus tolerance, and GSR, HRV and other physiological signals are still within the physiological stability characteristic interval even if they fluctuate. Among them, the behavior performance includes obtaining the user's avoidance behavior data or stress behavior performance data according to the camera or sensor behavior recognition and VR interaction data, and further evaluating whether the collected data meets the index requirements, for example, whether the avoidance behavior frequency is zero times in each training, avoidance behavior can include covering eyes, retreating, crying and other avoidance resistance behaviors. Whether the frequency of strong stress response behavior meets zero times in each training, strong stress response behavior can include screaming, curling up and other strong stress behaviors, and particular attention is paid to the behavior performance after the complex stimulus. Among them, the task completion quality includes whether the user's task completion rate determined by the camera or sensor behavior recognition and voice collection of the VR training interaction module reaches 90%, completes simple tasks such as finding designated targets, and completes designated tasks in classic social situations, such as buying candy and queuing up to pay at the supermarket, and saying thank you after receiving a gift. Whether the user's task interruption frequency is 0 times, whether the user's average completion time is less than the preset time (such as 10 minutes), and whether the user has no obvious procrastination behavior. Among them, the stability verification includes whether the user's continuous three times of training meet all the above-mentioned indicators. Among them, the caregiver collaborative confirmation includes confirming whether the user's mood is calm and whether there is no resistance expression (such as resistance expression of not wanting to play or being afraid) during the training process according to the multi-caregiver interaction background observation.

[0078] The switching condition from stage 3 to stage 4 is shown in Table 4 below.

[0079] Table 4 Switching condition from stage 3 to stage 4

[0080]

[0081] Switch from stage 4 to end of training (switch from the fourth level virtual scene to the end of training). Among them, the judgment dimension can include physiological stability, behavior performance, task completion quality, stability verification, and caregiver collaborative confirmation. Among them, the physiological stability dimension includes obtaining the physiological signals such as EEG, GSR, and HRV of the user according to the multi-modal data acquisition module, and further evaluating whether the collected signals meet the index requirements, for example, whether the proportion of alpha waves of EEG is greater than or equal to r2, where alpha waves reflect a relaxed state, the higher the proportion, the more stable, allowing moderate fluctuations, reflecting compound stimulus tolerance, and GSR, HRV and other physiological signals are still within the physiological stability characteristic interval even if they fluctuate. Among them, the behavior performance includes obtaining the user's avoidance behavior data or stress behavior performance data according to the camera or sensor behavior recognition and VR interaction data, and further evaluating whether the collected data meets the index requirements, for example, whether the avoidance behavior frequency is zero times in each training, avoidance behavior can include covering eyes, retreating, crying and other avoidance resistance behaviors. Whether the frequency of strong stress response behavior meets zero times in each training, strong stress response behavior can include screaming, curling and other strong stress behaviors, and particular attention is paid to the behavior performance after the environmental stimulus approaches or exceeds the threshold. Among them, the task completion quality includes whether the user's task completion rate determined by the camera or sensor behavior recognition and voice collection of the VR training interaction module reaches 80%, completes simple tasks such as finding specified targets, and completes specified tasks in classic social situations, such as buying candy and queuing up to pay at the supermarket, and saying thank you after receiving a gift. The user's task interruption times are less than or equal to 1 time, the user's average completion time is less than the preset time (such as 12 minutes), and the user has no obvious procrastination behavior. Among them, the stability verification includes whether the user's continuous three times of training determined by the multi-modal fusion analysis module all meet all the above indexes. Among them, the caregiver collaborative confirmation includes confirming whether the user's emotion is calm and whether there is no resistance expression (such as resistance expression of not wanting to play or being afraid) during the training process according to the multi-caregiver interaction background observation.

[0082] The switching condition from stage 4 to the end of training is shown in the following table 5.

[0083] Table 5 Switching condition from stage 4 to end of training

[0084]

[0085] Through the above detailed conditions, the stage switching is no longer dependent on subjective judgment, but based on quantifiable, verifiable multi-dimensional data, ensuring that the training difficulty matches the user's training adaptability, maximizing the pre-adaptation training effect.

[0086] It should be noted that, regarding the switching conditions of the multi-level virtual scene training stage, in the stability verification, the control degree of stage switching is more and more strict in the switching process from stage 1 to stage 4. For example, stage 1 and stage 2 are relatively simple, so only two consecutive times of reaching the standard in the current training stage are required for switching from stage 1 to stage 2 and from stage 2 to stage 3. Stage 3 and stage 4 are closer to the real environment, and the difficulty is increased, so three consecutive times of reaching the standard in the current training stage are required for switching from stage 3 to stage 4 and from stage 4 to the end of the prompt training.

[0087] In the above embodiment, the step of obtaining the alpha wave proportion threshold in the stage switching condition includes: through multi-center cross-sectional clinical research, special users meeting certain diagnostic criteria are included (each subgroup can be stratified according to the severity of ASD symptoms and the age of the user, and ≥30 cases in each subgroup). In a low-noise, soft-light environment, open-eye resting electroencephalogram signals are collected. In the process of the user receiving cognitive training, electroencephalogram signals are collected. For different subgroups, the alpha wave proportion under two collection conditions is calculated, and through reliability test, the 95% reference range (2.5%-97.5% quantile) of the alpha wave proportion in the relaxed state and the task awareness state of each subgroup is determined. When the user uses the device, the alpha wave proportion threshold is determined from the corresponding subgroup according to the score of the clinical diagnosis and the age. r1 is the lower limit of the alpha wave proportion in the relaxed state of the subgroup, and r2 is the lower limit of the alpha wave proportion in the task awareness state of the subgroup.

[0088] In some embodiments, the method further comprises: if at least one of the physiological stability feature interval, the stress reaction degree, the task completion degree, the index stability degree, and the care evaluation degree triggers a high-risk threshold of the corresponding level virtual environment, terminating the virtual sensory stimulation training adapted to the real scene. Specifically, the application also provides switching adjustment rules for special cases, including a safety bottomed pause mechanism. For example, if any of the following conditions occurs during the training process of the virtual scene, the training is immediately paused. When the physiological signal exceeds the stable interval and lasts for 30s, the electromyographic disturbance in the EEG increases sharply and lasts for more than 10s, abnormal behavior such as loud crying, forcibly removing the device, curling up and refusing to interact, etc. It should be noted that the above-mentioned time is only for example, and in other embodiments, the time value can be customized according to the needs.

[0089] In another embodiment, a rollback mechanism is also included, if the switching condition from training stage n to training stage n+1 is not met, the system automatically rolls back to stage n and re-trains stage n. If after re-training, it is still unable to enter the next stage (n+1), the training is paused and ended.

[0090] In some embodiments, the method provided by the present application further comprises a real-time monitoring module. During the training process, a stage completion progress bar is generated every 2 minutes by the multi-modal fusion analysis module, for example, physiological completion 100%, task completion 80%, and behavior anomaly 0, and is synchronized to the caregiver background.

[0091] In some embodiments, the present application further comprises a completion determination module. After the single-scene virtual scene training is completed, the system automatically compares the refined index requirements of stage switching to generate a single-scene completion report, such as whether this training is completed, and if there are uncompleted items such as physiological indicators GSR exceeding the stable interval, the system displays the uncompleted items.

[0092] The present application further comprises a continuous verification module, and the switching conditions corresponding to the continuous verification module are different in the switching process of different stages. For example, for stage 1 and stage 2, when the continuous 2 training shows completion, the system triggers an advanced prompt, which is executed after being confirmed by the caregiver, such as clicking to confirm the advancement to stage 3. For stage 3 and stage 4, when the continuous 3 training shows completion, the system triggers an advanced prompt, which is executed after being confirmed by the caregiver, such as clicking to confirm the advancement to stage 4.

[0093] The present application further comprises a scene synchronization module. After the level of switching virtual scenes is completed, the VR modeling module can automatically load the virtual scene version corresponding to the next stage, such as the stage 3 corresponding to the near-real version scene, and the training interaction module updates the task and social requirements.

[0094] In another embodiment, for users who cannot complete the training, the system further comprises difficulty degradation adjustment. When the user inputs the destination, the system automatically queries the training log of the same destination. If there are more than three records of “stage rollback, training suspended”, the system calculates the difficulty degradation correction factor , which is used as a decay factor to correct the intensity of environmental interference introduced in all training stages.

[0095] The calculation formula is as follows:

[0096]

[0097] Wherein is the lower limit of the proportion of alpha waves corresponding to the task awareness state of the user's subgroup, is the proportion of alpha band in the resting state electroencephalogram when the user does not wear the VR device.

[0098] In some embodiments, the virtual scene of the corresponding level is constructed by using different levels of environmental elements and social situation elements, comprising:

[0099] ​When the user first enters the virtual scene of the key level for pre-adapting to the virtual sensory stimulation training of the real scene, the sensory hypersensitive items of the user are not set in the environmental elements of the key level for the construction of the virtual scene of the key level; when the user is not first in the virtual scene of the key level for pre-adapting to the virtual sensory stimulation training of the real scene, the sensory hypersensitive items of the user are dynamically set in the environmental elements of the key level for the construction of the virtual scene of the key level, wherein the key level is a sub-high level.

[0100] In some embodiments, the application also sets the sensory hypersensitive items of the user in the virtual scene of the key level. For example, the composite environmental interference can be set in stage 3 and stage 4. Before switching from stage 3 to stage 4, it needs to meet the training standard in stage 3 for three times in a row. For example, the composite environmental interference is introduced N times in stage 3. In the first training of stage 3, the sensory hypersensitive items of the user are avoided each time the composite environmental interference is introduced. In the second training of stage 3, the sensory hypersensitive items of the user are added N / 2 times when the composite environmental interference is introduced. In the third training of stage 3, the sensory hypersensitive items of the user are added each time the composite environmental interference is introduced. For example, according to the clinical information of the user, such as the sensory processing ability profile, it can be known that the user is very sensitive to light flashes, so in the first training of stage 3, the light flashes are excluded when the composite stimulation is introduced. In stage 4, when the real environment is restored at a ratio of 1:1, if the sensory hypersensitive items of the user are detected, at this time, no filtering is done, only system automatic recording is done, which is used for subsequent review analysis. Combined with the sensory hypersensitive clinical information of the user, the introduced environmental interference is controlled, which gradually adapts the user to the environment on the one hand, and also helps to ensure the comprehensiveness of the pre-training on the other hand.

[0101] In some embodiments, the calculation step of the physiological stability feature interval in the environmental adaptation degree comprises: obtaining a first parameter of the user in the current scene according to a safe environment standard, the first parameter comprising an environmental comfort reference value, obtaining a scene risk correction weight corresponding to the current scene category, adjusting the first parameter according to a scene risk correction strategy, a scene risk correction weight, and a sensitivity correction strategy to obtain an environmental critical condition; obtaining a physiological stability feature of the user in the current scene category and the current activity state, correcting the physiological stability feature using a scene risk correction strategy to obtain a physiological stability feature interval of the user; collecting an electroencephalogram signal of the user, calculating an adaptability correction value of the user to the training device according to the electroencephalogram signal, and correcting the physiological stability feature interval according to the adaptability correction value.

[0102] In some embodiments, the application also includes a fatigue monitoring module, a certain length of rest period between stage switching, during which the resting EEG signal of the user is collected, the fatigue index is calculated, and if the fatigue index exceeds the threshold value, the system prompts to rest and automatically pauses the training. The fatigue index is calculated based on the EEG signal of the frontal lobe electrode, and the fatigue index calculation formula is as follows:

[0103]

[0104] Wherein, represents the fatigue index, represents the relative power ratio of the wave band, represents the relative power ratio of the wave band. Generally, a higher ratio indicates a more relaxed state, while a lower ratio reflects a higher cognitive activity or stress state.

[0105] In some embodiments, it also includes a device adaptability secondary correction, the difficulty level of stage 1 and stage 2 is low, in order to familiarize with the VR environment and adapt to the main purpose of slight stimulation, when switching from stage 3 to stage 4, the adaptability correction value of the user to the whole set of equipment will be calculated again based on the resting signal, and the stable physiological feature interval is corrected again.

[0106] In some embodiments, the collection of the user's EEG signal, the calculation of the adaptability correction value of the user to the training device according to the EEG signal, and the correction of the physiological stability feature interval according to the adaptability correction value, comprises: collecting the resting EEG signal of the user without wearing the training device, calculating the first feature parameter group; collecting the resting EEG signal of the user after wearing the training device, calculating the second feature parameter group; determining the adaptability correction value of the user to the training device according to the difference between the first feature parameter group and the second feature parameter group; determining the scaling factor according to the adaptability correction value, and correcting the physiological stability feature interval according to the scaling factor. Wherein, the first feature parameter group can include the relative power ratio of the alpha band calculated according to the resting EEG signal of the user without wearing the VR device, and the first feature parameter group can also include other EEG indicators such as EEG fractal dimension. The first feature parameter group can include the relative power ratio of the alpha band calculated according to the resting EEG signal of the user after wearing the VR device, and the second feature parameter group can also include other EEG indicators such as EEG fractal dimension.

[0107] Specifically, the steps of obtaining the adaptability correction value of the user to the VR device are as follows:

[0108] Collecting the resting EEG signal of the user without wearing the VR device, calculating the relative power ratio of the alpha band (the first feature parameter group), collecting the resting EEG signal of the user after wearing the VR device, calculating the relative power ratio of the alpha band a second feature parameter set, and further obtain an initial adaptability correction value of the user to the VR device and take as a scaling factor to correct the upper limit of the physiological stability feature interval of the user.

[0109]

[0110] If more than a certain proportion (such as 50%) of the environmental elements exceed the user's comfortable environment threshold, a warning is output, prompting the guardian that "the destination risk is high, and it is not easy to travel", and no subsequent gradual pre-training is carried out.

[0111] In some embodiments, the present application also provides a judgment instruction on whether the user is suitable for the training system, which is judged according to the resting electroencephalogram signal collected by the individualized threshold value adaptation module, and the lower limit of r1 corresponding to the user's subgroup is compared. If , the system prompts "the user's adaptability to the VR training system is low".

[0112] Specifically, referring to Figure 3 and Figure 4 , it is an individualized threshold value adaptation and calibration flowchart. First, the guardian inputs the destination information, the system judges the scene category of the destination, and obtains the scene risk value of the scene category for the user child from the historical behavior information of the user child, and calculates the comfortable threshold of each environmental information under the scene category in combination with the clinical information of the child. Further, in combination with the clinical information of the child, the physiological signal corresponding to the physiological stability feature interval of the user child under the scene category is calculated. Further, it also includes preliminary judgment of the rationality of the destination based on the initial inspection of each environmental element of the destination according to the comfortable environment threshold adapted to the user child.

[0113] In some embodiments, the physiological stability feature interval represents the threshold interval range of the user maintaining a stable physiological state. Specifically, the scene category can be automatically determined according to the destination information of the user, the scene risk value of the scene is obtained in combination with the historical behavior data and clinical information of the user, the comfortable environment threshold and stable physiological feature interval under the scene category are calculated, and the adaptability correction value of the user to the whole set of devices after wearing the VR device is obtained, and then the stable physiological feature interval under the wearing condition of the VR device is corrected. The determination steps of the physiological stability feature interval of the user are as follows.

[0114] In some embodiments, the wearable device retrieves the user's clinical information from the memory via a data processor. Specifically, the data processor retrieves the corresponding physiological stability features from the memory based on the user's current activity state. Specifically, the user's current activity state can be determined based on physiological signals and pose information, and the physiological stability feature range corresponding to the user's physiological signals can be determined based on the scene category, the current activity state, and the scene risk feature information corresponding to the current scene category.

[0115] In some embodiments, the step of determining the physiological stability feature interval includes: obtaining a third parameter from the user's historical behavior information, the third parameter including the user's physiological stability features in the current scenario and current activity state; obtaining the physiological stability feature interval based on a third co-correlation relationship reflected by the third parameter, the scenario risk correction strategy, and the scenario risk correction weight; wherein, in the third co-correlation relationship, the physiological stability feature interval increases in response to an increase in the third parameter, the scenario risk correction strategy, and the scenario risk correction weight. In some embodiments, the threshold discrimination and output decision module includes a physiological stability feature interval calculation unit. Specifically, the physiological stability features of the user in the current activity state are obtained from historical behavior information. Based on physiological homeostasis Scenario risk correction strategy Scenario risk correction weight Determine the stable physiological characteristic intervals of each physiological signal. Specifically, calculate the current scene category. Current activity status Below, the stable range of each physiological signal, based on physiological signals For example, its physiological stable characteristic range for:

[0116]

[0117] Among them, the current activity status Next, physiological signals Its stable characteristics include its mean and standard deviation , It is the scenario risk correction weight.

[0118] In the above embodiments, the physiological stability feature interval is obtained through a third synergistic positive correlation. This relationship emphasizes the synergistic effect between multiple factors or variables, jointly promoting the achievement of a certain goal or result. The physiological stability feature interval constructed through this relationship takes into account the correlation and synergy between data from multiple dimensions, making the construction of the physiological stability feature interval more accurate and comprehensive.

[0119] In some embodiments, the scene category can be determined based on the environmental information, and scene risk feature information corresponding to the scene category is further obtained. For example, scene risk feature information corresponding to the current scene category can be obtained from the historical behavior information of the user, and a scene risk correction strategy is obtained according to a first synergistic positive correlation relationship reflected by the scene risk feature information of the scene category.

[0120] In some embodiments, the present application can also include a personal information acquisition module, which includes a historical behavior information acquisition unit, and scene risk feature information corresponding to a historical scene matching the current scene category is obtained through the historical behavior information acquisition unit.

[0121] Specifically, the historical behavior data and clinical information of the user can be stored in the memory of the wearable device. The data processor in the wearable device retrieves the scene risk feature information matching the current scene category from the memory according to the current scene category.

[0122] In some embodiments, scene risk feature information can also be obtained by conducting a questionnaire survey on a large group of ASD children. Specifically, an initial value can be set according to the obtained population base value. Before the device is started, the scene risk feature information can be personalized adjusted according to the user historical behavior questionnaire filled in by the guardian. In the subsequent use process, the device will adjust the scene risk feature information according to the number of pre-alarm, warning and intervention in each scene. The guardian can also edit the use log about the correct detection, false alarm and missed alarm of the device through the feedback module of the APP, so as to help the device update the scene risk feature information. It should be noted that the user will show unique behavior characteristic changes in the scene overload state, and these changes directly affect the scene risk feature information. For example, noisy shopping malls, bright classrooms, living rooms and other scenes have different scene risk feature information.

[0123] In the above embodiments, the scene risk correction strategy is constructed according to the scene risk feature information and the first synergistic positive correlation relationship reflected by the scene risk feature information. The synergistic positive correlation relationship is used to indicate that there is a mutual promotion and common growth relationship between two or more variables. When one variable increases, one or more related variables will also increase accordingly. This relationship emphasizes the synergistic effect between multiple factors or variables, and promotes the realization of a certain target or result. The scene risk correction strategy constructed by this relationship considers the relevance and synergy between multiple dimensional data, so that the construction of the scene risk correction strategy is more accurate.

[0124] In some embodiments, the scene risk correction strategy is obtained according to a first synergistic positive correlation relationship reflected by the scene risk feature information of the scene category, including: obtaining the scene risk feature information corresponding to the current scene category from the historical behavior information of the user; obtaining the scene risk correction strategy according to the first synergistic positive correlation relationship reflected by the scene risk feature information of the scene category; wherein in the first synergistic positive correlation relationship, the scene risk correction strategy increases in response to an increase in the scene risk feature information.

[0125] The calculation formula of the scene risk correction strategy is as follows.

[0126]

[0127] wherein, is the scene risk feature information of the current scene category, is the scene risk correction strategy.

[0128] In the above embodiments, the scene risk feature information is associated with a specific target user, so that scene risk feature information corresponding to the user can be obtained, thereby ensuring that the evaluation of the user's sensory stimulation state can be based on the user's personalized information, making the evaluation result more accurate. And the scene risk correction strategy is obtained through the first synergistic positive correlation relationship, which emphasizes the synergistic effect between multiple factors or variables, and jointly promotes the realization of a certain target or result. The scene risk correction strategy obtained through this relationship considers the relevance and synergy between multiple dimensional data, making the construction of the scene risk correction strategy more accurate.

[0129] Currently, in the intervention means for ASD (Autism Spectrum Disorder) children in the outdoor scene adaptation, although various methods including cognitive behavior training have been introduced, there are still obvious limitations in the actual application process, such as weak correlation between training scene and real environment. The training scene used by the existing intervention means is mostly preset or general environment, which lacks dynamic correlation with the destination that the children will actually go to, resulting in a disconnection between the training content and the real scene, and it is difficult to achieve effective adaptive transfer. For example, there is a lack of individualized adaptation mechanism, and the individualized stimulation configuration and dynamic adjustment cannot be made according to the sensory hypersensitivity characteristics (such as abnormal sensitivity to specific sound and light) of ASD children, the training content lacks pertinence and adaptability, and it is difficult to accurately simulate the challenges that individuals may face in real scenes. For example, the training process lacks a quantitative driven progressive logic, and existing methods mostly rely on mechanical content replacement or fixed duration training phase advancement, and fail to construct a continuous and quantifiable evaluation system based on physiological indicators, behavior performance and other multi-dimensional data, resulting in a lack of dynamic matching between training difficulty and real-time adaptation ability of children, and making it difficult to achieve real "step by step" adaptation ability improvement.

[0130] Therefore, the existing intervention methods have deficiencies in scene authenticity, individual adaptability and process scientificity, which restrict the effectiveness and safety of ASD children's outdoor adaptation training. There is an urgent need for a systematic method that can be based on real scene modeling, integrate individual characteristics, and dynamically adjust the training process according to objective data, in order to improve the pre-adaptation training effect of ASD children before going out.

[0131] The method provided by the application is a progressive training method pre-adapted to a real scene. The method is a system and method for evaluating and training the destination adaptability of ASD children based on VR technology. The core goal is to build a training carrier highly consistent with the actual travel scene through multi-version VR scene modeling driven by real environment. Through multi-modal data-driven progressive training, dynamic matching of training difficulty and child adaptability is achieved. Through quantitative evaluation and risk warning, potential discomfort risks are identified in advance and intervention basis is provided. In the training process, not only is the VR device adaptability correction link added to establish the adaptation benchmark of the device and the child's physiological stable state, but also different intensity control is implemented for different environmental interference hypersensitivity items such as sound, light and touch in combination with clinical information to realize personalized adaptation of training stimuli. Finally, the adaptability and social participation willingness of ASD children to specific real outdoor environments are gradually improved, the stress response in real scenes is reduced, and safe travel is ensured.

[0132] In one specific embodiment, referring to Figure 6 A flowchart based on hierarchical modeling and progressive training is provided in one embodiment of the application. The method provided by the application is a progressive training method pre-adapted to a real scene. The method is a multi-level progressive virtual scene modeling process based on real scene modeling, and the flow includes the following steps.

[0133] 1) Perform two times of basic adaptation training in stage 1 in the basic training scene;

[0134] 2) Determine whether the switching condition from stage 1 to stage 2 is met, if yes, perform step 3), if not, perform step 14);

[0135] 3) Perform two times of low-intensity training in stage 2 in the simplified scene;

[0136] 4) Determine whether the switching condition from stage 2 to stage 3 is met, if yes, perform step 6), if not, perform step 5);

[0137] 5) Determine whether there is a rollback record in stage 2, if yes, perform step 14), if not, perform step 3);

[0138] 6) Obtain the secondary correction value of the user's device fitness, and re-correct the upper limit of the physiological stability feature interval. The secondary correction value of the fitness The calculation formula is as follows:

[0139]

[0140] Wherein is the relative power ratio of the alpha band when the user wears the VR device in the resting state when the phase 2 is switched to the phase 3. is the relative power ratio of the alpha band when the user does not wear the VR device in the resting state obtained in the personalized threshold adaptation step;

[0141] 7) Perform three times of medium-intensity training in phase 3 in the close-to-display version scene;

[0142] 8) Determine whether the switching condition from phase 3 to phase 4 is met, if yes, execute step 10), if not, execute step 9);

[0143] 9) Determine whether there is a rollback record in phase 3, if yes, execute step 14), if not, execute step 7);

[0144] 10) Perform three times of high-intensity training in phase 4 in the completely real version scene;

[0145] 11) Determine whether the condition of "training up to standard and ending training" is met, if yes, execute step 13), if not, execute step 12);

[0146] 12) Determine whether there is a rollback record in phase 4, if yes, execute step 14), if not, execute step 10);

[0147] 13) End the training and push "training passed, can travel" to the guardian;

[0148] 14) End the training and push the destination risk warning to the guardian.

[0149] In the above embodiments, the dynamic VR modeling based on real environment information is different from the existing VR training based on preset virtual scenes. The application dynamically constructs a highly realistic VR training environment using real-time or near real-time environment information (including images, videos, audio, etc.) of the target destination, and introduces common dynamic stressors (such as sudden high-decibel noise and strong flickering light) and typical social situations (such as shopping and queuing) in this environment category that may cause discomfort in ASD children. The adaptive quantitative evaluation and early warning of multi-modal fusion synchronously collects various physiological signals (EEG, EOG, GSR) and behavioral information (voice, facial expression, action) of the child during the VR training process; by extracting and jointly analyzing these multi-modal features, the system can objectively and quantitatively evaluate the child's adaptability, integration level, and potential discomfort level in the simulated environment; when the analysis results indicate a significant discomfort risk, the system can issue a preventive prompt before the actual outing to guide the caregiver to take appropriate measures. The gradual training mode management divides the VR training into multiple progressive stages based on the child's multi-modal data quantitative performance, sets clear quantitative indicators (physiological stability, task completion, social cooperation) for each stage, and dynamically triggers "progression / rollback" based on the child's continuous training performance, achieving precise matching of training difficulty and the child's adaptability.

[0150] The system realizes a closed loop of "personalized threshold adaptation, real scene hierarchical modeling, gradual training, data collection, quantitative evaluation, and risk warning" through the collaborative work of the six core modules. The scene realism and adaptability are significantly improved, multi-version VR scenes are generated based on real environment data, stressors are strongly associated with scenes, solving the "scene disconnection" problem of existing training, and four-level scene version design adapts to different training stages. Training efficiency and safety are guaranteed, gradual training and dynamic switching mechanism avoid resistance or ineffective training caused by difficulty mismatch, and the rollback mechanism reduces training risk and improves the child's adaptability to the target environment. The objectivity of evaluation and early warning, multi-modal data-driven quantitative evaluation replaces subjective judgment, and quantitative control of VR scene version and training stage switching. The system considers both universality and individuality, supports multiple types of outing scenes such as supermarkets, hospitals, and parks, and can adjust the environmental comfort threshold and stable physiological interval based on the child's historical behavior information and clinical data to adapt to the individual needs of different children.

[0151] In another embodiment, the application also provides a gradual training device pre-adapted to a real scene, which comprises:

[0152] A modeling module is configured to obtain scene information of a real scene, perform element hierarchical decomposition on the scene information, perform gradual hierarchical modeling on different hierarchical elements after decomposition, and construct a multi-level virtual scene simulating different hierarchical real environments.

[0153] The multi-modal data synchronous acquisition module is configured to acquire heterogeneous feature expressions of a user when the user wears a training device and performs pre-adaptation in the multi-level virtual scene.

[0154] The multi-modal feature fusion analysis module is configured to calculate sensing parameters and / or interaction parameters of the user according to the heterogeneous feature expressions.

[0155] The training interaction module is configured to dynamically switch levels of the multi-level virtual scene according to the sensing parameters and / or the interaction parameters of the user, and perform pre-adaptation training in the multi-level virtual scene of the switched level.

[0156] Referring to Figure 7 The device can further include a personalized threshold adaptation module, by which VR device adaptability correction is achieved. The modeling module can be a VR modeling module configured to construct a multi-level progressive virtual scene. The training interaction module can be a VR interaction module configured to perform training in the virtual scene of different levels and acquire training record data. Multi-modal heterogeneous data of the user during training is acquired by the multi-modal data synchronous acquisition module, and the data is analyzed by the multi-modal feature fusion analysis module to evaluate the training state of the user and further switch the adaptive training level. Furthermore, the evaluation result output and the pre-warning module can be used to prompt risk pre-warning and feedback of stage labeling progress.

[0157] For a detailed description of the progressive training device pre-adapted to a real scene provided by the present application, please refer to the above embodiment of the progressive training method pre-adapted to a real scene. The present application will not be repeated here.

[0158] The present application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the progressive training method pre-adapted to a real scene described in any of the above embodiments.

[0159] The present application also provides a computer program product including a computer program or instructions, which are executed by a processor to implement the steps of the progressive training method pre-adapted to a real scene described in any of the above embodiments.

[0160] It can be understood that the computer device provided by the present application can be a server, and its internal structure diagram can be as shown in Figure 8As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is used to store related data. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement the method provided in the present application.

[0161] Those skilled in the art can understand that, Figure 8 The skilled in the art can understand that, The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the flow of each method embodiment as described above. Among them, any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limited to, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0162] It should be appreciated that the processor mentioned in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0163] The technical features of the above embodiments can be combined in any manner. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present disclosure.

[0164] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for those skilled in the art, some modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A progressive training method for pre-adapting to real-world scenarios, characterized in that, The method includes: Obtain scene information from the real-world scenario; The scene information is split into elements at different levels, and progressive hierarchical modeling is performed on the elements at different levels after splitting to construct a multi-level virtual scene that simulates different levels of real environment. The heterogeneous feature expression of the user during pre-adaptation in the multi-level virtual scene is obtained. The level of the multi-level virtual scene is dynamically switched according to the heterogeneous feature expression. In the multi-level virtual scene after the switch, the user is trained to undergo progressive virtual sensory stimulation training to adapt to the real scene.

2. The method according to claim 1, characterized in that, The step of acquiring heterogeneous feature representations of users during pre-adaptation in the multi-level virtual scene, and dynamically switching the levels of the multi-level virtual scene based on the heterogeneous feature representations, includes: Acquire heterogeneous feature representations of users during pre-adaptation in the current level of the virtual scene; Based on the sensing parameters and interaction parameters in the heterogeneous feature representation, the user's fitness level in the current level of the virtual scene is determined, and the level of the virtual environment in which the user is located is dynamically switched according to the fitness level.

3. The method according to claim 2, characterized in that, The step of determining the user's adaptability in the current level of the virtual scene based on the sensing parameters and interaction parameters in the heterogeneous feature representation includes: Determine the environmental adaptability corresponding to the user's physiological stability based on the user's sensor parameters and / or interaction parameters; Determine the stress response level corresponding to the user's behavioral performance based on the user's sensor parameters and / or interaction parameters; The task completion rate corresponding to the quality of the user's task completion is determined based on the user's sensor parameters and / or interaction parameters. Determine the stability of the index corresponding to the user stability verification based on the user's sensing parameters and / or interaction parameters; Determine the care assessment level corresponding to the collaboration between the user and the caregiver based on the user's sensor parameters and / or interaction parameters; The user's adaptability in the current level of the virtual scene is determined based on at least one of the following: environmental adaptability, stress response, task completion, indicator stability, and care assessment.

4. The method according to claim 2, characterized in that, The step of determining the user's adaptability in the current level of the virtual scene based on the sensing parameters and interaction parameters in the heterogeneous feature representation, and determining the user's level adaptability based on the adaptability, includes: If the user's fitness level in the current virtual scene, as determined by the sensing parameters and / or interaction parameters, meets the switching conditions, then the user is switched from the current virtual scene to the next level virtual scene. This allows the user to undergo virtual sensory stimulation training in the next level virtual scene to adapt to the real scene. The virtual scene level is iteratively adjusted based on the user's fitness level until the user completes virtual sensory stimulation training in the highest level virtual scene to adapt to the real scene or the training is terminated. If the user's adaptation level in the current virtual scene, as determined by the sensing parameters and / or interaction parameters, does not meet the switching conditions, the system continues to check whether the user has a rollback record that has reached the preset maximum number of rollbacks in the current virtual scene. If so, a risk warning is issued or training is terminated. If not, the user continues to undergo virtual sensory stimulation training in the current virtual scene to adapt to the real scene, and the level of the virtual scene is iteratively adjusted according to the user's adaptation level until the user completes virtual sensory stimulation training in the highest level virtual scene to adapt to the real scene or training is terminated.

5. The method according to claim 1, characterized in that, The process of splitting the scene information into element-level hierarchical segments, and then performing progressive hierarchical modeling based on the elements at different levels to construct a multi-level virtual scene simulating different levels of real-world environments includes: The environmental elements and social context elements in the scene information are split into elements according to their levels to obtain environmental elements and social context elements at different levels. Virtual scenes of corresponding levels are constructed using environmental elements and social context elements at different levels; wherein, virtual scenes of different levels are used to progressively simulate real scenes, and as progressive training progresses, the virtual scenes continuously approach the real scenes.

6. The method according to claim 5, characterized in that, The construction of corresponding levels of virtual scenes using environmental elements and social context elements at different levels includes: When a user is first placed in a virtual scene at the key level for pre-adaptation to a real scene for virtual sensory stimulation training, no sensory hypersensitivity items are set for the user in the environmental elements at the key level during the construction of the virtual scene at the key level. When a user is not in a virtual scene at the key level for the first time to conduct virtual sensory stimulation training to adapt to the real scene, the user's sensory hypersensitivity items are dynamically set in the environmental elements of the key level to construct the virtual scene at the key level, wherein the key level is the second-highest level.

7. The method according to claim 3, characterized in that, The calculation steps for the physiological stability characteristic range in the environmental adaptability include: The user's first parameter in the current scenario is obtained based on the safety environment standard. The first parameter includes the environmental comfort reference value. The scenario risk correction weight corresponding to the current scenario category is obtained. The first parameter is adjusted according to the scenario risk correction strategy, the scenario risk correction weight and the sensitivity correction strategy to obtain the environmental critical conditions. Obtain the user's stable physiological characteristics in the current scenario category and current activity state, and use the scenario risk correction strategy to correct the stable physiological characteristics to obtain the user's stable physiological characteristic range. The user's electroencephalogram (EEG) signals are collected, and the user's fitness correction value for the training device is calculated based on the EEG signals. The physiological stability feature interval is then corrected based on the fitness correction value.

8. The method according to claim 7, characterized in that, The process of collecting the user's electroencephalogram (EEG) signals, calculating the user's fitness correction value for the training device based on the EEG signals, and correcting the physiological stability feature interval based on the fitness correction value includes: Collect resting-state EEG signals from users when they are not wearing training devices, and calculate the first set of characteristic parameters; Collect resting-state EEG signals of children after they wear the training device, and calculate the second set of characteristic parameters; The user's fitness correction value for the training device is determined based on the difference between the first set of feature parameters and the second set of feature parameters. A scaling factor is determined based on the fitness correction value, and the physiological stability feature interval is corrected based on the scaling factor.

9. The method according to claim 7, characterized in that, The method further includes: If at least one of the physiological stability characteristic interval, the stress response degree, the task completion degree, the indicator stability degree, and the care assessment degree triggers the high-risk threshold of the corresponding level of the virtual environment, the virtual sensory stimulation training for pre-adaptation to the real scene is terminated.

10. A progressive training device pre-adapted to real-world scenarios, characterized in that, The device includes: The modeling module is used to acquire scene information of real scenes, split the scene information into elements at different levels, and perform progressive hierarchical modeling based on the elements at different levels after splitting to construct multi-level virtual scenes that simulate real environments at different levels. The multimodal data synchronous acquisition module is used to acquire heterogeneous feature representations when a user wears a training device and performs pre-adaptation in the multi-level virtual scene; The multimodal feature fusion analysis module is used to calculate the user's sensing parameters and / or interaction parameters based on the heterogeneous feature expression; The training interaction module is used to dynamically switch the levels of a multi-level virtual scene based on the user's sensor parameters and / or interaction parameters, and to perform pre-adaptive training in the multi-level virtual scene after the switch.

Images