An autism behavior recognition and training guidance method and system

Abstract

The application discloses a kind of autism behavior recognition and training guide method and system.The application acquisition module acquires the multi-source behavior data of measured object, generates the availability mark and quality evaluation result of each modality, and extracts the structured feature related to behavior recognition from multi-modal data;Individualized reference model is constructed based on historical behavior data, and the deviation degree of current feature relative to historical behavior mode is calculated;Multi-modal features are fused and behavior recognition results are output;According to the recognition result, the corresponding training task or interactive guide content is generated;The behavior change after training guide is evaluated, and the model parameters are updated.The application takes behavior recognition as the core output, and takes training guide as the subsequent application link.Training guide is used to assist early intervention training, realizes multi-modal acquisition, feature extraction, fusion recognition and training guide for autism-related behaviors.

Description

Technical Field

[0001] This invention belongs to the fields of intelligent sensing, behavior computing and human-computer interaction technology, and specifically relates to a method and system for multimodal data collection, feature extraction, fusion recognition and training guidance of behaviors related to autism spectrum disorder based on mobile terminals and / or wearable devices. Background Technology

[0002] Children with autism spectrum disorder often exhibit behavioral characteristics such as insufficient social attention, eye avoidance, repetitive and stereotyped behaviors, abnormal emotional reactions, and insufficient interactive responses during their development. Currently, the assessment of these behaviors still relies heavily on manual observation, scale evaluations, or single-scene tests. These methods suffer from strong subjectivity, insufficient continuity, and limited coverage of natural scenarios, making it difficult to conduct long-term, objective, and quantitative analysis of behavioral changes in children at home, in the classroom, or during training.

[0003] Objective identification of autism-related behaviors is of great significance. On the one hand, behavioral identification allows for continuous monitoring of core behavioral manifestations such as stereotyped movements, insufficient eye contact, abnormal vocalizations, and interaction avoidance, providing a basis for early detection of abnormal behavioral trends. On the other hand, the identification results can be further used to generate appropriate training tasks, such as attention guidance, eye following, facial expression imitation, and interactive response training content, thereby assisting parents, teachers, or rehabilitation personnel in carrying out more targeted early intervention training, and assessing the training effect by observing changes in behavior before and after training.

[0004] In existing technologies, some solutions have begun to combine audio-visual analysis or human-computer interaction to assist in autism screening or training. For example, patent CN110363129A discloses an early autism screening system based on a smiling paradigm and audio-visual behavior analysis. This solution collects multi-view, multi-channel audio-visual data, extracts features such as facial features, orientation, gaze, and gestures, and uses a classification model to assess autism risk. Such solutions help improve the standardization of the screening process, but they mainly revolve around specific screening paradigms and are more suitable for structured, induced scenarios. Their support for continuous behavior recognition in natural scenarios, adaptation to individual differences, and subsequent training guidance remains limited.

[0005] For example, patent CN113096805A discloses an autism emotion recognition and intervention system. This system sets up teaching mode, game mode, and reinforcement training mode, collects attention, facial expression, and performance data, and gradually advances the training process based on the analysis results. While this type of system can support the organization and effectiveness analysis of training activities, its focus is on emotion recognition and facial expression imitation training. The unified identification of various behaviors such as stereotyped movements, eye avoidance, and abnormal vocalizations, as well as the data processing link and multimodal feature fusion mechanism at the identification front end, are not yet fully developed.

[0006] For example, patent CN120832581A discloses a method, system, and medium for identifying autistic behaviors in children based on data analysis. This solution collects multimodal data such as sensor data, positioning data, audio data, and video data, extracts nonlinear time-series features, and combines three-dimensional convolutional networks, temporal convolutional networks, and multi-classifiers for behavior recognition and severity assessment. While this type of solution improves multimodal recognition capabilities, it focuses more on behavior classification within a group environment, and there is still room for improvement in individual dynamic baselines, scene reliability compensation, and the training guidance loop driven by recognition results.

[0007] In summary, existing technologies have at least the following shortcomings: First, most solutions rely heavily on a single scene or task, making it difficult to simultaneously recognize multiple behavioral signals in natural scenes; second, existing feature processing is mostly focused on single-modal analysis or static statistics, failing to fully integrate temporal features, frequency domain features, spatial pose features, and individual baseline features; third, significant differences in behavioral baselines exist among different children, and fixed thresholds or uniform models can easily lead to false positives or false negatives; fourth, even if existing solutions can perform multimodal behavior recognition, they have not fully disclosed the closed-loop processing mechanism that scores candidate behavioral events separately based on available modal quality assessment results and updates individual baselines and task configurations under quality gating and anomaly exclusion conditions.

[0008] Therefore, there is an urgent need for a method and system for identifying and training autism-related behaviors. Starting from multimodal data collection, this system should perform structured feature extraction and fusion processing on visual, audio, motion, and interactive data. It should also implement auxiliary labeling of behavioral events while considering individual differences and scene changes. Furthermore, it should generate human-computer interaction task content based on the behavioral event labeling results to support the early recording, continuous statistics, and individualized training of observable behavioral events related to autism spectrum disorder. Summary of the Invention

[0009] The purpose of this invention is to provide a method and system for autism behavior recognition and training guidance, which solves the problems of insufficient recognition of natural scene behavior, weak adaptability to individual differences, insufficient utilization of multimodal features, and disconnect between recognition results and training process in the existing technology, and realizes multimodal collection, feature extraction, fusion recognition and training guidance of autism-related behaviors.

[0010] This invention takes behavior recognition as its core output and training guidance as its subsequent application. The training guidance is used to assist in early intervention training and does not involve the treatment steps themselves.

[0011] To achieve the above objectives, the present invention provides an autism behavior recognition and training guidance system, including a multimodal data acquisition module, a preprocessing and alignment module, a feature extraction module, an individual baseline modeling module, a fusion recognition module, a training guidance module, and an effect evaluation and update module;

[0012] The multimodal data acquisition module is used to acquire multi-source behavioral data of the object under test; the preprocessing and alignment module is used to remove invalid data segments, correct the time deviation between different modes, form a multimodal data sequence under a unified time series, and generate availability tags and quality evaluation results for each mode. The feature extraction module extracts structured features related to behavior recognition from preprocessed multimodal data; the individual baseline modeling module builds an individualized reference model based on historical behavior data, establishing statistical distributions, trend models, or individual baseline vectors for various features according to preset time periods, scene categories, or training task categories, and calculating the deviation of the current feature from historical behavior patterns to characterize the intensity of abnormal behavior or the magnitude of behavior change; the fusion recognition module fuses multimodal features and outputs behavior recognition results, specifically: based on the quality evaluation results of each modality, structured features, and the deviation of individual baselines, it performs weighted fusion, feature dimensionality reduction, temporal modeling, and classification discrimination on structured features, outputting recognition results for one or more target behaviors; the training guidance module generates corresponding training tasks or interactive guidance content based on the recognition results; and the effect evaluation and update module evaluates the behavior changes after training guidance and updates the model parameters.

[0013] Furthermore, the multimodal data acquisition module includes one or more of a camera, a microphone, an inertial sensor, and an interactive acquisition unit;

[0014] The camera is used to capture facial images, head posture images, limb movement images, and gaze-related images; the microphone is used to capture vocalization, speech rhythm, and environmental sound data; the inertial sensor is used to capture acceleration data, angular velocity data, and motion trajectory data; and the interactive acquisition unit is used to capture touch, click, pause, swipe, or task operation records.

[0015] The aforementioned devices can also be used to collect scene context data such as brightness, noise intensity, device posture, or position information.

[0016] Furthermore, the feature extraction module includes a visual feature extraction submodule, an audio feature extraction submodule, a motion feature extraction submodule, and an interaction feature extraction submodule. The visual feature extraction submodule is used to extract facial key points, head posture, gaze direction, gaze duration, facial expression changes, limb key point trajectories, and movement amplitude features. The audio feature extraction submodule is used to extract sound intensity, fundamental frequency, spectral distribution, speech rate, pauses, prosodic changes, and speech activity features. The motion feature extraction submodule is used to extract acceleration, angular velocity, motion intensity, periodicity, frequency band energy, trajectory changes, and repetitive motion features. The interaction feature extraction submodule is used to extract click repetition rate, dwell time, swipe pattern, operation sequence, and task completion process features.

[0017] Furthermore, the effect evaluation and update module evaluates the training guidance effect based on indicators such as the improvement of eye contact, the decrease in abnormal behavior, the task completion rate, the response latency, and the number of consecutive successes during or after training. The evaluation results are then fed back to the individual baseline modeling module and the fusion recognition module to update the threshold, weight, task difficulty, or model parameters.

[0018] Furthermore, the system also includes a remote collaboration module, which is used to send behavior recognition results, training records and trend summaries to the parent or teacher end to assist in training management and periodic assessment.

[0019] Based on the above system, the following methods for identifying and guiding autistic behaviors are proposed, specifically including the following steps:

[0020] Visual, audio, motion, and interaction data of the tested object are collected to form a raw multimodal dataset. The raw multimodal dataset is preprocessed and time-aligned to generate multimodal analysis data, and the quality of each modality is evaluated. Visual, audio, motion, and interaction features are extracted from the multimodal analysis data to construct a structured feature vector for the current time window. An individual baseline model is established based on historical behavior data, and the deviation of the current structured features from the individual baseline is calculated. Modal reliability parameters are generated by combining factors such as ambient brightness, audio signal-to-noise ratio, and sensor stability. The structured features, individual baseline deviation, and modal reliability parameters are input into a fusion recognition model to output recognition results for one or more target behaviors. Training guidance tasks are automatically generated based on the recognition results, and corresponding visual, audio, or interactive training content is output on the terminal interface. Feedback data during the training process is collected to quantitatively evaluate the training effect, and the individual baseline, recognition parameters, and training task configuration are updated based on the evaluation results.

[0021] Furthermore, the preprocessing and alignment module performs quality inspection, noise reduction, segmentation, standardization, and time synchronization processing on the multimodal data.

[0022] Furthermore, the target behavior includes one or more of the following: stereotyped actions, eye contact avoidance, abnormal vocalizations, interaction avoidance, and emotionally agitated behaviors;

[0023] Furthermore, the identification results include behavior labels, behavior confidence, behavior duration, behavior frequency, and / or behavior level.

[0024] Furthermore, the training guidance module can output one or more of the following tasks: visual attention guidance task, eye-following task, action imitation task, facial expression imitation task, interactive response task, rhythm soothing prompt, or task switching prompt;

[0025] The type, difficulty, duration, presentation order, and feedback method of the training tasks are adaptively adjusted based on the behavior recognition results, individual baseline, and historical training performance to achieve individualized training guidance for early intervention.

[0026] Compared with the prior art, the present invention has at least the following beneficial effects:

[0027] Firstly, this invention acquires behavioral data from multiple dimensions such as vision, audio, motion, and interaction, and improves the consistency of multi-source data through time alignment, quality assessment, and unified modeling, so that behavior recognition no longer depends on a single modality, thereby improving the recognition stability in complex scenarios.

[0028] Secondly, by extracting key point trajectories, eye contact features, rhythmic variation features, periodic movement features, repetitive operation features, and individual baseline deviation features, this invention achieves multi-level representation of autism-related behaviors. Compared with schemes that only use simple statistics or single-scene scoring, it is more conducive to improving the accuracy and interpretability of behavior recognition.

[0029] Third, this invention introduces individual baseline modeling and scene reliability correction mechanisms, which can dynamically adjust feature weights and discrimination results according to the behavioral differences of different children and different environmental conditions, thereby reducing false alarms and false negatives caused by fixed thresholds.

[0030] Fourth, this invention directly uses the behavior recognition results to generate training tasks, and updates the recognition parameters and training strategies through training feedback, forming a quantifiable closed-loop processing process, which is beneficial for providing continuous early intervention training support for parents, teachers or rehabilitation personnel. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments are briefly described below. It should be understood that the following drawings only show some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0032] Figure 1 This is a block diagram of the overall structure of an autism behavior recognition and training guidance system according to the present invention;

[0033] Figure 2 This is a flowchart illustrating a method for identifying and guiding autistic behaviors according to the present invention.

[0034] Figure 3 This is a schematic diagram of the process of multimodal data preprocessing, feature extraction, and individual baseline modeling in this invention;

[0035] Figure 4This is a schematic diagram of the multimodal fusion recognition and dynamic weight adjustment process in this invention;

[0036] Figure 5 This is a schematic diagram of the closed-loop process of training guidance, effect evaluation, and parameter update in this invention. Detailed Implementation

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. For those skilled in the art, equivalent substitutions or modifications made to the parameters, module combinations, data processing order, or training task styles in the embodiments without departing from the concept of the present invention should all fall within the scope of protection of the present invention.

[0038] I. System Overall Structure and Implementation Method

[0039] like Figure 1 As shown, this embodiment provides an autism behavior recognition and training guidance system. The system can be deployed on a mobile terminal, tablet terminal, smart wearable device, or a combination of both, and can optionally communicate with a remote server. The system includes at least a multimodal data acquisition module 1, a preprocessing and alignment module 2, a feature extraction module 3, an individual baseline modeling module 4, a fusion recognition module 5, a training guidance module 6, and an effect evaluation and update module 7.

[0040] in:

[0041] A multimodal data acquisition module 1 is used to collect behavioral data of the tested object in natural or training scenarios. The multimodal data acquisition module 1 may include a camera, microphone, accelerometer, gyroscope, touch acquisition unit, and optional brightness sensor, noise estimation unit, and position information acquisition unit. The camera is used to acquire facial images, head posture images, eye images, and upper limb or torso movement images; the microphone is used to acquire vocalization, speech prosody, and environmental sounds; the accelerometer and gyroscope are used to acquire information on action intensity, periodic motion, and posture changes; the touch acquisition unit is used to acquire data on clicks, swipes, pauses, repetitive operations, and task completion processes.

[0042] The preprocessing and alignment module 2 is used to perform denoising, segmentation, standardization, time synchronization, and modal quality assessment on the acquired multimodal raw data. Specifically, the preprocessing and alignment module 2 removes blurred frames, severely occluded frames, and low-confidence frames from the visual data; performs frame segmentation, endpoint detection, noise suppression, and speech activity detection on the audio data; performs filtering, drift removal, and wearing stability detection on the motion data; and performs event merging and timestamp unification processing on the interactive data, thereby obtaining time-aligned multimodal analysis data.

[0043] The feature extraction module 3 is used to extract structured features for each modality. The feature extraction module 3 includes at least a visual feature extraction submodule 31, an audio feature extraction submodule 32, a motion feature extraction submodule 33, and an interaction feature extraction submodule 34. The visual feature extraction submodule 31 extracts facial key points, eye key points, head Euler angles, gaze direction, gaze duration, facial expression change amplitude, limb key point trajectory, and movement speed features. The audio feature extraction submodule 32 extracts features such as fundamental frequency, short-time energy, Mel frequency cepstral coefficients, speech rate, pause ratio, spectral centroid, and peak energy ratio. The motion feature extraction submodule 33 extracts features such as acceleration modulus, angular velocity modulus, motion mean, variance, dominant frequency, frequency band energy, periodic stability, and trajectory repeatability. The interaction feature extraction submodule 34 extracts features such as click frequency, repeated click ratio, dwell time, number of accidental touches, operation sequence deviation, and task completion rate.

[0044] The individual baseline modeling module 4 is used to establish individual historical behavior baselines for different test subjects. The individual baseline modeling module 4 statistically analyzes historical behavioral characteristics according to preset time periods, task categories, or scenario categories to obtain individual mean, variance, trend vector, or baseline intervals; and during current behavior analysis, it calculates the degree of deviation of the current characteristics from the historical baseline to characterize the intensity of abnormal behavior and the trend of behavioral changes.

[0045] The fusion recognition module 5 is used to perform fusion calculations on visual, audio, motion, and interaction features, and output the target behavior recognition result. Based on modal availability, scene quality, individual baseline deviation, and temporal evolution characteristics, the fusion recognition module 5 performs weighting, dimensionality reduction, temporal modeling, and classification on the multimodal features, outputting behavior labels, behavior confidence, behavior frequency, behavior duration, and behavior level. The target behavior may include one or more of the following: stereotyped actions, eye contact avoidance, abnormal vocalizations, interaction avoidance, and emotionally agitated behaviors.

[0046] The training guidance module 6 is used to generate training tasks that match the behavior based on the behavior recognition results. The training guidance module 6 can call the display unit, speaker, and interactive interface to output attention guidance tasks, eye-following tasks, action imitation tasks, facial expression imitation tasks, rhythm following tasks, interactive rotation tasks, and prompt feedback information to assist in early intervention training.

[0047] The effect evaluation and update module 7 is used to collect feedback data after training guidance and evaluate the training effect. The effect evaluation and update module 7 quantifies the training task effect based on the increase in eye contact, the decrease in abnormal behavior, the task success rate, response latency, the number of consecutive successes, and the change in behavior score, and updates the individual baseline parameters, modality weights, recognition thresholds, and training difficulty level.

[0048] The remote collaboration module 8 is an optional module used to send recognition results, training task execution records, and trend reports to the parent or teacher's end for viewing and subsequent management.

[0049] II. Method Flow and Implementation

[0050] like Figure 2 As shown, using the above-described device, the present invention provides a method for identifying and guiding autistic behaviors, specifically including the following steps:

[0051] S101. Collect visual data, audio data, motion data, and interaction data of the tested object to form an original multimodal dataset;

[0052] S102. Preprocess and time-align the original multimodal dataset to generate multimodal analysis data, and evaluate the quality of each modality;

[0053] S103. Extract visual features, audio features, motion features, and interaction features from the multimodal analysis data, and construct a structured feature vector under the current time window;

[0054] S104. Establish an individual baseline model based on historical behavioral data, calculate the deviation of the current structured features from the individual baseline, and generate modal reliability parameters by combining factors such as ambient brightness, audio signal-to-noise ratio, and sensor stability.

[0055] S105. Input the structured features, individual baseline deviations, and modal reliability parameters into the fusion recognition model, and output the recognition results of one or more target behaviors;

[0056] S106. Automatically generate training guidance tasks based on the recognition results, and output the corresponding visual, audio or interactive training content in the terminal interface.

[0057] S107. Collect feedback data during the training process, quantitatively evaluate the training effect, and update the individual baseline, identification parameters, and training task configuration based on the evaluation results.

[0058] In this embodiment, steps S101 to S107 form a closed-loop process of "collection-processing-identification-training-evaluation-update", so that the identification results can not only be used for behavior judgment, but also further drive the individualized adjustment of training tasks.

[0059] III. Implementation Methods for Multimodal Data Acquisition

[0060] The multimodal data acquisition module 1 is deployed on mobile terminals and wearable terminals. The mobile terminal can be a mobile phone or tablet device, and the wearable terminal can be a smartwatch or wristband device.

[0061] The camera can be a front-facing camera or an external camera, with a sampling resolution preferably of 640×480 or 1280×720, and a frame rate preferably between 15 and 30 frames per second. The microphone sampling rate is preferably 16 kHz or 48 kHz. The accelerometer and gyroscope sampling rates are preferably not less than 50 Hz, and more preferably 100 Hz. The touch capture unit records the timestamp, coordinates, and control identifier for each click, swipe, long press, and pause.

[0062] To balance power consumption and real-time performance, the system employs a low-power wake-up mechanism. By default, only motion sampling and interactive event listening are maintained; when increased periodicity of movement, a higher frequency of repeated clicks, or the start of a preset training task are detected, the camera and microphone are woken up to enter full acquisition mode. Periodicity of movement can be detected by detecting the main frequency within a short time window; repeated clicks can be determined by the click ratio in the same area or the number of times the same control is repeatedly triggered.

[0063] IV. Preprocessing and Alignment Implementation Methods

[0064] like Figure 3 As shown, in step S102, the preprocessing and alignment module 2 first assigns a unified timestamp to the raw data of each modality and uses a monotonic clock as the alignment reference. Visual frames, audio frames, inertial sampling points, and interactive events are all mapped to a unified time axis. The time alignment error between different modalities is preferably controlled within 200ms, and more preferably within 100ms.

[0065] For visual data, face detection models and keypoint detection models can be used to analyze the sampled images frame by frame. When the face confidence is lower than a preset threshold, the corresponding frame is marked as a low-reliability frame; when the number of consecutive low-reliability frames exceeds a preset number, the fusion weight of the visual modality is reduced.

[0066] For audio data, a 25ms window length and a 10ms frame shift can be used for framing, followed by pre-emphasis, windowing, and spectral analysis. Furthermore, pure noise frames can be removed based on a speech activity detection algorithm, and audio quality parameters can be obtained through signal-to-noise ratio estimation.

[0067] For inertial data, low-pass filtering, median filtering, or Kalman filtering can be applied to the triaxial acceleration and triaxial angular velocity to reduce high-frequency jitter and occasional noise. For abnormal wearing conditions, such as prolonged stillness, sudden changes in posture, or significant detachment, a sensor instability flag can be output.

[0068] For interactive data, events can be aggregated within a preset time window. For example, continuous clicks can be grouped into click sequences, continuous swipes can be grouped into swipe segments, and interactive behavior time slices can be generated.

[0069] V. Feature Extraction Implementation Methods

[0070] 1. Visual Feature Extraction Implementation Method

[0071] The visual feature extraction submodule 31 extracts facial and eye keypoints from each frame of the image and estimates the head pose and gaze direction based on the keypoint positions. Head pose can be represented as pitch, yaw, and roll angles; gaze direction can be represented as a two-dimensional or three-dimensional vector. The gaze deviation angle is calculated based on the angle between the gaze direction and the center direction of the screen. If the gaze direction vector is denoted as g, and the screen or target direction vector is denoted as c, then the gaze deviation angle can be expressed as: .

[0072] Within a preset statistical window, the cumulative time during which the gaze deviation angle is less than the contact threshold is used to obtain the gaze contact duration Tcontact. Furthermore, the number of gaze deviation events, deviation duration, and number of return glances can be extracted to reflect gaze avoidance or gaze stability.

[0073] In addition, the visual feature extraction submodule 31 can also extract the trajectories of key points such as shoulders, elbows, and wrists through the pose estimation model, and calculate the movement speed, movement amplitude, trajectory repetition rate and local limb activity density to characterize stereotyped movements and the completion of imitation.

[0074] 2. Audio Feature Extraction Implementation Method

[0075] The audio feature extraction submodule 32 extracts 13-dimensional Mel-frequency cepstral coefficients, fundamental frequency F0, short-time energy, zero-crossing rate, and spectral centroid from audio frames, and calculates the mean, standard deviation, peak-to-peak ratio, and rate of change within a sliding time window. For abnormal vocalizations, emotional fluctuations, or insufficient interactive responses, the following features can be used for joint judgment: fundamental frequency variance, short-time energy peak-to-peak ratio, speech activity ratio, continuous pause length, number of vocal bursts, and response delay after external speech stimuli. The response delay can be defined as the time difference between the issuance of the training prompt and the first effective vocalization.

[0076] 3. Implementation Methods for Motion Feature Extraction

[0077] Motion feature extraction submodule 33 calculates the motion magnitude sequence based on acceleration and angular velocity:

[0078] ;

[0079] .

[0080] Then, frequency domain analysis is performed on a(t) and ω(t) within the time window to obtain the dominant frequency, peak power spectrum, and frequency band energy proportion. When the dominant frequency is between 1.5Hz and 4Hz and remains stable over multiple consecutive windows, a repetitive motion trend can be identified. To reflect periodic stability, the variance of the dominant peak frequency, Var(fpeak), can be calculated, and a periodic stability index, Stab, can be constructed. The smaller Var(fpeak) is, the more stable the motion rhythm is, and the closer it is to a repetitive and stereotyped motion pattern. Furthermore, the confidence level of motion anomaly can be formed by combining trajectory repeatability, motion amplitude stability, and duration.

[0081] 4. Implementation Method for Interaction Feature Extraction

[0082] In one embodiment, the interaction feature extraction submodule 34 statistically analyzes the click ratio of the same area, the number of times the same control is repeatedly triggered, the repetition rate of the swipe direction, the duration of screen dwell, and the number of task switching times within a preset time window. When the proportion of repeated operations at the same coordinate or the same control is higher than a preset threshold and the duration exceeds a preset time, it can be used as an auxiliary feature of repeated interaction behavior. In training scenarios, interaction features can also reflect task execution performance, such as target click hit rate, drag path deviation, facial expression matching accuracy, and action imitation completion rate.

[0083] VI. Implementation Methods for Individual Baseline Modeling

[0084] In step S104, the individual baseline modeling module 4 establishes an individual reference model based on the behavioral characteristics of the tested subject within a historical period. The historical period can be divided by calendar day, the same time period, the same training task type, or the same environmental scenario. For example, eye contact duration, vocalization frequency, movement periodicity score, and interaction repetition rate can be statistically analyzed within the same time period over the past 7 days. For each feature... The system can maintain its historical average. and standard deviation And calculate the standardized deviation of the current feature. : ;

[0085] Here, ε is a small constant to prevent the denominator from being zero. The deviations of multiple features can be further combined according to weights to form the overall anomaly degree Z.

[0086] When historical samples are insufficient, the system enters cold start mode. In cold start mode, a higher threshold can be used for judgment, or only behavioral trend records can be output without triggering complex training tasks. As historical samples gradually accumulate, the individual baseline modeling module 4 continuously updates the mean and fluctuation range through sliding or exponential updates.

[0087] VII. Fusion Recognition Implementation Method

[0088] like Figure 4 As shown, in step S105, the fusion recognition module 5 receives feature vectors and individual deviation values ​​from each modality and performs comprehensive discrimination on the multimodal information.

[0089] The fusion recognition module 5 first generates modal reliability weights based on ambient brightness E, audio signal-to-noise ratio SNR, face detection confidence pface, and inertial acquisition stability. , , and ,satisfy:

[0090] + =1;

[0091] For example, when the ambient light is too low, the weight of the visual modality is reduced; when the ambient noise is too high, the weight of the audio modality is reduced; and when the wearable device becomes detached or unstable, the weight of the motion modality is reduced. Subsequently, the confidence level for visual anomalies is calculated separately. Audio anomaly confidence level Confidence level of abnormal movement and interaction anomaly confidence And calculate the comprehensive behavioral score S:

[0092] ;

[0093] Dynamic thresholds can be set based on the statistical distribution of historical comprehensive behavioral scores. If the comprehensive behavioral score S is higher than the dynamic threshold The corresponding behavior label and confidence level will be output. The dynamic threshold can be generated based on the mean and standard deviation of the scores for the same time period over several recent days.

[0094] The fusion recognition results can include at least one of the following: stereotyped behaviors, eye contact avoidance, abnormal vocalizations, interaction avoidance, emotional agitation-related behaviors, insufficient task participation, insufficient motor imitation, and insufficient facial expression imitation. Furthermore, it can output behavioral levels, such as mild, moderate, and severe, or output continuous score ratings.

[0095] VIII. Training and Guidance Implementation Methods

[0096] like Figure 5As shown, in step S106, the training guidance module 6 generates a corresponding training task based on the fusion recognition result. This training task belongs to behavioral training or interactive training and does not involve treatment steps.

[0097] 1. Implementation methods for eye-guided training

[0098] When the fusion recognition results indicate that the subject is avoiding eye contact or making insufficient eye contact, the training guidance module 6 generates an eye-guided task. The task can present a dynamic target object on the screen, which moves along a preset path and is accompanied by visual highlighting, soft prompts, or target magnification effects to attract the subject's attention and follow.

[0099] The system records the duration of eye contact, number of regressions, target following rate, and first fixation delay within the training window. The training task is considered successful when the increase in eye contact duration reaches a preset threshold or the target following rate meets a preset requirement.

[0100] 2. Implementation methods for action imitation training

[0101] When the fusion recognition results indicate that the test subject exhibits repetitive swinging, patting, or other stereotyped movements, or insufficient upper limb movement involvement, the training guidance module 6 generates movement imitation training. This movement imitation training can display simplified movement demonstrations on the interface, such as raising a hand, swinging an arm, pointing, or touching, and guide the test subject to complete the corresponding movements through visual prompts.

[0102] The system detects and compares the similarity between the demonstration trajectory and the actual trajectory by detecting key points of the posture, and determines the imitation result by combining the action completion time and the number of repetitions. If the action similarity is greater than a preset threshold, the training is considered successful.

[0103] 3. Implementation methods for facial expression imitation training

[0104] When the system detects interaction avoidance, insufficient facial expression changes, or abnormal emotional reactions, it can generate an expression imitation training task. The training interface displays target expression images or animations, such as smiling, surprise, or calmness. The system judges the degree of expression completion of the test subject based on facial key point displacement and expression classification results, and outputs encouraging feedback or simplifies the next task.

[0105] 4. Interactive Response Training Implementation Method

[0106] When the system detects that the test subject is slow to respond to audio cues, visual cues, or interface interactions, the training guidance module 6 generates an interactive response training task. This task may include clicking a target after hearing a prompt, dragging and matching an icon after seeing it, or completing simple steps in sequence. The system evaluates the interaction capability based on response latency, click accuracy, and step correctness.

[0107] IX. Implementation Methods for Effect Evaluation and Parameter Update

[0108] In step S107, the effect evaluation and update module 7 receives feedback data during the training process and quantifies the training guidance effect. The quantification indicators include at least one of the following: improvement in eye contact, improvement in action imitation similarity, improvement in facial expression recognition accuracy, decrease in frequency of abnormal behavior, reduction in response latency, task success rate, and number of consecutive successes.

[0109] When training tasks are successfully completed consecutively, the system can increase the difficulty of the next round of tasks, such as increasing the movement speed of dynamic targets, shortening the prompt display time, increasing the complexity of actions, or increasing the number of task steps; when training tasks fail consecutively, the system can reduce the difficulty, extend the prompt time, reduce interference elements, or switch to a simpler task type.

[0110] Regarding parameter updates, the individual baseline modeling module 4 and the fusion recognition module 5 update the baseline parameters, modality weights, and recognition thresholds based on the success or failure results. For example, for a successfully completed gaze training task, the current optimal gaze duration can be included in the baseline update; for low-quality samples caused by environmental interference, their impact on the baseline update can be reduced.

[0111] 10. Specific Application Examples

[0112] Application Example 1: Eye Avoidance Recognition and Eye Guidance Training

[0113] In this application example, the system is deployed on a tablet and a smartwatch. The tablet's front-facing camera has a resolution of 640×480 and a frame rate of 30 frames per second; the microphone sampling rate is 16kHz; the accelerometer and gyroscope in the smartwatch both have a sampling rate of 100Hz.

[0114] The system collects facial images, audio, and inertial data within a 5-second statistical window. Visual feature extraction yields a gaze duration (Tcontact) of 0.4 seconds and a gaze deviation event frequency of 3 times per second. Simultaneously, the individual baseline model shows a historical average gaze duration of 1.2 seconds for the same time period. After standardization, if the visual abnormality deviation value exceeds a preset threshold, the system outputs a "gaze avoidance" behavior label.

[0115] Subsequently, the training guidance module generates an 8-second eye-tracking task, displaying a dynamic target on the screen accompanied by soft audio prompts. After training, the system measured that Tcontact improved to 1.1 seconds and the target tracking rate reached 78%, exceeding the set success threshold. Therefore, the training was deemed successful, and this data was recorded in the eye-tracking training effect record.

[0116] Application Example 2: Stereotyped Motion Recognition and Motion Imitation Training

[0117] In this application example, the system detected that the dominant frequency of motion in continuous multi-window movements in a natural activity scene was between 2.1Hz and 2.4Hz, and the variance of the dominant frequency was lower than a preset threshold, indicating high period stability. Combined with the results of visual analysis of hand key point trajectories, the system determined that the tested object exhibited a tendency for repetitive swinging movements and output a "stereotyped movement" behavior label.

[0118] The training guidance module then switches to motion imitation training, presenting three simplified actions in sequence on the interface: raising a hand, touching, and opening both hands. The system compares the trajectories of key human body points and finds that the similarity of the trajectory for the first two actions is 0.83 and 0.79 respectively, higher than the threshold of 0.75. However, the similarity for the third action is only 0.58, thus determining that the first two actions were successful and the last one failed. The effect evaluation and update module accordingly maintains the same action complexity for the next round but extends the prompt time for the third action.

[0119] Application Example 3: Abnormal Vocal Recognition and Interactive Response Training

[0120] In this application example, the audio feature extraction submodule detected a significant increase in the short-time energy peak ratio within a continuous window, enhanced fundamental frequency fluctuations, and a noticeable response delay after the training prompts appeared. Combined with the environmental signal-to-noise ratio evaluation results, the system outputs the recognition result of "abnormal sound accompanied by insufficient interactive response" after reducing the impact of environmental noise.

[0121] The training guidance module generates a "listen and click" task: after the terminal plays a short prompt tone, the test subject is required to click the target icon on the interface. The system records the response latency, click hit rate, and number of repeated false clicks. If the response latency is shorter than the historical average and the hit rate reaches the threshold, the training is considered effective, and the speed of the next round of tasks is improved.

[0122] Application Example 4: Degradation Identification under Multimodal Partial Failure

[0123] In this application example, the environment is in low light, resulting in consistently low confidence in face detection for the visual modality; simultaneously, the microphone detects significant ambient noise. The system automatically reduces the weights of the visual and audio modalities based on brightness and signal-to-noise ratio parameters, shifting the focus of recognition to motion and interaction features.

[0124] Despite the failure of the visual modality, the system still outputs the recognition result of "repetitive operation accompanied by abnormal movement" based on acceleration frequency domain features, the proportion of repeated touches, and historical individual baselines, and generates a relatively simple interactive following training task. Therefore, this invention still has good usability in complex scenarios.

[0125] XI. Optional Implementation Methods

[0126] The fusion recognition module 5 can be implemented by combining rule-based discrimination with machine learning models. For behaviors that can be clearly parameterized, such as insufficient eye contact or repeated clicking, a threshold discrimination method can be used; for situations where multiple behaviors occur simultaneously or have ambiguous boundaries, a sequence classification model, an ensemble classification model, or a lightweight neural network model can be used to output the recognition results.

[0127] The training guidance module 6 can prioritize generating a certain type of training task based on the training objectives preset by the parent or teacher. For example, when the user is more focused on social attention skills, tasks such as eye guidance and interactive response will be prioritized; when the user is more focused on motor control skills, tasks such as movement imitation and rhythm following will be prioritized.

[0128] The remote collaboration module 8 can output weekly or periodic trend reports, including the frequency of target behaviors, training completion rate, success rate of various tasks, and behavioral change trend graphs, so that parents or teachers can conduct long-term observation.

[0129] XII. Explanation

[0130] It should be noted that the "training guidance" mentioned in this invention refers to guiding the subject's attention, imitation, interaction, or rhythm through visual, audio, interactive, or action demonstration methods to support early intervention training; its purpose is to assist in behavior improvement and ability training, and does not constitute a limitation of medical treatment steps.

[0131] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, or combinations made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A system for identifying and guiding autistic behaviors, characterized in that: It includes a multimodal data acquisition module, a preprocessing and alignment module, a feature extraction module, an individual baseline modeling module, a fusion recognition module, a training guidance module, and an effect evaluation and update module; The multimodal data acquisition module is used to collect multi-source behavioral data of the object under test; The preprocessing and alignment module is used to remove invalid data segments, correct time deviations between different modalities, form a multimodal data sequence under a unified time series, and generate usability labels and quality evaluation results for each modality. The feature extraction module is used to extract structured features related to behavior recognition from the preprocessed multimodal data; The individual baseline modeling module is used to build an individualized reference model based on historical behavior data. According to the preset time period, scene category or training task category, it establishes statistical distribution, trend model or individual baseline vector for various features, and calculates the degree of deviation of the current feature from the historical behavior pattern to characterize the intensity of abnormal behavior or the magnitude of behavior change. The fusion recognition module performs weighted fusion, feature dimensionality reduction, temporal modeling and classification discrimination on the structured features based on the quality evaluation results of each modality, structured features and the degree of deviation of the individual baseline, and outputs the recognition result of one or more target behaviors. The training guidance module is used to generate corresponding training tasks or interactive guidance content based on the recognition results; the effect evaluation and update module is used to evaluate the behavioral changes after training guidance and update the model parameters.

2. The autism behavior recognition and training guidance system as described in claim 1, characterized in that: The multimodal data acquisition module includes one or more of the following: a camera, a microphone, an inertial sensor, and an interactive acquisition unit.

3. The autism behavior recognition and training guidance system as described in claim 1, characterized in that: The feature extraction module includes a visual feature extraction submodule, an audio feature extraction submodule, a motion feature extraction submodule, and an interaction feature extraction submodule; The visual feature extraction submodule is used to extract facial key points, head posture, gaze direction, gaze duration, facial expression changes, limb key point trajectories, and movement amplitude features. The audio feature extraction submodule is used to extract features such as sound intensity, fundamental frequency, spectral distribution, speech rate, pauses, prosodic variations, and speech activity. The motion feature extraction submodule is used to extract features such as acceleration, angular velocity, motion intensity, periodicity, frequency band energy, trajectory change, and repetitive motion. The interaction feature extraction submodule is used to extract features such as click repetition rate, dwell time, swipe pattern, operation sequence, and task completion process.

4. The autism behavior recognition and training guidance system as described in claim 1, characterized in that: The effect evaluation and update module evaluates the training guidance effect based on indicators such as the improvement of eye contact, the decrease in abnormal behavior, the task completion rate, the response latency, and the number of consecutive successes during or after training. The evaluation results are fed back to the individual baseline modeling module and the fusion recognition module to update the threshold, weight, task difficulty, or model parameters.

5. The autism behavior recognition and training guidance system as described in claim 1, characterized in that: The system also includes a remote collaboration module, which sends behavior recognition results, training records, and trend summaries to parents or teachers to assist in training management and periodic assessments.

6. A method for identifying and guiding autistic behaviors based on the system described in any one of claims 1 to 4, characterized in that, Specifically, the steps include the following: Visual, audio, motion, and interaction data of the tested object are collected to form a raw multimodal dataset. The raw multimodal dataset is preprocessed and time-aligned to generate multimodal analysis data, and the quality of each modality is evaluated. Visual, audio, motion, and interaction features are extracted from the multimodal analysis data to construct a structured feature vector for the current time window. An individual baseline model is established based on historical behavior data, the deviation of the current structured features from the individual baseline is calculated, and modal reliability parameters are generated. The structured features, individual baseline deviation, and modal reliability parameters are input into a fusion recognition model to output recognition results for one or more target behaviors. Training guidance tasks are automatically generated based on the recognition results, and the corresponding visual, audio, or interactive training content is output on the terminal interface. Collect feedback data during the training process, quantitatively evaluate the training effect, and update individual baselines, identification parameters, and training task configurations based on the evaluation results.

7. The autism behavior recognition and training guidance method as described in claim 6, characterized in that: The preprocessing and alignment module performs quality inspection, noise reduction, segmentation, standardization, and time synchronization on the multimodal data.

8. The method for identifying and training autistic behaviors as described in claim 6, characterized in that: The target behaviors include one or more of the following: stereotyped actions, eye contact avoidance, abnormal vocalizations, interaction avoidance, and emotionally agitated behaviors.

9. The method for identifying and training autistic behaviors as described in claim 6, characterized in that: The identification results include behavior labels, behavior confidence, behavior duration, behavior frequency, and / or behavior level.

10. The method for identifying and guiding autistic behaviors as described in claim 6, characterized in that: The training guidance module can output one or more of the following tasks: visual attention guidance task, eye-following task, action imitation task, facial expression imitation task, interactive response task, rhythm soothing prompt, or task switching prompt; The type, difficulty, duration, presentation order, and feedback method of the training tasks are adaptively adjusted based on the behavior recognition results, individual baseline, and historical training performance to achieve individualized training guidance for early intervention.

Images