Autism children stereotyped behavior recognition classification system and method based on RMG-transformer

Abstract

The application is suitable for the technical field of behavior recognition, and provides an autism child stereotyped behavior recognition classification system and method based on RMG-transformer, which comprises the following steps: video data acquisition and preprocessing; repeated action segment detection; super image construction; feature extraction and behavior classification; and recognition result output. The repeated action detection submodule and the super image submodule are innovatively introduced in the original Transformer network architecture, and the loss function is further optimized, so that the efficiency can be improved according to the characteristics of the stereotyped behavior repetition, and the time sequence characteristics are enhanced, and the recognition ability of continuous actions is improved.

Description

Technical Field

[0001] This invention belongs to the field of behavior recognition technology, and in particular relates to a system and method for identifying and classifying stereotyped behaviors in children with autism based on RMG-transformer. Background Technology

[0002] Autism Spectrum Disorder (ASD), a prevalent neurodevelopmental disorder, faces a severe and continuously increasing prevalence. Early intervention is crucial for improving the prognosis of ASD patients. Scientific research shows that intervention before the age of 6 can significantly improve patients' social skills and self-care abilities. However, traditional ASD stereotyped behavior identification and intervention models have significant shortcomings: relying on manual observation, interviews, questionnaires, and other methods for assessment not only consumes a large amount of manpower and has extremely low identification efficiency (the behavioral assessment of a single child requires continuous observation by a professional physician for 4-6 hours), but also suffers from significant accuracy fluctuations due to the experience differences of the assessors (the consistency between different assessors is only 65%-70%), making it difficult to meet the clinical needs of large-scale screening and refined identification.

[0003] In recent years, breakthroughs in deep learning technology in computer vision and video processing have provided a new technical approach for the recognition of stereotyped behaviors in ASD (Acute Repetitive Disorder). Existing research has achieved automated recognition of some stereotyped behaviors through methods such as skeleton information extraction, time series processing, multimodal data fusion, and attention mechanisms. For example, in 2024, the research by Tal Barami et al. combined YOLOv5 with the PoseConv3D algorithm, increasing the detection rate of stereotyped movements to 92.53%.

[0004] However, the existing solutions still have several limitations:

[0005] At the dataset level: Existing datasets are relatively small in scale (e.g., SSBD contains only 59 unstructured videos, and the Autism dataset covers data from 37 children), and the scenarios are singular, mostly concentrated in laboratory or medical institution environments, lacking data from natural scenarios such as families and communities, resulting in insufficient generalization ability of models in real and complex scenarios.

[0006] In terms of functional completeness: some algorithms can only identify the existence of stereotyped behaviors, but cannot subdivide the behavior types (such as distinguishing specific actions such as arm waving, head impact, and rotation), and the accuracy of marking the start and end times of behaviors is insufficient (the error often exceeds 5 seconds), making it difficult to meet the needs of clinical intervention for detailed analysis of behaviors.

[0007] In terms of model performance: Traditional Transformer networks are weak in capturing the temporal features of continuous actions and are easily interfered with by non-repetitive segments in videos, resulting in low processing efficiency (long video processing often takes more than 12 hours); at the same time, the original loss functions are mostly designed for single classification tasks, which makes it difficult to meet the multi-task collaborative optimization requirements of time localization and behavior classification, thus restricting further improvement of recognition accuracy.

[0008] These problems make it difficult for existing technologies to play a full role in large-scale clinical screening and personalized intervention program development, and there is an urgent need for an intelligent recognition solution that can balance recognition accuracy, efficiency and scenario adaptability. Summary of the Invention

[0009] The purpose of this invention is to provide a method for identifying and classifying stereotyped behaviors in children with autism based on RMG-transformer, aiming to solve the problems mentioned in the background art.

[0010] The present invention is implemented as follows: a method for identifying and classifying stereotyped behaviors in children with autism based on RMG-transformer, comprising the following steps:

[0011] Step 1: Video Data Acquisition and Preprocessing: Acquire the video data to be analyzed and perform preprocessing operations on the video data. The preprocessing operations include video decoding, frame extraction, and resolution normalization to obtain a standardized video frame sequence.

[0012] Step 2, Repetitive Action Segment Detection: For the video frame sequence, a self-similar matrix SSM is constructed based on the repetitive action detection module in the RMG-transformer model. By thresholding the diagonal elements of the prediction matrix, it is determined whether there are repetitive actions in the video, and one or more candidate repetitive action segments are output. The candidate repetitive action segments are represented in the form of time intervals or corresponding video frame index intervals.

[0013] Step 3: Hyperimage Construction: For each candidate repetitive action segment, based on the hyperimage construction and feature extraction module in the RMG-transformer model, several video frames are extracted in the corresponding time interval according to the preset frame sampling strategy, and the video frames are spatiotemporally rearranged according to the predefined grid structure to generate a hyperimage representing the candidate repetitive action segment.

[0014] Step 4: Feature Extraction and Behavior Classification: For the hyperimage, the transformer-based feature extraction network in the hyperimage construction and feature extraction module is used for feature extraction and classification. The predicted probability value corresponding to each preset behavior category is output, and the behavior category with the highest predicted probability is selected as the recognition result of the candidate repetitive action segment.

[0015] Step 5: Output the recognition results: Output the behavior recognition results, including: behavior information category, corresponding category confidence score, and the time period in which the behavior occurred in the initial input video.

[0016] Another objective of this invention is to provide a classification system for identifying stereotyped behaviors in children with autism based on RMG-transformer, used to implement the above-mentioned method, including:

[0017] The data acquisition and preprocessing module is used to acquire video data to be analyzed and to perform preprocessing operations on the video data. The preprocessing operations include video decoding, frame extraction, and resolution normalization to obtain a standardized video frame sequence.

[0018] The RMG-transformer model module includes:

[0019] The repeat action detection submodule is used to construct a self-similar matrix (SSM). By applying a threshold to the diagonal elements of the prediction matrix, it determines whether there are repeating actions in the video and outputs one or more candidate repeating action segments. The candidate repeating action segments are represented in the form of time intervals or corresponding video frame index intervals.

[0020] The hyperimage construction and feature extraction submodule is used to extract several video frames in the corresponding time interval according to a preset frame sampling strategy, and to perform spatiotemporal rearrangement of the video frames according to a predefined grid structure to generate a hyperimage representing the candidate repetitive action segment; the feature extraction network based on transformer performs feature extraction and classification processing, outputs the predicted probability value corresponding to each preset behavior category, and selects the behavior category with the highest predicted probability as the recognition result of the candidate repetitive action segment;

[0021] The output module is used to output the results of behavior recognition, including: behavior information category, corresponding category confidence score, and the time period in which the behavior occurred in the initial input video.

[0022] This invention provides an intelligent recognition scheme that balances recognition accuracy, efficiency, and scene adaptability, and has the following advantages:

[0023] Breaking through the limitations of predefined behavior categories, it can locate any repetitive behavior segment from long videos. Combining 3DCNN and U-Net, it introduces a repetitive action detection submodule to improve the accuracy and efficiency of stereotyped behavior localization. The output periodic segments provide intuitive analysis evidence for clinical practice.

[0024] By rearranging video frames into a "super image," the spatiotemporal information of the video is encoded into a 2D image, enabling Transformer to simultaneously model short-term action details and long-term behavioral evolution, enhancing the ability to resolve dynamic temporal features, and maintaining compatibility with existing architectures to reduce development difficulty.

[0025] This approach integrates multiple loss functions to optimize classification and sample learning. To address the imbalance in the number of training samples across classes, a class-weighted loss is used to increase the weight of rare class samples. Simultaneously, the weights of low-confidence samples are increased to improve training performance.

[0026] It can be adapted to clinical diagnostic needs, providing doctors with decision support that combines temporal accuracy and classification reliability, helping to develop personalized intervention plans and promoting the digital transformation of autism intervention. Attached Figure Description

[0027] Figure 1 A schematic diagram illustrating the method for identifying and classifying stereotyped behaviors in children with autism based on RMG-transformer, provided in an embodiment of the present invention.

[0028] Figure 2 A flowchart illustrating the G-TAD loss function provided in an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0030] The method provided in this embodiment of the invention can be implemented by the following hardware devices and software modules:

[0031] Hardware devices include: a 10-megapixel camera (25-30fps), a GPU accelerator card (≥16GB VRAM), storage devices (≥1TB capacity), and a monitor (≥1920×1080 resolution), etc.

[0032] The software modules include: data acquisition and preprocessing module, RMG-transformer model module, human-computer interaction module, etc. Among them, the RMG-transformer model module is the core, which includes repetitive action detection sub-module, hyperimage construction and feature extraction sub-module, etc. Each module is implemented in Python programming language, and model training and deployment are carried out in conjunction with PyTorch deep learning framework.

[0033] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0034] like Figure 1The diagram shown is a schematic representation of a method for identifying and classifying stereotyped behaviors in children with autism based on RMG-transformer, provided in an embodiment of the present invention. The method includes the following steps:

[0035] Step 1: Data Collection and Preprocessing: Data sources include online datasets collected in home settings and clinical data provided by a partner medical institution (The First Hospital of Jilin University), covering common stereotyped behaviors of autistic patients of different ages (0-16 years old), such as arm waving, head banging, and spinning.

[0036] Preprocessing steps: The video format is unified to MP4, and the video frame size is fixed by scaling or cropping; to improve the model's generalization ability, various data augmentation methods are applied to the input image, such as color dithering and random horizontal flipping; important structures and patterns are effectively identified from neighborhood information through local feature extraction, which helps to filter out local noise; then video encoding and feature extraction are performed.

[0037] Given an untrimmed long video, extract all frames from the entire video frame by frame or at fixed intervals to form a frame sequence:

[0038] ;

[0039] in, This represents the first frame of the image. This represents the second frame of the image. This represents the T-th frame image, where T is the total number of video frames. Each frame is an RGB image, scaled to a uniform size (e.g., 224x224).

[0040] Step 2, Repetitive Action Segment Detection: By analyzing the self-similarity patterns of video frame sequences, the temporal localization of repetitive action segments is achieved. The specific process includes:

[0041] The obtained video frame sequence was encoded using a 3D-ResNet-18 convolutional neural network to extract spatiotemporal features from the video. The features of each frame were then embedded into a feature vector, resulting in the following feature sequence:

[0042] ;

[0043] in, This represents an upsampling operation in the time dimension, used to restore the same number of frames as the original video. This represents the feature vector of the first frame. This represents the feature vector of the second frame. Let T represent the feature vector of the T-th frame. Represents a three-dimensional convolutional coding network;

[0044] Maintaining the same temporal resolution as the original video through bilinear upsampling;

[0045] For each pair of frame features, calculate the inner product similarity and construct a self-similarity matrix (SSM). The self-similarity matrix is ​​as follows:

[0046] ;

[0047] in, This represents the similarity between the i-th frame and the j-th frame. The inner product operation is used to calculate similarity;

[0048] When there are repetitive actions in a video, the related frames will exhibit a periodic diagonal stripe structure in the self-similarity matrix;

[0049] For the constructed self-similar matrix, a periodic prediction network based on an encoder-decoder structure is used. The U-Net structure is used to analyze the SSM and output the prediction result of whether each frame belongs to the repeating action interval.

[0050] During the training phase, frame-level repetitive annotations are converted into a binary matrix form B, and the network is optimized using a binary cross-entropy loss function, defined as follows:

[0051] ;

[0052] in, This indicates the repetition relationship of the actual annotation. This represents the network prediction result. This refers to the number of video frames.

[0053] During the inference phase, a binary determination result is obtained for whether each frame belongs to a repetitive action by thresholding the diagonal elements of the prediction matrix. Finally, the frames that are continuously determined to be repetitive actions are time-aggregated. When the number of consecutive frames exceeds a preset threshold, it is determined to be a valid repetitive action time period, and its start time and end time, as well as the valid video frame sequence, are output as input data for the next step.

[0054] Step 3: To address the issues of complex temporal modeling and high computational costs associated with existing video behavior recognition methods, this embodiment of the invention employs a video spatiotemporal feature module based on superimage construction. This module reassembles video frames into a superimage through a predefined spatial layout, achieving the mapping of video temporal features to spatial features of 2D images. The specific process includes:

[0055] Phase 1: Input Preparation and Frame Sampling

[0056] Input: An unprocessed video;

[0057] Frame sampling: A fixed number of frames are extracted from the entire video according to a certain strategy (uniform sampling is commonly used). A fixed number of keyframes are extracted from the input video according to a preset sampling strategy. The sampling strategy includes, but is not limited to, uniform sampling, random sampling, or time-interval sampling. A frame sequence is obtained: [Frame1, Frame2,..., Frame8]. Each frame is a standard RGB image, and the size is usually scaled to a uniform size, such as 224×224 pixels.

[0058] Phase Two: Constructing the Hyperimage

[0059] Select grid layout: Determine a grid structure for arranging frames. The sampled arrangement structure includes, but is not limited to, a grid structure of row arrangement (1*n), column arrangement (n*1), or rectangular arrangement (i*j).

[0060] Rearrangement and stitching: The sampled frame sequence is sequentially filled into a predefined grid according to time order. Assuming 8 frames were sampled and a 2*4 grid was selected, the filling order is usually row-first (fill the first row first, then the second row, and so on). Through this step, a sequence of 8 224×224 images is stitched into a large (2*224)×(4*224) = 448×896 hyperimage. This process can be represented by the following mapping relationship:

[0061] ;

[0062] in, This represents N frames of images sampled from a video. This represents a predefined spatial layout rule used to indicate the arrangement position of each frame within the superimage. This represents the spatiotemporal rearrangement mapping function. This represents the generated hyperimage;

[0063] In one embodiment, the spatial layout rule adopts a regular two-dimensional grid structure, and each frame is filled into the corresponding grid position in the time order of the video frames. When the number of frames is less than the grid capacity, the hyperimage can be constructed by filling blank images or preset placeholder images.

[0064] Phase 3: Image Classifier Processing of Hyperimages

[0065] Input: The hyperimage generated in stage two;

[0066] Backbone network: This hyperimage is processed using a Transformer-based improved feature extraction network;

[0067] Feature extraction and pattern recognition: Image classifiers extract hierarchical features from hyperimages just like they process ordinary images;

[0068] Local spatial mode: When the attention window of the feature extraction network (e.g., a 7x7 window) is completely inside a single frame, it learns the spatial features of that frame (e.g., identifying a person or a ball).

[0069] Local temporal mode: When the attention window spans two adjacent frames in the grid (as shown in the 3*3 window in the figure), it directly learns the changes between frames (i.e., short-range temporal dependencies) in the local range, realizing the joint modeling of single-frame spatial features, short-temporal features of adjacent frames, and long-temporal features across frames.

[0070] Global temporal mode: Global self-attention is used in the last layer of the network, which means that the receptive field of the last layer covers the entire superimage, thus enabling the integration of information from all frames, capturing long-range temporal dependencies, and understanding the global context of the entire action.

[0071] In addition, since the probability of different stereotyped behaviors in autistic patients varies and the complexity of different stereotyped behaviors varies, there are significant differences in the number and complexity of different categories of stereotyped behaviors among the existing video samples of autistic patients that can be collected. During the training process, existing behavior recognition models are easily affected by a large number of easily classifiable samples and high-frequency category samples, resulting in insufficient learning ability of the model for difficult samples and low-frequency categories, thereby reducing the overall recognition performance.

[0072] To address the above problems, embodiments of the present invention propose a G-TAD loss function, such as... Figure 2 As shown, by introducing sample difficulty weights and class weights, the basic classification loss is weighted and adjusted. This adjustment is based on the cross-entropy loss and is defined as follows:

[0073] ;

[0074] in, Represents the basic cross-entropy loss. Let i be the true label of the i-th training sample. The prediction output is for the i-th sample. The comprehensive weight is the weight corresponding to the i-th sample. The comprehensive weight of each sample is the main part that balances the impact of different sample numbers and different action difficulties on the training results, and is determined by two parts: sample difficulty weight and category number weight.

[0075] The sample difficulty weight is related to the average confidence of the prediction results for a certain category. Based on the model's prediction confidence of the sample, the sample difficulty weight of a certain category is dynamically adjusted so that samples with lower prediction confidence receive higher weights, thereby guiding the model to focus on learning difficult-to-classify samples.

[0076] The sample number weight is related to the number of samples in a certain category. Based on the distribution of samples of different categories in the training set, higher weights are assigned to categories with lower frequency of occurrence, and lower weights are assigned to categories with higher frequency of occurrence, in order to improve the model's ability to identify rare categories.

[0077] Through the weighted loss design described above, this embodiment of the invention ensures stable training of the model as a whole while effectively improving the learning effect on difficult samples and low-frequency categories under conditions of imbalanced datasets and large differences in complexity between different categories, thereby improving the accuracy and generalization ability of behavior recognition.

[0078] The above embodiments are applied to specific examples as follows:

[0079] 1. Experimental Dataset Construction: A total of 100 unstructured video data collected in a home environment were selected as the input dataset. The video data were collected in real home scenes and had the characteristics of complex background, obvious lighting changes and non-fixed shooting angle. The above videos contained three types of stereotypical behaviors: abnormal hand swinging behavior, body rotation behavior, and head impact behavior.

[0080] 2. Experimental setup and training method: Five-fold cross-validation was used to train and test the dataset to reduce the impact of randomness in data partitioning on the experimental results and improve the stability and reliability of the evaluation results.

[0081] 3. Evaluation metrics: To comprehensively evaluate the effectiveness of the above methods, the following evaluation metrics are used: Macro Sensitivity, Macro Specificity, Micro Sensitivity, Micro Specificity, and Top-1 Classification Accuracy (TOP1_acc).

[0082] Among them, the macro-average index is used to measure the overall balance performance of the model across different categories, while the micro-average index is used to reflect the model's comprehensive recognition ability at the sample level.

[0083] Recognition results:

[0084] Under the above experimental setup, the method of the present invention achieves the following recognition results on the dataset:

[0085] The macro-average sensitivity reached 0.9683, the macro-average specificity reached 0.9849, the micro-average sensitivity reached 0.9700, the micro-average specificity reached 0.9850, and the Top-1 classification accuracy reached 0.9700.

[0086] To further illustrate the technical effects of the method in the embodiments of the present invention, a comparative example is used for comparison. In this comparative example, an action recognition method without introducing a repetitive action detection submodule, a hyperimage construction submodule, and a weighted loss function is used. This method directly performs feature extraction and classification training based on the original video data, without screening repetitive action segments in the video, and without weighting the sample difficulty and class imbalance issues.

[0087] Except for the method not used in this embodiment of the invention, the other experimental conditions are consistent with those in Specific Embodiment 1, including: using the same dataset, using the five-fold cross-validation method (the validation set and training set are randomly split with the same seed), and using the same evaluation index system;

[0088] Under the above conditions, the comparative examples obtained the following recognition results on the same dataset: macro average sensitivity of 0.9583, macro average specificity of 0.9784, micro average sensitivity of 0.9600, micro average specificity of 0.9800, and Top-1 classification accuracy of 0.9600.

[0089] The specific comparison results are shown in Table 1:

[0090] Table 1

[0091] Evaluation indicators Example Comparative example contrast Macro Sensitivity 0.9683 0.9583 +0.0100 Macro Specificity (Average Macro Specificity) 0.9849 0.9784 +0.0065 Micro Sensitivity 0.9700 0.9600 +0.0100 Micro Specificity 0.9850 0.9800 +0.0050 TOP-1 0.9700 0.9600 +0.0100

[0092] It can be seen that, under the same experimental conditions, the method of this invention achieves better recognition results in terms of macro-average index, micro-average index, and overall classification accuracy. This indicates that the present invention effectively improves the ability to recognize repetitive stereotyped behaviors by introducing repetitive action detection, hyperimage representation, and weighted loss function, especially in complex home environment videos and multi-class conditions, showing better stability and accuracy.

[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying and classifying stereotyped behaviors in children with autism based on RMG-transformer, characterized in that, Includes the following steps: Step 1: Video Data Acquisition and Preprocessing: Acquire the video data to be analyzed and perform preprocessing operations on the video data. The preprocessing operations include video decoding, frame extraction, and resolution normalization to obtain a standardized video frame sequence. Step 2, Repetitive Action Segment Detection: For the video frame sequence, a self-similar matrix SSM is constructed based on the repetitive action detection module in the RMG-transformer model. By thresholding the diagonal elements of the prediction matrix, it is determined whether there are repetitive actions in the video, and one or more candidate repetitive action segments are output. The candidate repetitive action segments are represented in the form of time intervals or corresponding video frame index intervals. Step 3: Hyperimage Construction: For each candidate repetitive action segment, based on the hyperimage construction and feature extraction module in the RMG-transformer model, several video frames are extracted in the corresponding time interval according to the preset frame sampling strategy, and the video frames are spatiotemporally rearranged according to the predefined grid structure to generate a hyperimage representing the candidate repetitive action segment. Step 4: Feature Extraction and Behavior Classification: For the hyperimage, the transformer-based feature extraction network in the hyperimage construction and feature extraction module is used for feature extraction and classification. The predicted probability value corresponding to each preset behavior category is output, and the behavior category with the highest predicted probability is selected as the recognition result of the candidate repetitive action segment. Step 5: Output the recognition results: Output the behavior recognition results, including: behavior information category, corresponding category confidence score, and the time period in which the behavior occurred in the initial input video.

2. The method for identifying and classifying stereotyped behaviors in autistic children based on RMG-transformer according to claim 1, characterized in that, The video data undergoes preprocessing operations, including video decoding, frame extraction, and resolution normalization, to obtain a standardized video frame sequence. Specifically, the preprocessing includes: Based on the acquired video data, all frames are extracted frame by frame or at fixed intervals to form a video frame sequence: ； in, This represents the first frame of the image. This represents the second frame of the image. This represents the T-th frame image, where T is the total number of video frames. Each frame is an RGB image scaled to a uniform size.

3. The method for identifying and classifying stereotyped behaviors in autistic children based on RMG-transformer according to claim 2, characterized in that, For a video frame sequence, the steps for constructing a self-similarity matrix (SSM) based on the repeat action detection module in the RMG-transformer model specifically include: A 3D convolutional neural network (3D-ResNet-18) was used as the encoder to extract spatiotemporal features from the video frame sequence, and the features of each frame were embedded into a feature vector, resulting in the following feature sequence: ； in, This represents an upsampling operation in the time dimension, used to restore the same number of frames as the original video. This represents the feature vector of the first frame. This represents the feature vector of the second frame. Let T represent the feature vector of the T-th frame. Represents a three-dimensional convolutional coding network; Maintaining the same temporal resolution as the original video through bilinear upsampling; For each pair of frame features, calculate the inner product similarity and construct the self-similarity matrix SSM. The self-similarity matrix is ​​as follows: ； in, This represents the similarity between the i-th frame and the j-th frame. The inner product operation is used to calculate similarity; When there are repetitive actions in a video, the related frames will exhibit a periodic diagonal stripe structure in the self-similarity matrix.

4. The method for identifying and classifying stereotyped behaviors in autistic children based on RMG-transformer according to claim 3, characterized in that, The steps of determining whether repetitive actions exist in a video by thresholding the diagonal elements of the prediction matrix and outputting one or more candidate repetitive action segments specifically include: For the constructed self-similarity matrix, a periodic prediction network based on an encoder-decoder structure is used. The U-Net structure is used to analyze the self-similarity matrix and output the prediction result of whether each frame belongs to the repeated action interval: During the training phase, frame-level repetitive annotations are converted into a binary matrix form B, and the network is optimized using a binary cross-entropy loss function, defined as follows: ； in, This indicates the repetition relationship of the actual annotation. This represents the network prediction result. This refers to the number of video frames. During the inference phase, a binary determination result is obtained as to whether each frame belongs to a repetitive action by thresholding the diagonal elements of the prediction matrix. For consecutive frames that are determined to be repetitive actions, time aggregation is performed. When the number of consecutive frames exceeds a preset threshold, it is determined to be a valid repetitive action time period, and its start time and end time, as well as the valid video frame sequence, are output.

5. The method for identifying and classifying stereotyped behaviors in autistic children based on RMG-transformer according to claim 4, characterized in that, The steps involved in constructing a hyperimage include: Choose a grid layout: Determine a grid structure for arranging frames; Rearrangement and stitching: The video frame sequence is filled into a predefined grid structure in chronological order. This process can be represented by the following mapping relationship: ； in, This represents N frames of images sampled from a video. This represents a predefined spatial layout rule used to indicate the arrangement position of each frame within the superimage. This represents the spatiotemporal rearrangement mapping function. This represents the generated superimage.

6. The method for identifying and classifying stereotyped behaviors in autistic children based on RMG-transformer according to claim 5, characterized in that, The steps of feature extraction and behavior classification specifically include: The superimage is processed using a feature extraction network based on Transformer, which extracts hierarchical features from the superimage. When the attention window of the feature extraction network is completely within a single frame, it learns the spatial features of that frame. When the attention window spans two adjacent frames in the grid, it learns the changes between frames in a local range, thus achieving joint modeling of single-frame spatial features, short-term features of adjacent frames, and long-term features across frames. The last layer of the feature extraction network uses global self-attention to integrate information from all frames, capture long-range temporal dependencies, and understand the global context of the entire action.

7. The method for identifying and classifying stereotyped behaviors in autistic children based on RMG-transformer according to claim 1, characterized in that, The RMG-transformer model also includes the G-TAD loss function, which is based on cross-entropy loss and is defined as follows: ； in, Indicates the total number of sample batches. Represents the basic cross-entropy loss. Let i be the true label of the i-th training sample. The prediction output is for the i-th sample. Let be the comprehensive weight corresponding to the i-th sample.

8. A stereotyped behavior recognition and classification system for autistic children based on RMG-transformer, used to implement the method as described in any one of claims 1-7, characterized in that, include: The data acquisition and preprocessing module is used to acquire video data to be analyzed and to perform preprocessing operations on the video data. The preprocessing operations include video decoding, frame extraction, and resolution normalization to obtain a standardized video frame sequence. The RMG-transformer model module includes: The repetitive action detection module is used to construct a self-similar matrix (SSM). By applying a threshold to the diagonal elements of the prediction matrix, it determines whether there are repetitive actions in the video and outputs one or more candidate repetitive action segments. The candidate repetitive action segments are represented in the form of time intervals or corresponding video frame index intervals. The hyperimage construction and feature extraction module is used to extract several video frames in the corresponding time interval according to a preset frame sampling strategy, and to perform spatiotemporal rearrangement of the video frames according to a predefined grid structure to generate a hyperimage representing the candidate repetitive action segment; the feature extraction network based on transformer performs feature extraction and classification processing, outputs the predicted probability value corresponding to each preset behavior category, and selects the behavior category with the highest predicted probability as the recognition result of the candidate repetitive action segment; The output module is used to output the results of behavior recognition, including: behavior information category, corresponding category confidence score, and the time period in which the behavior occurred in the initial input video.

Images