Classroom behavior analysis method and analysis system for autistic children

Abstract

The invention aims to solve the problem that the analysis result is affected due to the single visual angle of video acquisition in the existing autistic child classroom analysis. According to the autism child classroom behavior analysis method and system, multi-view collection, stable key point detection, high-precision multi-view three-dimensional reconstruction and deep behavior analysis modes are organically integrated through the links of projection matrix construction, weighted calculation, abnormal point filtering, algorithm optimization, error elimination and the like; a complete and efficient system is formed, the accuracy of classroom behavior analysis of children with autism is greatly improved, a solid and scientific basis is provided for personalized education intervention, and a more valuable analysis result is brought.

Description

Technical Field

[0001] The present invention relates to the technical field of video image processing, and in particular to a method and system for analyzing the classroom behavior of children with autism. Background Art

[0002] Autism spectrum disorder is a common neurodevelopmental disorder characterized by deficits in social and communication skills, as well as repetitive and stereotyped behaviors. Current treatments primarily include educational intervention, psychotherapy, and medication. Educational intervention involves behavioral analysis, speech therapy, and social training, among other components. Classroom behavioral analysis is crucial for the educational intervention of children with autism, as it can reinforce positive behaviors, reduce problematic behaviors, and improve children's social skills and behavioral patterns.

[0003] In traditional methods, behavioral analysis of autistic children mainly relies on manual observation and recording. This method is not only inefficient, but also easily affected by subjective factors, making it difficult to fully and accurately capture the behavioral characteristics of children. In order to overcome the problems in traditional methods, analysis and screening systems based on audio and video have emerged, such as the content shown in the patent with authorization announcement number CN110313923B. However, in existing video analysis methods, video data collected from a single perspective is usually used, which cannot provide complete panoramic information, easily leads to information loss, and affects subsequent analysis and diagnosis results. Therefore, a multi-perspective observation method is needed to monitor and analyze the behavior of autistic children in an all-round way. Summary of the Invention

[0004] The purpose of the present invention is to solve the deficiencies of the prior art and to provide a method and system for analyzing the classroom behavior of autistic children.

[0005] In order to solve the above problems, the present invention adopts the following technical solutions: A method for analyzing classroom behavior of children with autism, comprising the following steps: Step 1: Collect video from one or more cameras at designated locations in the classroom; Step 2: Use the checkerboard calibration method to unify the different collected videos into a three-dimensional coordinate system for representation; Step 3: Preprocess the collected video and use the VGG network to extract deep semantic features. After multi-stage optimization, the two-dimensional coordinates and confidence levels of the key points of the human body in the video are obtained. Step 4: Calculate the mean, standard deviation, and dynamic threshold based on the confidence of key points in the past N frames of video images, and divide the confidence levels; Step 5: Depending on the confidence level, the preservation, Kalman filtering smoothing, or linear interpolation recovery processing methods are adopted to obtain stable and reliable two-dimensional coordinates of the key points; Step 6: Collect the visible angles of key points, construct a projection matrix, and calculate weights based on confidence, baseline length, and depth information. Then, use the RANSAC algorithm to filter outliers. Based on the inliers, construct the objective function. Then, use the LM algorithm for iterative optimization and perform light velocity adjustment to eliminate accumulated errors. Step 7: Use the trained deep learning model to determine whether typical autism behaviors are present in the video based on the selected key points. If so, identify the category of the autism behavior and record the action recognition data. Otherwise, terminate the step. Step 8: Based on the recorded action recognition data, use the trained LSTM network to analyze the action recognition data, predict behavioral trends, generate a behavior analysis report, and provide personalized intervention recommendations.

[0006] Furthermore, in step 2, the cameras set in the four corners of the classroom respectively capture videos in four directions, and the videos captured in the four directions are aligned in time, geometrically corrected in space, and then these videos are uniformly converted into a global coordinate system.

[0007] Furthermore, the pre-processing of the video in step 3 includes normalization, size adjustment, and color channel adjustment.

[0008] Furthermore, in step 3, the OpenPose network architecture is used to analyze and detect the key points of the human body in the video. The key points include nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, hip center, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear, left foot inside, left foot outside, left heel, right foot inside, right foot outside, and right heel, and the two-dimensional coordinates (x, y) of each key point in the corresponding video frame image and its confidence C are extracted.

[0009] Furthermore, the process of dividing the confidence levels in step 4 is as follows: First, calculate the mean confidence value of the key point in the past N frames of image , then calculate the confidence standard deviation σ of the past N frames and set the dynamic threshold , as shown below: , , , in, Represents the confidence of the i-th frame image in the past at the current time t; is the set value used to control the size of the dynamic threshold; In step 5, for the confidence level greater than The key points of are directly retained as high confidence key points; for confidence ≤C≤ , as the key points with medium confidence, the Kalman filter is used for smoothing; for the key points with confidence C less than , as the degree confidence key point, and restore it using linear interpolation.

[0010] Furthermore, the specific process of step 6 is as follows: Step 61: Collect the key point visible viewpoints; determine which key points under which viewpoints are available through the two-dimensional key points and their confidence in the four viewpoints, so as to retain the available key points; Step 62: Construct the projection matrix; A 3D point in the world coordinate system The two-dimensional projection under multiple viewing angles is , the projection relationship of each camera is expressed as: , , in, is the camera's projection matrix; Camera intrinsic parameter matrix; 、 Camera extrinsics; Step 63: Calculate the weight by combining the confidence, baseline length, and depth information to weight each projection equation. The formula is as follows: , in, , It is depth information; is a monotonically increasing function of the baseline length; is the baseline length between the i-th perspective and other perspectives; Represents the confidence of the key point of the i-th perspective; Step 64: RANSAC algorithm filters abnormal matching points and selects inliers; Step 65: Based on the inliers selected by RANSAC, construct an objective function that minimizes the reprojection error: , in, represents the set of interior points; Step 66: Use the LM algorithm to iteratively optimize the objective function and continuously update the 3D points until the convergence condition is met; Step 67: Use the light velocity adjustment method to eliminate the cumulative error and minimize the sum of the reprojection errors of all 3D points under each camera perspective. The objective function is as follows: , in, is the set of parameters of all cameras; is the coordinate set of all three-dimensional points; is the number of cameras; is the number of 3D points; is the two-dimensional projection coordinate of the j-th three-dimensional point in the perspective of the i-th camera; is the projection matrix of the i-th camera; is the weight; Step 68: Use the LM algorithm to solve the objective function in step 67. During the solution process, update the camera parameters and the 3D point coordinates simultaneously until the convergence condition is met.

[0011] Furthermore, the process of filtering abnormal matching points in step 64 includes: Step 641: Randomly select two viewpoints from all visible viewpoints and calculate the triangulated position points; Step 642: Project the calculated position point back to all viewing angles and calculate the error ; Step 643: Error Points with errors smaller than the set threshold are considered as inliers, and points with large errors are considered as outliers and are removed. Step 644: Repeat steps 641 to 643 to find the triangulation with the most inliers as the final structure, and retain these inliers.

[0012] Furthermore, in step 7, the sorted three-dimensional key point data is first input into the three-dimensional convolutional neural network model, and the input data is convolved using the three-dimensional convolution kernel to extract features in the time and space dimensions; through multiple convolutional layers and pooling layers, more advanced and abstract features are gradually extracted; and then a time series analysis is performed through the LSTM model to determine autism-related behavioral manifestations.

[0013] Furthermore, in step 8, the behavioral trend is predicted by first analyzing the behavioral data of autistic children through the LSTM model to predict the behavioral trend in the future. The formula is as follows: , in, It's prediction time behavioral state; It is historical behavioral data; is the prediction function of the LSTM model.

[0014] A classroom behavior analysis system for children with autism, based on the above analysis method, includes: The multi-view video acquisition and synchronization module collects videos from cameras set up at different locations in the classroom, performs time alignment, spatial geometry correction, and camera extrinsic calibration, and unifies the data from different camera perspectives into a three-dimensional coordinate system. The joint point detection module is responsible for normalizing the video, adjusting the size and color channels, and then using the VGG network to extract deep semantic features. After multi-stage optimization, the two-dimensional coordinates of the key points and their confidence levels are obtained. The temporal dynamic threshold module calculates the mean, standard deviation, and dynamic threshold based on the confidence of key points in the past N frames, divides the confidence levels, and uses retention, Kalman filtering smoothing, or linear interpolation recovery processing for different levels to obtain more accurate and stable key point coordinates; The multi-view 3D reconstruction module collects the visible viewpoints of key points, constructs a projection matrix, calculates weights based on confidence, baseline length, and depth information, filters outliers using a random sample consistency algorithm, constructs an objective function based on the screening of inliers, and performs bundle adjustment after iterative optimization using the LM algorithm. The autism-specific action recognition module uses a trained deep learning model to identify autism-specific actions. The model uses 3DCNN to extract spatial features from video frames and then uses a timing model to perform time series analysis to identify autism-related behavioral manifestations. The behavioral analysis and feedback module conducts in-depth behavioral assessments based on the identified autism-specific movement data, predicts trends and generates behavioral analysis reports. It also combines reinforcement learning technology to provide personalized educational intervention recommendations.

[0015] The beneficial effects of the present invention are: By setting up multiple cameras in the classroom, all-round data collection is achieved, and through precise data synchronization technology, classroom behavior is fully covered to reduce information loss; By leveraging OpenPose technology and combining it with time-series dynamic threshold optimization, the problem of key point jitter in existing technologies is effectively overcome, providing more stable and reliable key point data. By constructing a projection matrix, weighted calculation, filtering outliers, optimizing algorithms, and eliminating errors, high-precision multi-view 3D reconstruction is achieved, accurately restoring the behavior in 3D space. By closely integrating DSM-5 core behaviors and symptoms of motor coordination disorders, and using deep learning models to analyze specific movement patterns from multiple dimensions, children's behavior can be assessed more scientifically and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is the overall framework diagram of the classroom behavior analysis system for children with autism; Figure 2 An overview of behavioral analysis of children with autism from a teacher's perspective; Figure 3 Behavior analysis and feedback interface for autistic children from the teacher's perspective. DETAILED DESCRIPTION

[0017] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0018] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention. Therefore, the figures only show components related to the present invention and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0019] A classroom behavior analysis system for children with autism includes a multi-view video acquisition and synchronization module, a joint point detection module, a dynamic threshold module, a multi-view 3D reconstruction module, an autism-specific action recognition module, and a behavior analysis and feedback module.

[0020] The multi-view video acquisition and synchronization module collects videos through four cameras in different corners of the classroom, performs time alignment, spatial geometry correction and camera external parameter calibration, to achieve complete coverage of students' classroom behavior, reduce information loss, and unify data from different camera perspectives in a three-dimensional coordinate system, providing panoramic and accurate basic data for subsequent analysis.

[0021] The joint point detection module is responsible for normalizing the video, adjusting the size and color channels, and then using the VGG network to extract deep semantic features. After multi-stage optimization, the two-dimensional coordinates of the key points and their confidence levels are obtained. This module can accurately determine the positions of the key points on the human body and provide key information for behavioral analysis.

[0022] The temporal dynamic threshold module calculates the mean, standard deviation and dynamic threshold based on the confidence of the key points of the past N frames, divides the confidence levels, and adopts retention, Kalman filtering smoothing or linear interpolation recovery processing methods for different levels to obtain more accurate and stable key point coordinates, thereby improving the stability and reliability of joint point detection.

[0023] The multi-view 3D reconstruction module collects the visible viewpoints of key points, constructs a projection matrix, calculates weights based on confidence, baseline length, and depth information, filters outliers through RANSAC, constructs an objective function based on the screening of inliers, and performs Bundle Adjustment after iterative optimization using the LM algorithm to eliminate cumulative errors, improve the accuracy and consistency of multi-view 3D reconstruction, and provide accurate spatial information for behavior analysis.

[0024] The autism-specific action recognition module uses a deep learning model to identify autism-specific actions, such as self-stimulation, stereotyped behaviors, and other specific behaviors in the Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition) (DSM-5). The model extracts spatial features in video frames through 3DCNN, and then performs time series analysis through timing models and LSTM models to accurately judge autism-related behavioral manifestations and provide a basis for subsequent intervention.

[0025] The behavior analysis and feedback module generates a behavior analysis report based on in-depth behavior assessment and trend prediction based on motion recognition data, and provides personalized educational intervention suggestions to help teachers and parents understand children's behavioral status and develop targeted education and intervention measures to help the growth and development of children with autism.

[0026] A method for analyzing classroom behavior of children with autism, comprising the following steps: Step 1: Video is collected by one or more cameras at designated locations in the classroom. In this example, high-definition cameras are installed at the four corners of the top of a standard square classroom: the front left, front right, rear left, and rear right. Step 2: Use the checkerboard calibration method to unify the different collected videos into a three-dimensional coordinate system for representation; Step 3: Preprocess the collected video and use the VGG network to extract deep semantic features. After multi-stage optimization, the two-dimensional coordinates and confidence levels of the key points of the human body in the video are obtained. Step 4: Calculate the mean, standard deviation, and dynamic threshold based on the confidence of key points in the past N frames of video images, and divide the confidence levels; Step 5: Depending on the confidence level, the preservation, Kalman filtering smoothing, or linear interpolation recovery processing methods are adopted to obtain stable and reliable two-dimensional coordinates of the key points; Step 6: Collect the visible angles of key points, construct a projection matrix, and calculate weights based on confidence, baseline length, and depth information. Then, use the RANSAC algorithm to filter outliers. Based on the inliers, construct the objective function. Then, use the LM algorithm for iterative optimization and perform light velocity adjustment to eliminate accumulated errors. Step 7: Use the trained deep learning model to determine whether typical autism behaviors are present in the video based on the selected key points. If so, identify the category of the autism behavior and record the action recognition data. Otherwise, terminate the step. Step 8: Based on the recorded action recognition data, use the trained LSTM network to analyze the action recognition data, predict behavioral trends, generate a behavior analysis report, and provide personalized intervention recommendations.

[0027] In step 1, four cameras continuously collect video data at a frame rate of 30 frames per second, the video resolution is set to 1920×1080, and the cameras are connected to the computer via a high-speed network cable to ensure the stability and real-time performance of data transmission.

[0028] In step 2, time alignment is achieved using a software synchronization method. By using NTP to synchronize computer time, the system time of the computer connected to the camera is synchronized with a high-precision time server on the Internet. When the camera captures video, the system records the timestamp corresponding to each frame of image captured. The timestamp deviation for each frame is then calculated, and linear interpolation is used to adjust the frame's time alignment error. For example, at a certain moment, the 500th frame of image is captured by the left front camera. The timestamp recorded for this image is compared with the timestamp of the corresponding frame captured by the right front camera to calculate the timestamp deviation. The system then uses a linear interpolation algorithm to fine-tune the time of subsequent frames captured by the right front camera based on the timestamps and deviation values ​​of the previous and next frames. Ultimately, the video streams of all cameras will be aligned under the same time reference, ensuring consistency of data between frames.

[0029] In step 2, after completing the time alignment, spatial geometric correction is also required. Because the perspectives of multiple cameras are different, perspective transformation and geometric alignment are required for subsequent stereo matching and multi-view 3D reconstruction. By setting multiple obvious image feature points in the classroom, the image processing algorithm is used to obtain the coordinates of these feature points in different camera images; using the left front camera as a reference, the perspective matrix of other cameras relative to the left front camera is calculated; for example, for the right front camera, according to the feature point coordinates Calculate the perspective matrix ; Use the perspective transformation formula to transform the image captured by the right front camera to obtain the coordinates in the left front camera's perspective : , in, is the coordinate in the target perspective, that is, the perspective of the left front camera; is the coordinate in the source view, that is, the right front camera time; is the perspective matrix, which is calculated through image feature points.

[0030] The camera extrinsic calibration is used for global alignment. Checkerboard calibration is used to obtain the rotation matrix R and translation vector T of each camera. Specifically, a checkerboard pattern is placed at different locations and angles in the classroom, with four cameras circling the pattern. By analyzing the coordinate positions of feature points on the checkerboard in the images of different cameras, a calibration algorithm is used to calculate the rotation matrix and translation vector of each camera. Finally, the camera at the specified position is used as the reference for the global coordinate system. In this example, the left front camera is specified, and the other cameras are unified into a global coordinate system. The transformation formula for the other cameras is: , in, is a point in the camera coordinate system; is a point in the global coordinate system; and Calculated by checkerboard calibration. Through geometric transformation of image coordinates, the perspective data of different cameras are uniformly represented in a 3D coordinate system, laying the foundation for subsequent multi-view 3D reconstruction.

[0031] In step 3, when obtaining the key points of the human body, each frame of the captured video needs to be preprocessed first. During the preprocessing process, normalization is first performed to map the pixel values ​​of the image to the interval [0, 1]. At the same time, according to the input requirements of the subsequent VGG network, the image size is adjusted to 224×224, and the color channels are adjusted to conform to the input format of the VGG network. Subsequently, the pre-trained VGG network is used to extract features from the preprocessed image, and the OpenPose network architecture is used to analyze the initial two-dimensional coordinates (x, y) of the key points of the human body and their confidence C; the key points of the human body include nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, hip center, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear, left foot inner, left foot outer, left heel, right foot inner, right foot outer, right heel and other parts; this process is the basis for the recovery of three-dimensional human body key points, which can ensure the accuracy of subsequent depth information matching and multi-view three-dimensional reconstruction.

[0032] Since the OpenPose network may experience jitter, false detection, or key point loss during single-frame detection, a temporal dynamic threshold is used in combination with the previous and next frame information to optimize the key point confidence and improve detection stability.

[0033] In step 4, the confidence level is calculated by first calculating the mean confidence level of the key point in the past N frames. , in this case, the past 5 frames of images; then calculate the confidence standard deviation σ of the past N frames to measure the fluctuation; finally set the dynamic threshold ; The formula is as follows: , , , in, Represents the confidence of the i-th frame image in the past at the current time t; is a set value used to control the size of the dynamic threshold; the current frame confidence C is compared with the dynamic threshold and the confidence mean, and the results shown in Table 1 are obtained: Table 1 Comparison results of the current frame confidence C with the dynamic threshold and the confidence mean

[0034] For the key point confidence C of the current frame, if C> , then it is high confidence and the key points are retained directly; if ≤C≤ , use Kalman filtering for smoothing; if C< , try to fix it using linear interpolation.

[0035] In step 5, Kalman filtering can smooth the positions of key points, making them more stable, reducing jitter, and making the trajectory of key points more reasonable. When performing Kalman filtering, the current moment is k, and the state of the key point at the previous moment is The state transfer matrix F is set according to the physical law of key point motion, and the control input When there is no external control, it is set to 0, and the uncertainty covariance matrix of the forecast Initialized according to the empirical value, the process noise covariance matrix Q is set according to the environmental noise conditions, and the process noise Obeys normal distribution.

[0036] The process of Kalman filtering of key points is as follows: Step 51: Based on the state of the previous moment Predict the current state : , in, is the predicted state; is the state transition matrix; It is the state of the key point at the previous moment; is the control input; the uncertainty covariance matrix of the prediction is then determined : , in, is the uncertainty covariance matrix of the forecast; is the process noise covariance matrix; is the process noise; represents the transposed matrix of the state transition matrix; Represents the uncertainty covariance matrix at the previous moment.

[0037] Step 52: Observe and update to get the final smooth keypoint coordinates and the updated uncertainty covariance matrix ; Among them, when obtaining the key point coordinates calculated by the OpenPose network When , calculate the Kalman gain: , in, is the Kalman gain, which is used to balance the predicted value and the measured value; is the observation matrix, which is set according to the coordinate observation method. is its transposed matrix; is the observation noise covariance matrix, which is set according to the observation error; then the observation update is performed to obtain: , , in, are the key point coordinates observed by OpenPose; is the final smooth keypoint coordinate; is the updated uncertainty covariance matrix; Represents the identity matrix.

[0038] The linear interpolation method for key points with low confidence can reasonably estimate the key point coordinates of the lost frame, making the motion trajectory of the key point smoother. Assuming that the key point is reliably detected in the two frames at time t−1 and t+1, but is lost in the tth frame, the following formula can be used for interpolation: , in, is the key point information in the t-frame image, which is the coordinate information in this case; is the key point information in the t-1 frame image; is the keypoint information in the t+1 frame. It should be noted that if a keypoint is missing in both the preceding and following frames, it is not interpolated and is removed from the interpolated frame, marked as "missing." If a keypoint exists only in the preceding or following frame, the keypoint coordinates of the most recent frame are used as the keypoint information in the interpolated frame.

[0039] In step 6, the specific process is as follows: Step 61: Collect the key point visible viewpoints; determine which key points under which viewpoints are available through the two-dimensional key points and their confidence in the four viewpoints, so as to retain the available key points; Step 62: Construct the projection matrix; Step 63: Calculate weights by combining confidence, baseline length, and depth information to weight each projection equation. Step 64: RANSAC algorithm filters abnormal matching points and selects inliers; Step 65: Based on the inliers selected by RANSAC, construct an objective function that minimizes the reprojection error; Step 66: Use the LM algorithm to iteratively optimize the objective function and continuously update the 3D points until the convergence condition is met; Step 67: Use the light velocity adjustment method to eliminate the cumulative error and minimize the sum of the reprojection errors of all 3D points under each camera perspective; Step 68: Use the LM algorithm to solve the objective function in step 67. During the solution process, update the camera parameters and the 3D point coordinates simultaneously until the convergence condition is met.

[0040] In step 61, the 2D keypoints and their confidence levels in the four viewpoints are used to determine which keypoints are usable, thereby retaining the usable keypoints. Specifically, the usable keypoints are determined based on the 2D keypoints collected by the four cameras and their confidence levels. For example, if a keypoint is detected in the images from the left front, right front, and left rear cameras with a high confidence level, the keypoint from all three viewpoints is retained as usable data.

[0041] In step 62, the number of projection matrices to be constructed is determined based on the number and distribution of available key points, and the projection matrix is ​​constructed. For example, in the above example, there are three available key point viewing angles, and the two-dimensional projection of each key point at different viewing angles is , the three-dimensional point in the world coordinate system is For each camera, its projection relationship can be expressed as: , in, is the camera's projection matrix, calculated as follows: , in, Represents the camera intrinsic parameter matrix, including focal length, principal point offset, etc. 、 Represents the camera extrinsic parameters, rotation matrix and translation vector.

[0042] In step 63, the weight is calculated by combining the confidence, baseline length and depth information, so as to weight each projection equation and improve the accuracy of triangulation. The calculation of weights usually considers three factors: the confidence of the key point, the baseline length between the perspectives and the depth information; the higher the confidence, the more reliable the detection result of the key point, so it should be given a higher weight; the longer the baseline, the greater the difference in the perspective of the two cameras, and the higher the accuracy of triangulation. Therefore, the longer the baseline length between the perspectives, the higher the weight should be given to the corresponding projection equation; the depth information reflects the distance between the object and the camera. Generally speaking, the closer the object is to the camera, the more accurate the key point detection is. Therefore, the smaller the depth value, the higher the weight should be given to the corresponding projection equation. The formula for calculating weights is as follows: , in, , It is depth information; is a monotonically increasing function of the baseline length; is the baseline length between the i-th perspective and other perspectives; Represents the confidence of the key point of the i-th perspective; In this example, it is specifically expressed as: , in, is the two-dimensional point detection noise; the projection equation weights of different cameras are calculated.

[0043] In step 64, RANSAC filters outliers because in practical applications, some key points may be mismatched due to occlusion, noise, and other issues. Therefore, RANSAC is needed to remove outliers. The specific steps are as follows: Step 641: Randomly select two viewpoints from all visible viewpoints and calculate the triangulated position; Step 642: Project the calculated 3D points back to all viewing angles and calculate the error; Step 643: Error Points with errors smaller than the set threshold are considered as inliers, and points with large errors are considered as outliers and are removed. Step 644: Repeat steps 641 to 643 to find the triangulation solution containing the most interior points as the final result.

[0044] In this example, two viewpoints are randomly selected from all visible viewpoints of the key point at the corresponding moment, such as the left front viewpoint and the right front viewpoint; the 3D position of the triangulation is calculated based on the projection matrix and the 2D key point coordinates; the calculated 3D point is projected back to all viewpoints and the projection error is calculated ; In this example, the error threshold is set to 5 pixels. Points with errors less than 5 pixels are considered inliers, and points with errors greater than 5 pixels are considered outliers and eliminated. To shorten the calculation time, the above steps are repeated 100 times in this example, and the triangulation solution containing the most inliers is found as the final result, and these inliers are retained.

[0045] In step 65, an objective function is constructed based on the screening of interior points, which is expressed as: , in, represents the set of interior points.

[0046] In step 66, the LM algorithm is used to iteratively optimize and continuously update the three-dimensional points. , until the convergence condition is met, where the convergence condition includes that the change in the reprojection error is less than a certain set threshold. In this example, in each iteration of the LM algorithm, the reprojection error is calculated based on the current 3D point coordinates, and the 3D point coordinates are adjusted until the change in the reprojection error is less than a certain threshold of 0.01, at which point the algorithm is considered to have converged.

[0047] In step 67, the bundle adjustment method (BundleAdjustment) is used to address the problem that errors may gradually accumulate during the reconstruction process of LM based on local optimization of a single key point, thereby affecting the global reconstruction accuracy and consistency; BundleAdjustment is used to eliminate the accumulated errors and improve the accuracy and consistency of the entire reconstruction result. Finally, the LM algorithm is used to solve the objective function of BundleAdjustment. During the solution process, the camera parameters and 3D point coordinates are updated simultaneously. The camera parameters include camera intrinsic parameters and camera extrinsic parameters until the convergence condition is met, where the convergence condition includes the change in reprojection error being less than a set threshold, as shown in the following formula: , in, is the set of parameters of all cameras; is the coordinate set of all three-dimensional points; is the number of cameras; is the number of 3D points; is the two-dimensional projection coordinate of the j-th three-dimensional point in the perspective of the i-th camera; is the projection matrix of the i-th camera; In this example, during the iteration of the LM algorithm, the camera parameters and 3D point coordinates are continuously adjusted until the change in the reprojection error is less than the threshold of 0.001, thereby improving the accuracy and consistency of the entire reconstruction result.

[0048] In step 7, the 3DCNN model is used for feature extraction. The specific process includes inputting the sorted three-dimensional key point data into the 3DCNN model. The 3DCNN uses a three-dimensional convolution kernel to perform a convolution operation on the input data to extract features in the time and space dimensions. Through multiple convolution layers and pooling layers, more advanced and abstract features are gradually extracted. The first layer of convolution may extract simple local features of the action. As the network depth increases, subsequent layers can extract complex action pattern features.

[0049] The feature maps extracted by the 3DCNN are then input into the LSTM network in chronological order. LSTMs excel at processing data with time series characteristics and can capture temporal trends and long-term dependencies in behavior. LSTM cells process the input feature sequence through a gating mechanism, including an input gate, a forget gate, and an output gate. The forget gate determines which information to discard from the cell state, the input gate determines which new information to add to the cell state, and the output gate determines which information to output from the cell state. In this way, LSTMs learn the patterns and dynamics of behavioral sequences. For example, when identifying self-stimulatory behaviors, LSTM networks are able to capture the duration and frequency of the behavior, as well as the correlations between actions.

[0050] The output of the LSTM network is used to classify the current behavior through a fully connected layer to determine whether the current behavior conforms to a specific behavior pattern in the DSM-5 or whether it conforms to symptoms of motor coordination disorder. The following is a table of typical motor behavior categories of autism: Table 2 Typical action and behavior categories of autism Behavior Category Specific behavior Social interaction deficits (Criterion A) Social avoidance (A1: Nonverbal communication deficits) Social interaction deficits (Criterion A) Joint attention deficit (A2: social reciprocity deficit) Social interaction deficits (Criterion A) Insufficient social initiation (A3: Barriers to relationship building) Repetitive and stereotyped behaviors (Criterion B) Stereotyped hand movements (B1: repetitive movements) Repetitive and stereotyped behaviors (Criterion B) The arrangement of items is stereotyped (B2: sticking to an unchanging pattern) Repetitive, stereotyped behaviors (Criterion B) Self-stimulation (B4: paresthesia) Abnormal motor coordination Walking on tiptoe Abnormal motor coordination joint stiffness The above table is explained in detail for the small kindergarten classroom scenario: Social avoidance (A1: Non-verbal communication deficit): The algorithm output will include the calculated hip social distance Shoulders facing the angle , low head angle and the duration t of these states. >1 meter, (turn your back to others), (head down and avoidance), and when the duration t>15 seconds that these three conditions are met at the same time, it can be determined that the young autistic children in the kindergarten show symptoms of social avoidance.

[0051] Lack of joint attention (A2: Lack of social reciprocity): The algorithm outputs the angle between the hand gesture and the head direction And the number of times n that the angle is less than a certain threshold per hour. (no angle threshold for joint attention), and within each hour When the number of times n is less than 1, the child is judged to have symptoms of joint attention deficit.

[0052] Insufficient social initiation (A3: relationship building barriers): The algorithm output includes the change in hip distance at different times And the number of times N that one approaches others actively per hour. A distance >0.4 meters is considered an active approach. When the number of active approaches N is <3 times per hour, it indicates that the child has symptoms of insufficient social initiation.

[0053] Hand stereotyped movements (B1: repetitive movements): The algorithm outputs the calculated cosine similarity of the wrist trajectory (for fine hand movements) and joint motion entropy For fine hand stereotyped movements, when the cosine similarity of wrist trajectories of three consecutive frames is >0.85, and joint motion entropy When the score is <0.35, the child is judged to have symptoms of hand stereotypy.

[0054] Object arrangement stereotype (B2: Invariant pattern): The algorithm outputs the three-dimensional variance of position changes per minute calculated by replacing the wrist coordinates with the position of the object held by the hand For small items commonly found in kindergartens (volume less than 0.01 cubic meters), When the area is less than 0.005 square meters, the child is considered to have stereotyped behavior in arranging objects.

[0055] Self-stimulation (B4: Paresthesia): For severe head shaking, the algorithm outputs the head acceleration calculated by second-order difference ; For large torso swings, the algorithm outputs the calculated torso swing distance .

[0056] When the head accelerates , or trunk swing distance If the height is >0.12 m, the child is judged to have self-stimulation symptoms.

[0057] Tiptoe walking: The algorithm outputs ankle Z-axis height information. If the ankle Z-axis height is greater than 0.04 meters for 10 consecutive steps, the child is diagnosed with tiptoe walking, a symptom of motor coordination abnormality.

[0058] Joint stiffness: The algorithm outputs the elbow range of motion and the standard deviation of the knee angle. , or the standard deviation of knee angle is less than , it was determined that the child had symptoms of abnormal movement coordination with joint stiffness.

[0059] By comparing the algorithm output results with pre-set thresholds and standards, it is possible to more accurately determine whether autistic children in kindergartens exhibit corresponding autism-related symptoms. For example, the algorithm calculates that the social distance between the hips of a child and his peers at a certain moment is 1.2 meters, and the shoulder angle is , the low head angle is , and these states last for 20 seconds, then it meets the judgment criteria of social avoidance; within an hour, the algorithm detects that the angle between the child's hand pointing movement and the head direction is more than , and the angle is less than If the number of times is 0, it can be judged that there is a lack of joint attention; the algorithm statistics that a child has only 2 cases in which the hip distance decreases by more than 0.4 meters within an hour, then it can be judged that the child has insufficient social initiation; the algorithm calculates that the cosine similarity of a child's wrist trajectory for 3 consecutive frames is 0.88, and the joint activity entropy is 0.3, which meets the judgment criteria of hand stereotyped movements; the algorithm calculates that the three-dimensional variance of the position change of a child's hand holding object (replaced by wrist coordinates) per minute is 0.003 square meters, then it can be judged that there is a symptom of object arrangement stereotype; the algorithm detects that the head acceleration of a child is 2.2 , the trunk swing distance is 0.15 meters, then the current behavior is judged as self-stimulation behavior; the algorithm monitors that the ankle Z-axis height of a child in 10 consecutive steps is 0.05 meters, then it can be judged that there is a symptom of tiptoe walking; the algorithm calculates that the range of motion of a child's elbow is Or the standard deviation of knee angle is It should be noted that when autism-typical movements and behaviors are detected, information such as the time, duration, and frequency of the behaviors also needs to be recorded.

[0060] In step 8, the action recognition data comes from the output of the autism-specific action recognition module, including the child's behavior category, action duration, action frequency, and spatial location of the action.

[0061] The trend prediction is to analyze the behavioral data of autistic children through time series modeling and LSTM networks to predict behavioral trends in the future. For example, based on the statistical number of self-stimulatory behaviors of a child in a class, the duration of each behavior, and the distribution of behaviors in different areas of the classroom, the severity of the child's behavior and the impact on learning activities are evaluated and analyzed, and the possibility of self-stimulatory behavior in the next three days and whether the frequency of the behavior will increase or decrease are predicted. Finally, the behavioral evaluation and trend prediction are compiled into a detailed behavioral analysis report, which includes the child's basic information, historical behavioral records, behavioral deviation indicators, behavioral frequency and duration statistics, and behavioral trend predictions. In the behavioral analysis report, it is presented in a combination of charts and text, using charts to show the frequency, average duration, and severity of different behaviors, using line charts to show behavioral trend predictions, and using bar charts to show behavioral deviation indicators.

[0062] The personalized intervention plan is based on an LSTM network that performs reinforcement learning. Its goal is to help children reduce inappropriate behaviors and enhance adaptive behaviors through appropriate education and behavior modification. Incorporating reinforcement learning and other techniques, the system can predict the potential effects of different intervention strategies. For example, it can determine whether a specific behavior management strategy can reduce a child's self-injurious behavior within a certain period of time, or whether a specific social interaction training program can improve a child's verbal communication skills.

[0063] When typical behaviors of autism are identified in step 7, real-time feedback will be provided to the teacher. Specifically, when a child's behavior changes, the system can provide real-time feedback to the teacher and, combined with the intervention plan given in step 8, suggest how to adjust the teaching strategy, such as diverting the child's attention by asking questions, guiding participation in activities, etc.

[0064] Compared with existing technologies, this classroom behavior analysis system for children with autism has significant advantages. In the data collection process, existing technologies mostly use a single or limited perspective, making it difficult to fully capture behavior. However, this system uses four cameras for all-round data collection and, through precise data synchronization technology, fully covers classroom behavior and reduces information loss. In key point detection, with the help of OpenPose technology and combined with time-series dynamic threshold optimization, it effectively overcomes the key point jitter problem in existing technologies and provides more stable and reliable data. In terms of multi-perspective 3D reconstruction, this system achieves high-precision multi-perspective 3D reconstruction through a series of complex and sophisticated operations, such as constructing a projection matrix, weighted calculations, filtering outliers, algorithm optimization, and error elimination, accurately restoring the situation of behavioral movements in three-dimensional space. In behavioral analysis, it closely combines DSM-5 core behaviors and symptoms of motor coordination disorders, and uses deep learning models to analyze specific movement patterns from multiple dimensions, which can more scientifically and accurately assess children's behavior. Overall, this system organically integrates advantages such as multi-perspective acquisition, stable key point detection, high-precision multi-perspective 3D reconstruction, and in-depth behavioral analysis to form a complete and efficient system, which greatly improves the accuracy of classroom behavior analysis for children with autism, provides a solid and scientific basis for personalized educational intervention, and brings more valuable results.

[0065] The above description is merely a specific example of the present invention and does not constitute any limitation thereto. It is apparent to those skilled in the art, after understanding the content and principles of the present invention, that various modifications and alterations in form and detail may be made without departing from the principles and structure of the present invention. However, such modifications and alterations based on the concepts of the present invention remain within the scope of protection of the claims of the present invention.

Claims

1. A method for analyzing classroom behavior of children with autism, characterized by: The steps include: Step 1: Collect video from one or more cameras at designated locations in the classroom; Step 2: Use the checkerboard calibration method to unify the different collected videos into a three-dimensional coordinate system for representation; Step 3: Preprocess the collected video and use the VGG network to extract deep semantic features. After multi-stage optimization, the two-dimensional coordinates and confidence levels of the key points of the human body in the video are obtained. Step 4: Calculate the mean, standard deviation, and dynamic threshold based on the confidence of key points in the past N frames of video images, and divide the confidence levels; Step 5: Depending on the confidence level, the preservation, Kalman filtering smoothing, or linear interpolation recovery processing methods are adopted to obtain stable and reliable two-dimensional coordinates of the key points; Step 6: Collect the visible angles of key points, construct a projection matrix, and calculate weights based on confidence, baseline length, and depth information. Then, use the RANSAC algorithm to filter outliers. Based on the inliers, construct the objective function. Then, use the LM algorithm for iterative optimization and perform light velocity adjustment to eliminate accumulated errors. Step 7: Use the trained deep learning model to determine whether typical autism behaviors are present in the video based on the selected key points. If autism behaviors are present, identify the category of the autism behaviors and record the action recognition data. Otherwise, end the step; Step 8: Based on the recorded action recognition data, use the trained LSTM network to analyze the action recognition data, predict behavioral trends, generate a behavior analysis report, and provide personalized intervention recommendations.

2. A method for analyzing classroom behavior of children with autism according to claim 1, characterized in that: In step 2, the cameras installed in the four corners of the classroom respectively capture videos in four directions, and the videos captured in the four directions are aligned in time, geometrically corrected in space, and then the videos are uniformly converted into a global coordinate system.

3. The method for analyzing classroom behavior of autistic children according to claim 1, characterized in that: The pre-processing of the video in step 3 includes normalization, size adjustment, and color channel adjustment.

4. The method for analyzing classroom behavior of autistic children according to claim 1, characterized in that: In step 3, the OpenPose network architecture is used to analyze and detect the key points of the human body in the video. The key points include nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, hip center, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear, left foot inside, left foot outside, left heel, right foot inside, right foot outside, and right heel, and the two-dimensional coordinates (x, y) of each key point in the corresponding video frame image and its confidence C are extracted.

5. A method for analyzing classroom behavior of children with autism according to claim 4, characterized in that: The process of dividing the confidence levels in step 4 is as follows: First, calculate the mean confidence value of the key point in the past N frames of image , then calculate the confidence standard deviation σ of the past N frames and set the dynamic threshold , as shown below: ， ， ， in, Represents the confidence of the i-th frame image in the past at the current time t; is the set value used to control the size of the dynamic threshold; In step 5, for the confidence level greater than The key points of are directly retained as high confidence key points; for confidence ≤C≤ , as the key points with medium confidence, the Kalman filter is used for smoothing; for the key points with confidence C less than , as the degree confidence key point, and restore it using linear interpolation.

6. The method for analyzing classroom behavior of autistic children according to claim 5, characterized in that: The specific process of step 6 is as follows: Step 61: Collect the key point visible viewpoints; determine which key points under which viewpoints are available through the two-dimensional key points and their confidence in the four viewpoints, so as to retain the available key points; Step 62: Construct the projection matrix; A 3D point in the world coordinate system The two-dimensional projection under multiple viewing angles is , the projection relationship of each camera is expressed as: ， ， in, is the camera's projection matrix; Camera intrinsic parameter matrix; 、 Camera extrinsics; Step 63: Calculate the weight by combining the confidence, baseline length, and depth information to weight each projection equation. The formula is as follows: ， in, , It is depth information; is a monotonically increasing function of the baseline length; is the baseline length between the i-th perspective and other perspectives; Represents the confidence of the key point of the i-th perspective; Step 64: RANSAC algorithm filters abnormal matching points and selects inliers; Step 65: Based on the inliers selected by RANSAC, construct an objective function that minimizes the reprojection error: ， in, represents the set of interior points; Step 66: Use the LM algorithm to iteratively optimize the objective function and continuously update the 3D points until the convergence condition is met; Step 67: Use the light velocity adjustment method to eliminate the cumulative error and minimize the sum of the reprojection errors of all 3D points under each camera perspective. The objective function is as follows: ， in, is the set of parameters of all cameras; is the coordinate set of all three-dimensional points; is the number of cameras; is the number of 3D points; is the two-dimensional projection coordinate of the j-th three-dimensional point in the perspective of the i-th camera; is the projection matrix of the i-th camera; is the weight; Step 68: Use the LM algorithm to solve the objective function in step 67. During the solution process, update the camera parameters and the 3D point coordinates simultaneously until the convergence condition is met.

7. The method for analyzing classroom behavior of children with autism according to claim 6, characterized in that: The process of filtering abnormal matching points in step 64 includes: Step 641: Randomly select two viewpoints from all visible viewpoints and calculate the triangulated position points; Step 642: Project the calculated position point back to all viewing angles and calculate the error ; Step 643: Error Points with errors smaller than the set threshold are considered as inliers, and points with large errors are considered as outliers and are removed. Step 644: Repeat steps 641 to 643 to find the triangulation with the most inliers as the final structure, and retain these inliers.

8. The method for analyzing classroom behavior of children with autism according to claim 1, characterized in that: In step 7, the sorted three-dimensional key point data is first input into the three-dimensional convolutional neural network model, and a three-dimensional convolution kernel is used to perform a convolution operation on the input data to extract features in the time and space dimensions; through multiple convolutional layers and pooling layers, more advanced and abstract features are gradually extracted; and then a time series analysis is performed through the LSTM model to determine the behavioral manifestations related to autism.

9. The method for analyzing classroom behavior of children with autism according to claim 1, characterized in that: In step 8, the behavioral trend is predicted by first analyzing the behavioral data of autistic children through the LSTM model to predict the behavioral trend in the future. The formula is as follows: ， in, It's prediction time behavioral state; It is historical behavioral data; is the prediction function of the LSTM model.

10. A classroom behavior analysis system for children with autism, characterized by: Based on the analysis method according to any one of claims 1 to 9, the analysis system comprises: The multi-view video acquisition and synchronization module collects videos from cameras set up at different locations in the classroom, performs time alignment, spatial geometry correction, and camera extrinsic calibration, and unifies the data from different camera perspectives into a three-dimensional coordinate system. The joint point detection module is responsible for normalizing the video, adjusting the size and color channels, and then using the VGG network to extract deep semantic features. After multi-stage optimization, the two-dimensional coordinates of the key points and their confidence levels are obtained. The temporal dynamic threshold module calculates the mean, standard deviation, and dynamic threshold based on the confidence of key points in the past N frames, divides the confidence levels, and uses retention, Kalman filtering smoothing, or linear interpolation recovery processing for different levels to obtain more accurate and stable key point coordinates; The multi-view 3D reconstruction module collects the visible viewpoints of key points, constructs a projection matrix, calculates weights based on confidence, baseline length, and depth information, filters outliers using a random sample consistency algorithm, constructs an objective function based on the screening of inliers, and performs bundle adjustment after iterative optimization using the LM algorithm. The autism-specific action recognition module uses a trained deep learning model to identify autism-specific actions. The model uses 3DCNN to extract spatial features from video frames and then uses a timing model to perform time series analysis to identify autism-related behavioral manifestations. The behavioral analysis and feedback module conducts in-depth behavioral assessments based on the identified autism-specific movement data, predicts trends and generates behavioral analysis reports. It also combines reinforcement learning technology to provide personalized educational intervention recommendations.

Images