A method and system for analyzing classroom behavior of autistic children

Abstract

The application provides a kind of autism children classroom behavior analysis method and analysis system, and the system will multi-view collection, stable key point detection, high-precision multi-view three-dimensional reconstruction and deep behavior analysis mode, through the construction of projection matrix, weighted calculation, filtering abnormal points, algorithm optimization and error elimination etc. Link realizes organic integration, forms a complete and efficient system, greatly improves the accuracy of autism children classroom behavior analysis, provides solid and scientific basis for individualized education intervention, and brings more valuable analysis results.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video image processing, in particular to a method and system for analyzing classroom behavior of autistic children. BACKGROUND

[0002] Autism spectrum disorder is a common neurodevelopmental disorder characterized by deficits in social interaction, communication, and repetitive and stereotyped behaviors. Current treatment methods mainly include educational intervention, psychological therapy and drug therapy; in the process of educational intervention, there are multiple links such as behavior analysis, speech therapy and social training, among which classroom behavior analysis is crucial for the educational intervention process of autistic children. Through behavior analysis, positive behaviors can be strengthened, problem behaviors can be reduced, and children's social skills and behavior patterns can be improved.

[0003] In the traditional way, the behavior analysis of autistic children mainly relies on manual observation and recording. This method is not only inefficient, but also easily affected by subjective factors, making it difficult to fully and accurately capture the behavior characteristics of children. In order to overcome the problems in the traditional way, an audio and video-based analysis screening system has appeared, such as the content shown in the patent with the authorization announcement number CN110313923B. However, in the existing video analysis method, video data collected from a single perspective is usually used, which cannot provide complete panoramic information, easily leading to information loss and affecting the subsequent analysis and diagnosis results. Therefore, a multi-perspective observation method is needed to monitor and analyze the behavior of autistic children in all directions. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the prior art and provide a method and system for analyzing classroom behavior of autistic children.

[0005] In order to solve the above problems, the present application adopts the following technical solutions:

[0006] A method for analyzing classroom behavior of autistic children, comprising the following steps:

[0007] Step 1: Collecting video from one or more set positions of cameras in the classroom;

[0008] Step 2: Unifying different videos collected to a three-dimensional coordinate system for representation by using a checkerboard calibration method;

[0009] Step 3: Preprocessing the collected video and extracting deep semantic features using VGG network to obtain two-dimensional coordinates of key points of human body in the video and their confidence levels after multi-stage optimization;

[0010] Step 4: Calculating the mean, standard deviation and dynamic threshold of the key point confidence levels in the past N frames of video images respectively, and dividing the confidence levels into grades;

[0011] Step 5: Based on the confidence level, adopt the following processing methods respectively: preservation, Kalman filtering smoothing, or linear interpolation recovery, to obtain stable and reliable two-dimensional coordinates of key points;

[0012] Step 6: Collect the visible viewpoints of key points, construct the projection matrix, and calculate the weights by combining confidence, baseline length and depth information. Then, use the RANSAC algorithm to filter out outliers, construct the objective function based on the selected inliers, and perform light speed adjustment after iterative optimization using the LM algorithm to eliminate accumulated errors.

[0013] Step 7: Using the trained deep learning model, based on the selected key points, determine whether there are typical autistic behaviors in the video; if autistic behaviors are found, identify the category of autistic behaviors and record the action recognition data; otherwise, end the step.

[0014] Step 8: Based on the recorded action recognition data, use the trained LSTM network to analyze the action recognition data, predict behavioral trends, generate a behavior analysis report, and provide personalized intervention suggestions.

[0015] Furthermore, in step 2, cameras located in the four corners of the classroom collect videos from four different directions. The videos collected from the four directions are then aligned in time and geometrically corrected in space before being converted into a unified global coordinate system.

[0016] Furthermore, the video preprocessing in step 3 includes normalization, resizing, and color channel adjustment.

[0017] Furthermore, in step 3, the OpenPose network architecture is used to analyze and detect key points of the human body in the video. The key points include the nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, center of the hip, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear, inside left foot, outside left foot, left heel, inside right foot, outside right foot, and right heel. The two-dimensional coordinates (x, y) of each key point in the corresponding video frame image and its confidence C are extracted.

[0018] Furthermore, the process of classifying confidence levels in step 4 is as follows:

[0019] First, calculate the mean confidence score C of this keypoint within the past N frames of images. avg Then, the confidence standard deviation σ of the past N frames is calculated, and a dynamic threshold T is set. dynamic The details are as follows:

[0020]

[0021] Tdynamic = C avg -k·σ,

[0022] where C t-i represents the confidence of the i-th frame image at the current time t; k is a set value for controlling the size of the dynamic threshold;

[0023] In step 5, for the key points with confidence greater than C avg , they are directly reserved as high-confidence key points; for the key points with confidence T dynamic ≤ C ≤ C avg , they are smoothed by Kalman filtering as medium-confidence key points; for the key points with confidence C < T dynamic , they are recovered by linear interpolation as low-confidence key points.

[0024] Further, the specific process of step 6 is as follows:

[0025] Step 61: Collect the visible view angles of the key points; determine which key points in which view angles are available by the two-dimensional key points and their confidences in the four view angles, so as to retain the available key points;

[0026] Step 62: Construct the projection matrix;

[0027] A three-dimensional point X = (X, Y, Z) T in a world coordinate system is projected in multiple view angles as x i = (x i , y i ) T The projection relationship of each camera is expressed as:

[0028] x i = Π i X,

[0029] ∏ i = K i [R i ∣t i ],

[0030] where Π i is the projection matrix of the camera; K i is the intrinsic matrix of the camera; R i , t i are the camera extrinsic parameters;

[0031] Step 63: Calculate the weight combining the confidence, baseline length and depth information, so as to weight each projection equation, the formula is as follows:

[0032] w i = c pose,i × f(baselinei ) x g(D),

[0033] wherein, D is depth information; f(baseline i ) is a monotonically increasing function with respect to baseline length; baseline i is the baseline length between the i-th view and other views; c pose,i represents the key point confidence of the i-th view;

[0034] Step 64: the RANSAC algorithm filters the abnormal matching points to screen out inliers;

[0035] Step 65: based on the inliers screened out by RANSAC, a target function of minimizing the re-projection error is constructed:

[0036]

[0037] wherein, inliers represents the set of inliers;

[0038] Step 66: the LM algorithm is used to iteratively optimize the target function, and the coordinates of the three-dimensional points X are continuously updated until the convergence condition is met;

[0039] Step 67: the method of photometric adjustment is used to eliminate cumulative errors and minimize the total sum of re-projection errors of all three-dimensional points under each camera view, and the target function is as follows:

[0040]

[0041] wherein, C is the parameter set of all cameras; X is the coordinate set of all three-dimensional points; n c is the number of cameras; n p is the number of three-dimensional points; x ij is the two-dimensional projection coordinates of the j-th three-dimensional point under the i-th camera view; Π(C i ) is the projection matrix of the i-th camera; w ij is the weight;

[0042] Step 68: the LM algorithm is used to solve the target function in step 67, and the camera parameters and three-dimensional point coordinates are updated simultaneously during the solving process until the convergence condition is met.

[0043] Further, the process of filtering abnormal matching points in step 64 comprises:

[0044] Step 641: randomly select two views from all visible views, and calculate the triangulation position point;

[0045] Step 642: project the calculated position point back to all views, and calculate the error ei ;

[0046] Step 643: error e i Points less than the set threshold are considered as inner points, and points with large errors are considered as outer points and are removed.

[0047] Step 644: repeat steps 641-643 to find the triangulation containing the most inner points as the final structure, and retain these inner points.

[0048] Further, in step 7, the sorted three-dimensional key point data is first input into a three-dimensional convolutional neural network model, and a three-dimensional convolution kernel is used to perform convolution operation on the input data to extract features in time and space dimensions; through multiple convolution layers and pooling layers, higher-level and more abstract features are gradually extracted; then, time series analysis is performed through an LSTM model to judge autism-related behavior performance.

[0049] Further, in step 8, the behavior trend is predicted, first, the behavior data of autistic children is analyzed through an LSTM model to predict the behavior trend in the future period of time, and the formula is as follows:

[0050]

[0051] wherein, is the behavior state at the prediction time t+k; X t ,X t-1 ,...,X t-k is historical behavior data; f() is the prediction function of the LSTM model.

[0052] An autism child classroom behavior analysis system based on the above analysis method, the analysis system comprises:

[0053] A multi-view video acquisition and synchronization module acquires video through cameras arranged at different positions in the classroom, performs time alignment, spatial geometry correction and camera external parameter calibration, and unifies different camera perspective data in a three-dimensional coordinate system;

[0054] A joint point detection module is responsible for normalizing, adjusting size and color channel, etc. after preprocessing of the video, using a VGG network to extract deep semantic features, and obtaining two-dimensional coordinates of key points and their confidence through multi-stage optimization;

[0055] A time sequence dynamic threshold module calculates the mean, standard deviation and dynamic threshold according to the confidence of the past N frames of key points, divides the confidence level, and adopts retention, Kalman filter smoothing or linear interpolation recovery processing mode for different levels to obtain more accurate and stable key point coordinates;

[0056] The multi-view three-dimensional reconstruction module can realize high-precision multi-view three-dimensional reconstruction by means of projection matrix construction, weighted calculation, filtering of abnormal points, algorithm optimization and error elimination, and accurately restore the behavior action in the three-dimensional space.

[0057] The autism-specific action recognition module adopts a trained deep learning model to recognize autism-specific actions, wherein the model extracts spatial features in the video frame through a 3DCNN, and then performs time series analysis through a time series model to judge autism-related behavior performance.

[0058] The behavior analysis and feedback module performs in-depth behavior evaluation according to the recognized autism-specific action data, predicts trends and generates a behavior analysis report, and proposes individualized education intervention suggestions in combination with reinforcement learning technology.

[0059] The autism-specific action recognition module adopts a trained deep learning model to recognize autism-specific actions, wherein the model extracts spatial features in the video frame through a 3DCNN, and then performs time series analysis through a time series model to judge autism-related behavior performance.

[0060] By setting multiple cameras in the classroom, all-around collection is realized, and through accurate data synchronization technology, classroom behavior is completely covered, and information loss is reduced.

[0061] By means of OpenPose technology and in combination with time series dynamic threshold optimization, the problem of key point jittering in the prior art is effectively overcome, and more stable and reliable key point data is provided.

[0062] By means of projection matrix construction, weighted calculation, filtering of abnormal points, algorithm optimization and error elimination, high-precision multi-view three-dimensional reconstruction is realized, and behavior actions are accurately restored in three-dimensional space.

[0063] By closely combining DSM-5 core behaviors and movement coordination abnormal symptoms, and using a deep learning model to analyze specific action patterns from multiple dimensions, the behavior of children is more scientifically and accurately evaluated. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 It is a whole framework diagram of the classroom behavior analysis system for autistic children.

[0065] Figure 2 It is a behavior analysis overview of autistic children from the perspective of teachers.

[0066] Figure 3 It is a behavior analysis and feedback interface of autistic children from the perspective of teachers. DETAILED DESCRIPTION

[0067] Following, the advantages and effects of the present application can be easily understood by those skilled in the art from the disclosure of the present specification. The present application can also be implemented or applied by other different specific embodiments, and various modifications or changes can be made to the details in the specification without departing from the spirit of the present application. It should be noted that the following examples and features in the examples can be combined with each other without conflict.

[0068] It should be noted that the diagrams provided in the following examples only illustrate the basic concept of the present application in a schematic manner, and only the components related to the present application are shown in the diagrams, not the number, shape and size of the components when actually implemented. The actual implementation of each component may be a random change in type, number and proportion, and the component layout pattern may also be more complex.

[0069] An autism classroom behavior analysis system includes a multi-view video acquisition and synchronization module, a joint point detection module, a dynamic threshold module, a multi-view three-dimensional reconstruction module, an autism-specific action recognition module, and a behavior analysis and feedback module.

[0070] The multi-view video acquisition and synchronization module collects video through four cameras in different corners of the classroom, performs time alignment, spatial geometry correction and camera external parameter calibration, realizes complete coverage of student classroom behavior, reduces information loss, and unifies different camera perspective data in a three-dimensional coordinate system, providing panoramic and accurate basic data for subsequent analysis.

[0071] The joint point detection module is responsible for normalizing, adjusting the size and color channel of the video after preprocessing, extracting deep semantic features using the VGG network, and obtaining two-dimensional coordinates of key points and their confidence through multi-stage optimization. It can accurately determine the position of human key points and provide key information for behavior analysis.

[0072] The time sequence dynamic threshold module calculates the mean, standard deviation and dynamic threshold based on the key point confidence of the past N frames, divides the confidence level, and uses retention, Kalman filter smoothing or linear interpolation recovery processing for different levels to obtain more accurate and stable key point coordinates, improving the stability and reliability of joint point detection.

[0073] The multi-view three-dimensional reconstruction module collects key point visible perspectives, constructs a projection matrix, calculates weights based on confidence, baseline length and depth information, filters abnormal points through RANSAC, constructs a target function based on the selection of inliers, and performs BundleAdjustment after iterative optimization using the LM algorithm to eliminate cumulative errors and improve the accuracy and consistency of multi-view three-dimensional reconstruction, providing accurate spatial information for behavior analysis.

[0074] The autism-specific action recognition module uses a deep learning model to recognize autism-specific actions such as self-stimulation, stereotyped behavior, and other specific behaviors in the Diagnostic and Statistical Manual of Mental Disorders (5th Edition) (DSM-5). The model extracts spatial features in video frames through 3DCNN, and then performs time series analysis through a time series model, LSTM model, to accurately determine autism-related behavior performance and provide a basis for subsequent intervention.

[0075] The behavior analysis and feedback module generates a behavior analysis report based on the action recognition data, and provides personalized education intervention suggestions to help teachers and parents understand the behavior of children, develop targeted education and intervention measures, and provide help for the growth and development of autistic children.

[0076] An autism child classroom behavior analysis method, comprising the following steps:

[0077] Step 1: Collect video from one or more set positions in the classroom; in this example, high-definition cameras are installed in the four corners of the top of the standard square classroom, i.e. left front, right front, left rear, and right rear;

[0078] Step 2: Use the checkerboard calibration method to unify the different videos collected into a three-dimensional coordinate system for representation;

[0079] Step 3: Preprocess the collected video and extract deep semantic features using the VGG network, and obtain the two-dimensional coordinates of the key points of the human body in the video and their confidence levels through multi-stage optimization;

[0080] Step 4: Calculate the mean, standard deviation, and dynamic threshold based on the key point confidence levels in the past N frames of video images, and divide the confidence levels into grades;

[0081] Step 5: Based on the confidence level, respectively take the retention, Kalman filter smoothing or linear interpolation recovery processing method to obtain stable and reliable key point two-dimensional coordinates;

[0082] Step 6: Collect the visible perspective of the key points, construct the projection matrix, and calculate the weight combining the confidence level, baseline length and depth information, then use the RANSAC algorithm to filter the abnormal points, and construct the target function based on the selected inner points, and then perform light speed adjustment using the LM algorithm for iterative optimization to eliminate cumulative errors;

[0083] Step 7: Use the trained deep learning model to determine whether there is a typical behavior of autism in the video based on the filtered key points; if there is an autism behavior, identify the category of the autism behavior and record the action recognition data; otherwise, end the step;

[0084] Step 8: Based on the recorded action recognition data, the trained LSTM network is used to analyze the action recognition data and predict behavior trends, generate behavior analysis reports, and give personalized intervention suggestions.

[0085] In step 1, four cameras continuously collect video data at a frame rate of 30 frames per second, with a video resolution of 1920x1080. The cameras are connected to the computer via high-speed network cables to ensure data transmission stability and real-time performance.

[0086] In step 2, a software synchronization method is used to achieve time alignment. By using NTP to synchronize computer time, the system time of the computer connected to the camera is synchronized with the high-precision time server on the Internet. When the camera collects video, it records the timestamp corresponding to each frame of image. Then, the timestamp deviation of each frame is calculated, and linear interpolation is used to adjust the time alignment error of the frame. For example, at a certain time, the 500th frame of image is collected by the left front camera, and the timestamp recorded by this image is compared with the timestamp of the corresponding frame of image collected by the right front camera. The timestamp deviation is calculated, and the system uses linear interpolation algorithm to fine-tune the time of subsequent right front camera frames. Finally, all camera video streams are aligned under the same time reference, ensuring the consistency of inter-frame data.

[0087] After completing the time alignment in step 2, spatial geometric correction is also needed because the angles of view of multiple cameras are different. Perspective transformation and geometric alignment are required for subsequent stereo matching and multi-view three-dimensional reconstruction. By setting multiple obvious image feature points in the classroom, image processing algorithms are used to obtain the coordinates of these feature points in different camera images. Taking the left front camera as a reference, the perspective matrix of other cameras relative to the left front camera is calculated. For example, for the right front camera, the perspective matrix H is calculated according to the feature point coordinates x. Using the perspective transformation formula, the image collected by the right front camera is transformed to obtain the coordinates x' in the left front camera view:

[0088] x' = H x, where x' is the coordinate in the target view, i.e., the left front camera view; x is the coordinate in the source view, i.e., the right front camera view; H is the perspective matrix, which is calculated by image feature points.

[0089] The camera external parameter calibration is used for global alignment, and the rotation matrix R and the translation vector T of each camera are obtained by the chessboard calibration; specifically, the chessboard pattern is placed at different set positions and angles in the classroom, and the chessboard pattern is respectively detected by the four cameras; by analyzing the coordinate positions of the feature points on the chessboard in the images of different cameras, the rotation matrix and the translation vector of each camera are calculated by using a calibration algorithm. Finally, the camera at the specified position is taken as the reference of the global coordinate system, and in this case, the left front camera is specified, and the other cameras are unified into a global coordinate system, and the transformation formula of the other cameras is:

[0090] P world =R·P camera +T,

[0091] wherein P camera is a point in the camera coordinate system; P world is a point in the global coordinate system; R and T are calculated by the chessboard calibration. By geometric transformation of the image coordinates, the visual angle data of different cameras is uniformly represented in a three-dimensional coordinate system, laying a foundation for subsequent multi-view three-dimensional reconstruction.

[0092] In step 3, when obtaining the key points of the human body, first, the pre-processing of each frame of image in the collected video is needed; in the pre-processing process, first, the image pixel value is mapped to the [0, 1] interval by using the normalization processing; at the same time, according to the input requirements of the subsequent VGG network, the image size is adjusted to 224*224, and the color channel is adjusted to meet the input format of the VGG network. Subsequently, the pre-trained VGG network is used to extract features from the pre-processed image, and the initial two-dimensional coordinates (x, y) and the confidence C of the human body key points are obtained by using the OpenPose network architecture analysis; wherein the human body key points include nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, hip center, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear, left foot in, left foot out, left heel, right foot in, right foot out, right heel, etc. This process is the basis of three-dimensional human key point recovery, which can ensure the accuracy of subsequent depth information matching and multi-view three-dimensional reconstruction.

[0093] Since the OpenPose network may have jitter, false detection or key point loss when detecting a single frame, a time sequence dynamic threshold is used to optimize the key point confidence combined with the information of the previous and subsequent frames to improve the detection stability.

[0094] In step 4, the calculation method of the confidence level is first to calculate the confidence mean C avg of the key point in the past N frames, which is 5 frames of images in this case; then the confidence standard deviation σ of the past N frames is calculated to measure the fluctuation; finally, a dynamic threshold Tdynamic ; the formula is as follows:

[0095]

[0096] T dynamic =C avg -k·σ,

[0097] wherein, C t-i represents the confidence of the i-th frame image in the past time t; k is a set value, used for controlling the size of the dynamic threshold; the current frame confidence C is compared with the dynamic threshold and the confidence average value, to obtain the results shown in Table 1 as follows:

[0098] Table 1 Comparison results of the current frame confidence C with the dynamic threshold and the confidence average value

[0099] For the key point confidence C of the current frame, if C>C avg , it is high confidence, and the key point is directly reserved; if T dynamic ≤C≤C avg , Kalman filtering is used for smoothing processing; if C<T dynamic , linear interpolation repair is tried.

[0100] In step 5, Kalman filtering can smooth the positions of the key points, so that they are more stable, reduce jitter, and make the trajectory of the key points more reasonable. When Kalman filtering is performed, the current time is k, the state of the key point at the last time is X k-1 , the state transition matrix F is set according to the physical law of key point movement, and the control input item BU k is set to 0 when there is no external control, and the predicted uncertainty covariance matrix P k - is initialized according to the experience value, and the process noise covariance matrix Q is set according to the environmental noise, and the process noise w k obeys normal distribution.

[0101] The process of Kalman filtering of the key point is as follows:

[0102] Step 51: predicting the state of the current time according to the state X k-1 of the last time

[0103]

[0104] wherein, is the predicted state; F is the state transition matrix; X k-1 is the state of the key point at the last time; BU kis the control input; then determine the predicted uncertainty covariance matrix P k - :

[0105] P k - = FP k-1 F T + Q, wherein P k - is the predicted uncertainty covariance matrix; Q is the process noise covariance matrix; w k is the process noise; F T represents the transpose matrix of the state transition matrix; P k-1 represents the uncertainty covariance matrix at the previous time.

[0106] Step 52: observation update, to obtain the final smoothed key point coordinates X k and the updated uncertainty covariance matrix P k ; wherein, when the key point coordinates Z k calculated by the OpenPose network are obtained, the Kalman gain is calculated:

[0107] K k = P k - H T (HP k - H T + R) -1 ,

[0108] wherein K k is the Kalman gain, used to balance the predicted value and the measured value; H is the observation matrix, set according to the coordinate observation mode, H T is the transpose matrix thereof; R is the observation noise covariance matrix, set according to the observation error condition; then the observation update is performed, to obtain:

[0109]

[0110] P k = (I-K k H)P k - ,

[0111] wherein Z k is the key point coordinates observed by the OpenPose; X k is the final smoothed key point coordinates; P k is the updated uncertainty covariance matrix; I represents the unit matrix.

[0112] The linear interpolation method for low-confidence key points can reasonably estimate the key point coordinates of the missing frame, making the motion trajectory of the key points smoother; assuming that the key points are reliably detected in both frames at time t-1 and t+1, but missing in the t-th frame, the following formula can be used for interpolation:

[0113]

[0114] where x t is the key point information in the t-th frame image, which is coordinate information in this example; x t-1 is the key point information in the t-1-th frame image; and x t+1 is the key point information in the t+1-th frame image. It should be noted that if the key points are missing in both the previous and subsequent frames, the key points are not interpolated, and the key points in the interpolated frame image are excluded and marked as "missing"; if the key point information only exists in the previous frame or the subsequent frame, the key point coordinates of the nearest frame are used as the key point information in the interpolated frame image.

[0115] In step 6, the specific process is as follows:

[0116] Step 61: Collect the visible view angles of the key points; determine which key points in which view angles are available through the two-dimensional key points and their confidence in the four view angles, so as to retain the available key points;

[0117] Step 62: Construct the projection matrix;

[0118] Step 63: Calculate the weight in combination with the confidence, baseline length and depth information, so as to weight each projection equation;

[0119] Step 64: Filter the abnormal matching points by using the RANSAC algorithm, and screen out the inliers;

[0120] Step 65: Based on the inliers screened out by RANSAC, construct a target function for minimizing the re-projection error;

[0121] Step 66: Use the LM algorithm to iteratively optimize the target function, continuously update the coordinates of the three-dimensional points X, until the convergence condition is met;

[0122] Step 67: Use the method of light speed adjustment to eliminate cumulative errors and minimize the total re-projection error of all three-dimensional points in each camera view;

[0123] Step 68: Use the LM algorithm to solve the target function in step 67, and simultaneously update the camera parameters and three-dimensional point coordinates during the solving process, until the convergence condition is met.

[0124] In step 61, it is necessary to determine which key points in which view are available by using the two-dimensional key points and their confidence in the four views, so as to retain the available key points. Specifically, it is necessary to determine which key points in which view are available according to the two-dimensional key points and their confidence collected by the four cameras; for example, if a key point is detected in the images of the left front, right front and left rear cameras and has high confidence, then the key point in the three views is retained as available data.

[0125] In step 62, the number of projection matrices to be constructed is determined according to the number and distribution of available key points, and the projection matrices are constructed. For example, in the above example, there are three views with available key points, and the two-dimensional projection of each key point in different views is x i =(x i ,y i ) T , and the three-dimensional point in the world coordinate system is X=(X, Y, Z) T . For each camera, the projection relationship can be expressed as: x i =Π i X,

[0126] wherein Π i is the projection matrix of the camera, and the calculation method is as follows:

[0127] Π i =K i [R i ∣t i ],

[0128] wherein K i represents the camera intrinsic matrix, including focal length, principal point offset, etc.; R i , t i represent the camera extrinsic parameters, rotation matrix and translation vector.

[0129] In step 63, the weight is calculated in combination with the confidence, baseline length and depth information, so as to weight each projection equation and improve the accuracy of triangulation. The calculation of the weight usually considers three factors: the confidence of the key point, the baseline length between the views and the depth information; the higher the confidence, the more reliable the detection result of the key point, so a higher weight should be given; the longer the baseline, the greater the difference between the views of the two cameras, and the higher the accuracy of triangulation, so the longer the baseline length between the views, the higher the weight of the corresponding projection equation; the depth information reflects the distance between the object and the camera, and generally speaking, the closer the object to the camera, the more accurate the detection of the key point, so the smaller the depth value, the higher the weight of the corresponding projection equation. The formula for calculating the weight is as follows:

[0130] w i =c pose,ix f(baseline i ) x g(D),

[0131] wherein, D is depth information; f(baseline i ) is a monotonically increasing function about baseline length; baseline i is the baseline length of the ith view and other views; c pose,i represents the key point confidence of the ith view;

[0132] In this example, it is specifically represented as:

[0133]

[0134] wherein, σ i 2 is a two-dimensional point detection noise; the projection equation weight of different cameras is calculated.

[0135] In the step 64, the RANSAC filters the abnormal points, because in actual application, some key points may be mismatched due to occlusion, noise and other problems, and therefore RANSAC is needed to eliminate abnormal points, and the specific steps are as follows:

[0136] Step 641: two views are randomly selected from all visible views, and the triangulation position is calculated;

[0137] Step 642: the calculated 3D point is projected back to all views, and the error is calculated;

[0138] Step 643: the error e i of the point is less than the set threshold value, and the point is regarded as an inlier, and the point with large error is regarded as an outlier and eliminated;

[0139] Step 644: steps 641-643 are repeated to find the triangulation solution containing the most inliers as the final result.

[0140] In this example, two views are randomly selected from all visible views of the key points at the corresponding moment, such as the left front view and the right front view; the 3D position of triangulation is calculated according to the projection matrix and the two-dimensional key point coordinates; the calculated 3D point is projected back to all views, and the projection error e i is calculated; in this example, the error threshold is set to 5 pixels, the point with an error less than 5 pixels is regarded as an inlier, and the point with an error greater than 5 pixels is regarded as an outlier and eliminated; in order to shorten the calculation time, the above steps are repeated 100 times in this example, and the triangulation solution containing the most inliers is found as the final result, and the inliers are retained.

[0141] In the step 65, the target function is constructed based on the screened inliers, and is represented as:

[0142]

[0143] wherein inliers represents a set of inliers.

[0144] In step 66, the coordinates of the three-dimensional points X are continuously updated by iterative optimization using the LM algorithm until a convergence condition is met, wherein the convergence condition includes that the change of the re-projection error is less than a certain threshold value; in this example, in each iteration of the LM algorithm, the re-projection error is calculated according to the current three-dimensional point coordinates, the three-dimensional point coordinates are adjusted, and the algorithm is considered to converge until the change of the re-projection error is less than a certain threshold value 0.01.

[0145] In step 67, the Bundle Adjustment method is used to solve the problem that the LM algorithm is based on a single key point for local optimization, and the error may gradually accumulate in the reconstruction process, thereby affecting the global reconstruction accuracy and consistency; the Bundle Adjustment is used to eliminate the accumulated error and improve the accuracy and consistency of the entire reconstruction result, and finally the LM algorithm is used to solve the objective function of the Bundle Adjustment. In the solving process, the camera parameters and the three-dimensional point coordinates are updated simultaneously, the camera parameters include the camera intrinsic parameters and the camera extrinsic parameters, and the convergence condition includes that the change of the re-projection error is less than a certain threshold value, as shown in the following formula:

[0146]

[0147] wherein C is a parameter set of all cameras; X is a coordinate set of all three-dimensional points; n c is the number of cameras; n p is the number of three-dimensional points; x ij is the two-dimensional projection coordinates of the jth three-dimensional point under the visual angle of the ith camera; Π(C i ) is the projection matrix of the ith camera; w ij is the weight. In this example, when the LM algorithm is iterated, the camera parameters and the three-dimensional point coordinates are continuously adjusted until the change of the re-projection error is less than the threshold value 0.001, thereby improving the accuracy and consistency of the entire reconstruction result.

[0148] In step 7, the 3DCNN model feature extraction is used, and the specific process includes inputting the sorted three-dimensional key point data into the 3DCNN model, the 3DCNN uses a three-dimensional convolution kernel to perform convolution operation on the input data, and extracts features in the time and space dimensions; through multiple convolution layers and pooling layers, higher-level and more abstract features are gradually extracted; the first layer of convolution may extract simple local features of actions, and with the increase of the network depth, the subsequent layers can extract complex action pattern features.

[0149] The 3D CNN extracted feature maps are then input into an LSTM network in chronological order. LSTM is good at processing data with time series characteristics and can capture the trend of behavior changes over time and long-term dependencies. LSTM units process the input feature sequence through a gating mechanism, including an input gate, a forget gate, and an output gate. The forget gate determines which information needs to be discarded from the cell state, the input gate determines which new information needs to be added to the cell state, and the output gate determines which information needs to be output from the cell state. In this way, LSTM learns the patterns and dynamic changes of the behavior sequence. For example, when identifying self-stimulating behaviors, the LSTM network can capture the duration, frequency, and association between actions of the behavior.

[0150] The output of the LSTM network is classified by a fully connected layer to determine whether the current behavior matches the pattern of a specific behavior in DSM-5 or whether it matches the symptoms of a movement coordination disorder. The following is a table of typical action behaviors for autism:

[0151] Table 2: Table of typical action behaviors for autism

[0152] Behavioral categories Specific behaviors Social interaction deficits (Criterion A) Social avoidance (A1 : Deficits in nonverbal communicative behaviors) Social interaction deficits (Criterion A) Joint attention deficits (A2: Deficits in social reciprocity) Social interaction deficits (Criterion A) Social initiation deficits (A3: Impaired ability to make use of social overtures) Repetitive stereotyped behaviors (Criterion B) Hand stereotypy (B1 : Repetitive motor movements) Repetitive stereotyped behaviors (Criterion B) Object collection stereotypy (B2: Persistent fascination with nonfunctional objects) Repetitive stereotyped behaviors (Criterion B) Self-stimulatory behaviors (B4: Sensory atypical behaviors) Abnormal motor movements Toe walking Abnormal motor movements Joint stiffness

[0153] The above table is explained in detail for a kindergarten small classroom scenario:

[0154] Social avoidance (A1: Non-verbal communication defect): The algorithm output includes the calculated hip social distance d hip , shoulder angle θ shoulder , head angle θ head , and the duration of these states t. When d hip > 1 meter, θ shoulder > 135° (facing away from others), θ head > 30° (head down avoidance), and the duration t > 15 seconds, it is determined that the kindergarten child with autism exhibits symptoms of social avoidance.

[0155] Joint attention deficit (A2: Lack of social reciprocity): The algorithm outputs the angle θ hand-head between the hand pointing action and the head orientation, as well as the number of times n per hour that the angle is less than a certain threshold. For kindergarten children with autism, when θ hand-head > 55° (no joint attention angle threshold), and the number of times n per hour that θ hand-head < 55° is less than 1, it is determined that the child has symptoms of joint attention deficit.

[0156] Social initiation deficiency (A3: Relationship establishment disorder): The algorithm output includes the hip distance change Δd hipand the number of times of active approaching N per hour. If Δd hip > 0.4 meters is regarded as an active approach, and when the number of active approaches N per hour is < 3 times, it is suggested that the child has symptoms of social initiation deficiency.

[0157] Hand stereotypy (B1: repetitive movement): the algorithm outputs the calculated wrist trajectory cosine similarity s1 (for fine hand movement) and joint activity entropy E. For fine hand stereotypy, when the cosine similarity s1 of the wrist trajectory of 3 consecutive frames is > 0.85, and the joint activity entropy E is < 0.35, it is determined that the child has symptoms of hand stereotypy.

[0158] Object arrangement stereotypy (B2: persistent pattern): the algorithm outputs the three-dimensional variance σ 2 of the position change per minute calculated by replacing the hand-held object position with the wrist coordinates. 2 For small objects commonly seen in kindergarten (volume < 0.01 cubic meters), when σ

[0159] Self-stimulation (B4: abnormal sensation): for head shaking, the algorithm outputs the head acceleration a head calculated by second-order difference; for trunk swinging, the algorithm outputs the trunk swinging distance d trunk calculated.

[0160] When the head acceleration a head > 2 m / s 2 , or the trunk swinging distance d trunk > 0.12 meters, it is determined that the child has symptoms of self-stimulation.

[0161] Toe-walking: the algorithm outputs ankle Z-axis height information. If the ankle Z-axis height is greater than 0.04 meters for 10 consecutive steps, it is determined that the child has symptoms of toe-walking, a movement coordination abnormality.

[0162] Joint stiffness: the algorithm outputs the elbow range of motion and the knee angle standard deviation. If the elbow range of motion is less than 20°, or the knee angle standard deviation is less than 6°, it is determined that the child has symptoms of joint stiffness, a movement coordination abnormality.

[0163] By comparing the algorithm output results with the pre-set threshold and standard, it can be more accurately judged whether the low-age autistic children in kindergarten show corresponding autism-related symptoms. For example, the algorithm calculates that the hip social distance of a child and his peers at a certain time is 1.2 meters, the shoulder angle is 140°, the head angle is 35°, and these states last for 20 seconds, which meets the judgment standard of social avoidance; within an hour, the algorithm detects that the angle between the child's hand indicating action and head orientation is greater than 55° many times, and the number of times when the angle is less than 55° is 0, which can be judged as the existence of joint attention deficiency; the algorithm counts that within an hour, a child has only 2 times of hip distance reduction greater than 0.4 meters, which can be judged as the child's social initiation deficiency; the algorithm calculates that the cosine similarity of the wrist trajectory of a child for 3 consecutive frames is 0.88, and the joint activity entropy is 0.3, which meets the judgment standard of hand stereotypy; the algorithm calculates that the three-dimensional variance of the position change of a child's hand holding an object (replaced by wrist coordinates) per minute is 0.003 square meters, which can be judged as the existence of object arrangement stereotypy; the algorithm detects that the head acceleration of a child is 2.2 m / s 2 , the trunk swing distance is 0.15 meters, so the current behavior is determined to be self-stimulating behavior; the algorithm monitors that the ankle Z-axis height of a child is 0.05 meters in 10 consecutive steps, which can be judged as toe walking; the algorithm calculates that the elbow range of motion of a child is 15° or the knee angle standard deviation is 5°, which is judged as joint stiffness. It should be noted that when the typical autistic motion behavior is detected, the time, duration, motion frequency and other information of the behavior occurrence need to be recorded.

[0164] In step 8, the motion recognition data comes from the output of the autism-specific motion recognition module, including the behavior category of the child, the duration of the motion, the frequency of the motion, the spatial position of the motion, etc.

[0165] The trend prediction is analyzed by time series modeling, LSTM network, to predict the behavior trend in the future period of time; for example, according to the number of times of self-stimulating behavior of a child in a class, the duration of each time, the behavior distribution in different areas of the classroom, the behavior severity and the impact on learning activities are evaluated and analyzed, and the possibility of self-stimulating behavior in the next three days and whether the behavior frequency will increase or decrease are predicted; finally, the behavior evaluation and trend prediction are arranged into a detailed behavior analysis report, which includes the basic information of the child, the historical behavior record, the behavior deviation index, the behavior frequency and duration statistics, the behavior trend prediction; in the behavior analysis report, the occurrence frequency, average duration and severity of different behaviors are displayed using charts and texts, the behavior trend prediction is displayed using a line chart, and the behavior deviation index is displayed using a bar chart.

[0166] The personalized intervention scheme needs to be based on an LSTM network that completes reinforcement learning, aiming to help children reduce inappropriate behaviors and enhance adaptive behaviors through appropriate education and behavior correction methods. In combination with technologies such as reinforcement learning, the system can predict the effects that different intervention strategies may bring. For example, whether a certain behavior management strategy can reduce a child's self-injurious behavior within a certain period of time, or whether a certain social interaction training can improve a child's language communication ability.

[0167] When the typical behavior of autism is identified in step 7, real-time feedback will also be given to the teacher side; Specifically, when the child's behavior changes, the system can give real-time feedback to the teacher, and combined with the intervention scheme given in step 8, it suggests how to adjust the teaching strategy, such as diverting the child's attention through questions, guiding participation in activities, etc.

[0168] Compared with the prior art, the classroom behavior analysis system for autistic children has significant advantages. Among them, in the data acquisition link, the prior art mostly uses single or limited visual angle, which is difficult to fully capture behavior, while the system uses four cameras to collect all-round, and through accurate data synchronization technology, it covers the classroom behavior completely and reduces information loss; In the key point detection, with the help of OpenPose technology and combined with time sequence dynamic threshold optimization, the problem of key point jitter in the prior art is effectively overcome, providing more stable and reliable data; In the aspect of multi-angle three-dimensional reconstruction, the system realizes high-precision multi-angle three-dimensional reconstruction through a series of complex and delicate operations such as constructing a projection matrix, weighted calculation, filtering abnormal points, algorithm optimization and error elimination, etc., accurately restoring the behavior action in three-dimensional space; In behavior analysis, closely combined with the core behaviors of DSM-5 and the symptoms of motor coordination abnormalities, the deep learning model is used to analyze specific action patterns from multiple dimensions, which can more scientifically and accurately evaluate the child's behavior. Overall, the system organically integrates multi-angle collection, stable key point detection, high-precision multi-angle three-dimensional reconstruction, and deep behavior analysis, etc. Advantageous links form a complete and efficient system, greatly improving the accuracy of classroom behavior analysis for autistic children, providing solid and scientific basis for personalized education intervention, and bringing more valuable results.

[0169] The above description is only one specific example of the present application and does not constitute any limitation on the present application. Obviously, for those skilled in the art, after understanding the content and principles of the present application, various modifications and changes in form and details can be made without departing from the principles and structures of the present application, but these modifications and changes based on the idea of the present application are still within the protection scope of the claims of the present application.

Claims

1. A method of analyzing classroom behavior of autistic children, characterized by, The method comprises the following steps: Step 1: collecting videos from cameras set in one or more positions in the classroom; Step 2: using a checkerboard calibration method to unify different videos collected to a three-dimensional coordinate system for representation; Step 3: pre-processing the collected videos, and extracting deep semantic features by using a VGG network, and obtaining two-dimensional coordinates of key points of a human body in the videos and confidence thereof through multi-stage optimization; Step 4: calculating mean, standard deviation and dynamic threshold according to confidence of key points in past N frames of video images respectively, and dividing confidence levels; Step 5: taking retaining, Kalman filter smoothing or linear interpolation recovery processing mode according to confidence levels respectively, to obtain stable and reliable two-dimensional coordinates of key points; Step 6: collecting visible view angles of key points, constructing a projection matrix, calculating weights in combination with confidence, baseline length and depth information, filtering abnormal points by using a RANSAC algorithm, constructing a target function based on screened inliers, and performing light speed adjustment to eliminate cumulative errors through iterative optimization by using an LM algorithm; Step 7: using a deep learning model trained, judging whether there is a typical behavior of autism in the video based on screened key points; if there is an autism behavior, identifying a category of the autism behavior, and recording motion recognition data; Otherwise, ending the step; Step 8: analyzing motion recognition data by using a trained LSTM network based on recorded motion recognition data, predicting a behavior trend, generating a behavior analysis report, and giving personalized intervention suggestions; The specific process of step 6 is as follows: Step 61: collecting visible view angles of key points; determining which key points in which view angles are available through two-dimensional key points and confidence thereof in four view angles, so as to retain available key points; Step 62: constructing a projection matrix; A three-dimensional point X = (X, Y, Z) in a world coordinate system T A two-dimensional projection under multiple viewing angles is x i = (x i , y i ) T The projection relationship of each camera is represented as: x i = Π i X, Π i = K i [R i | t i ], wherein, Π i is a projection matrix of the camera; K i is an intrinsic matrix of the camera; R i , t i is an extrinsic parameter of the camera; Step 63: calculating weights in combination with confidence, baseline length and depth information, so as to weight each projection equation, and the formula is as follows: w i = c pose,i × f(baseline i ) × g(D) wherein, D is depth information; f(baseline i ) is a monotonically increasing function with respect to baseline length; baseline i is the baseline length between the i-th view and other views; c pose,i denotes the keypoint confidence of the i-th view; Step 64: filtering abnormal matching points by using a RANSAC algorithm, and screening inliers; Step 65: constructing a target function of minimizing re-projection error based on inliers screened by the RANSAC; Wherein, inliers represents a set of inliers; Step 66: using an LM algorithm to iteratively optimize the target function, and continuously updating coordinates of three-dimensional points X until a convergence condition is met; Step 67: using a light speed adjustment method to eliminate cumulative errors, and minimizing a total sum of re-projection errors of all three-dimensional points in each camera view, and the target function is as follows: where C is the parameter set of all cameras; X is the coordinate set of all three-dimensional points; n c is the number of cameras; n p is the number of three-dimensional points; x ij is the two-dimensional projection coordinate of the jth three-dimensional point under the ith camera view; Π(C i ) is the projection matrix of the ith camera; w ij is the weight; Step 68: using an LM algorithm to solve the target function in step 67, and simultaneously updating camera parameters and three-dimensional point coordinates in the solving process until a convergence condition is met.

2. The method of claim 1, wherein the method comprises: In step 2, four cameras set in four corners of the classroom respectively collect four orientation videos, and after time alignment and spatial geometric correction, the videos are converted to a global coordinate system.

3. The method of claim 1, wherein the method further comprises: In step 3, the pre-processing of the video includes normalization, size adjustment and color channel adjustment.

4. The method of claim 1, wherein the method further comprises: In step 3, the key points of the human body in the video are analyzed and detected using the OpenPose network architecture, including nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, hip center, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear, left foot in, left foot out, left heel, right foot in, right foot out, and right heel, and the two-dimensional coordinates (x, y) of each key point in the corresponding video frame image and its confidence C are extracted.

5. The method of claim 4, wherein the autistic child's classroom behavior is analyzed by, In step 4, the confidence level division process is as follows: First, the confidence mean C of the key point within the past N frames of images is calculated avg Then, the confidence standard deviation σ of the past N frames is calculated, and a dynamic threshold T is set dynamic as shown below: T dynamic = C avg -k·σ, wherein C t-i represents the confidence of the i-th frame image in the past of the current time t; k is a set value for controlling the size of the dynamic threshold; The step 5, for the confidence greater than C avg , the key point is directly reserved as a high-confidence key point; for the confidence T dynamic ≤ C ≤ C avg , the key point is smoothed by using Kalman filtering as a medium-confidence key point; for the confidence C less than T dynamic , the key point is recovered by using linear interpolation as a low-confidence key point.

6. The method of claim 1, wherein the method further comprises: In step 64, the process of filtering abnormal matching points includes: Step 641: randomly select two views from all visible views, and calculate the triangulation position points; Step 642: Project the calculated position point back to all the perspectives, calculate the error e i ; Step 643: error e i Points with error less than a set threshold are considered inliers, and points with large error are considered outliers and are discarded. Step 644: repeat steps 641-643 to find the triangulation containing the most inliers as the final structure, and retain these inliers.

7. The method of claim 1, wherein the method further comprises: In step 7, the sorted three-dimensional key point data is first input into a three-dimensional convolutional neural network model, and a three-dimensional convolution kernel is used to perform convolution operation on the input data to extract features in time and space dimensions; through multiple convolution layers and pooling layers, higher-level and more abstract features are gradually extracted; then, time series analysis is performed through an LSTM model to judge autism-related behavior performance.

8. The method of claim 1, wherein the method further comprises: In step 8, the behavior trend is predicted by first analyzing the behavior data of autistic children through the LSTM model to predict the behavior trend in the future period of time, and the formula is as follows: wherein, is the behavior state at prediction time t+k; X t ,X t-1 ,...,X t-k is historical behavior data; f() is a prediction function of the LSTM model.

9. An autism child classroom behavior analysis system, characterized by, The analysis method according to any one of claims 1-8, the analysis system comprises: A multi-view video acquisition and synchronization module acquires videos through cameras arranged at different positions in the classroom, performs time alignment, spatial geometric correction, and camera external parameter calibration, and unifies different camera view data in a three-dimensional coordinate system; A joint point detection module is responsible for normalizing, adjusting the size and color channel of the video after preprocessing, extracting deep semantic features using a VGG network, and obtaining key point two-dimensional coordinates and their confidence through multi-stage optimization; A time series dynamic threshold module calculates the mean, standard deviation, and dynamic threshold based on the key point confidence of the past N frames, divides the confidence level, adopts retention, Kalman filter smoothing, or linear interpolation recovery processing for different levels, and obtains more accurate and stable key point coordinates; A multi-view three-dimensional reconstruction module collects key point visible views, constructs a projection matrix, calculates weights based on confidence, baseline length, and depth information, filters abnormal points through a random sample consistency algorithm, constructs a target function based on the selection of inliers, and performs bundle adjustment after iterative optimization by the LM algorithm; an autism-specific action recognition module uses a trained deep learning model to recognize autism-specific actions, wherein the model extracts spatial features in the video frame through a 3DCNN, and then performs time series analysis through a time series model to judge autism-related behavior performance; An action analysis and feedback module performs in-depth behavior evaluation based on the recognized autism-specific action data, predicts trends, generates behavior analysis reports, and proposes individualized education intervention suggestions combined with reinforcement learning technology.

Images