Remote delivery classroom multi-classroom collaborative teaching method and system based on video interaction

Abstract

The application relates to the technical field of distance education, and discloses a multi-classroom cooperative teaching method and system for a distance education classroom based on video interaction, which comprises the following steps: acquiring continuous video of a remote classroom and analyzing the video frame by frame to extract a facial feature point set of a remote student; calculating the offset of a pupil center relative to an eye contour and the deformation degree of a mouth edge according to the facial feature point set, constructing a line-of-sight direction vector and an expression numerical matrix; sequentially performing time sequence alignment, interpolation filling and dependency relationship extraction on the line-of-sight direction vector and the expression numerical matrix to obtain a numerical behavior sequence; inputting the numerical behavior sequence into a pre-trained classification model to obtain a mapping relationship probability value, and determining the learning state category of the remote student through threshold judgment; and if the learning state category belongs to a preset negative state set, generating a teaching control instruction and sending the teaching control instruction to a display terminal in a main classroom. The method can realize accurate perception of non-verbal behaviors of remote students and learning state recognition, and timely teaching intervention.

Description

Technical Field

[0001] This invention relates to the field of distance education technology, and in particular to a method and system for collaborative teaching in multiple classrooms via video-interactive remote delivery classrooms. Background Technology

[0002] Remote classrooms connect the main lecture hall with multiple remote classrooms via network technology, enabling the cross-regional sharing of high-quality educational resources. Some existing systems incorporate motion sensing technology to capture students' body movements such as raising their hands and nodding, attempting to assess participation in remote classrooms.

[0003] In one existing technology, a skeletal keypoint detection algorithm extracts the upper limb joint positions and head posture parameters of students. When an arm is raised and the hand is higher than the shoulder, it is identified as a hand-raising action; when a head is detected to be swinging back and forth in the vertical direction with an amplitude exceeding a preset angle threshold, it is identified as a nodding action. The system statistically summarizes the detected hand-raising and nodding frequencies within a preset time window and presents the statistical results as a numerical panel on the lecturer's display screen for the lecturer to refer to and judge the participation level of students at a distance. However, the above-mentioned motion perception technology is limited to preset large-amplitude limb movements such as raising hands and nodding. It cannot effectively capture non-verbal behaviors such as subtle deviations in gaze direction, gradual changes in the degree of mouth opening and closing, and instantaneous changes in facial expressions. Furthermore, the statistical summary can only reflect the frequency of students' actions at discrete time points and cannot map the dynamic temporal correlation between gaze deviation trajectory and facial expression change process to learning state recognition.

[0004] Therefore, existing technologies can only indirectly reflect the surface participation level in remote classrooms by recognizing the statistical frequency of large-scale physical movements such as raising hands and nodding. It is difficult to detect and respond in a timely manner to negative learning states caused by decreased attention or cognitive overload of remote students. Summary of the Invention

[0005] This invention provides a video-interactive remote classroom multi-classroom collaborative teaching method and system to solve the problem that existing technologies can only indirectly reflect the surface participation level of remote classrooms by recognizing the statistical frequency of large-amplitude body movements, making it difficult to detect and intervene in the negative learning state of remote students in a timely manner.

[0006] In a first aspect, to address the aforementioned technical problems, this invention provides a video-interactive remote delivery classroom multi-classroom collaborative teaching method, comprising: Acquire continuous video from a remote classroom, analyze the continuous video frame by frame, and extract the set of facial feature points of the remote students in the continuous video; Spatial coordinate points are extracted from the set of facial feature points. The offset of the pupil center relative to the eye contour and the degree of deformation of the mouth edge are determined. The offset is mapped by coordinates to construct a gaze direction vector. The degree of deformation is processed by matrix transformation to obtain the expression value matrix. The gaze direction vector and the facial expression numerical matrix are aligned in time series to obtain a synchronous feature sequence, which is then interpolated and filled to obtain a complete feature sequence. The temporal dependency between gaze deflection and facial expression change in the complete feature sequence is extracted to obtain a numerical behavior sequence. The numerical behavior sequence is input into a pre-trained classification model to obtain the mapping relationship probability value between the numerical behavior sequence and the preset cognitive state label. Based on the mapping relationship probability value, the learning state category of the remote student is determined by threshold judgment. If the learning state category belongs to a preset negative state set, then the pre-established intervention rule base is retrieved according to the learning state category, teaching control instructions for the remote student are generated, and sent to the display terminal of the main classroom.

[0007] Secondly, the present invention provides a video-interactive remote delivery classroom multi-classroom collaborative teaching system, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described above.

[0008] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention extracts the offset of the pupil center relative to the eye contour from the set of facial feature points of remote students to construct a gaze direction vector, and extracts the degree of deformation of the mouth edge to construct an expression value matrix. This enables synchronous quantitative perception of subtle deviations in gaze direction, gradual changes in mouth opening and closing, and instantaneous changes in facial expressions. In this way, it can capture non-verbal behavioral information other than large-scale limb movements such as raising hands and nodding. This overcomes the problem of coarse granularity in existing motion perception technologies and significantly improves the perception accuracy of the real classroom state of remote students.

[0009] (2) This invention transforms the gaze direction vector and facial expression numerical matrix into a numerical behavior sequence with temporal dependencies by performing time series alignment, interpolation filling and extraction of temporal dependencies. This allows the two non-verbal behavioral information, gaze and facial expression, to establish an interrelated evolutionary pattern on a continuous time axis. This enables a comprehensive judgment of the student's learning state from the perspective of gaze deflection direction and facial expression change trend. This solves the problem that existing technologies can only judge isolated behavioral data in a single frame and cannot use the change trend of non-verbal behavior on a continuous time axis to analyze the learning state. This makes the learning state recognition result more consistent with the actual change pattern of the student's cognitive state.

[0010] (3) This invention identifies the state by inputting numerical behavior sequences into a classification model, and when it is determined that the remote student is in a negative learning state, it retrieves the intervention rule base to generate teaching control instructions and sends them to the display terminal of the main classroom. This allows the main teacher to obtain the abnormal state information of the remote student in real time and trigger intervention without interrupting the teaching rhythm. It realizes a closed loop from state perception to teaching intervention, overcomes the shortcomings of existing solutions that only provide delayed feedback through Q&A or tests after class, and effectively improves the timeliness and precision of interactive teaching in multi-classroom collaborative teaching in remote delivery classrooms. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the process of a video-interactive remote delivery classroom multi-classroom collaborative teaching method provided in the first embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] Reference Figure 1 The first embodiment of the present invention provides a multi-classroom collaborative teaching method for remote delivery classrooms based on video interaction, including the following steps: S1, acquire continuous video from the remote classroom, analyze the continuous video frame by frame, and extract the set of facial feature points of the remote students in the continuous video; S2, extract spatial coordinate points from the set of facial feature points, determine the offset of the pupil center relative to the eye contour and the degree of deformation of the mouth edge, perform coordinate mapping on the offset, construct the gaze direction vector, and perform matrix transformation on the degree of deformation to obtain the expression value matrix. S3, perform time series alignment on the gaze direction vector and the facial expression numerical matrix to obtain a synchronous feature sequence and perform interpolation filling to obtain a complete feature sequence, extract the temporal dependency between gaze deflection and facial expression change in the complete feature sequence to obtain a numerical behavior sequence; S4, input the numerical behavior sequence into the pre-trained classification model to obtain the mapping relationship probability value between the numerical behavior sequence and the preset cognitive state label, and determine the learning state category of the remote student by threshold judgment based on the mapping relationship probability value. S5. If the learning state category belongs to a preset negative state set, then the pre-established intervention rule base is retrieved according to the learning state category, a teaching control instruction for the remote student is generated, and sent to the display terminal of the main classroom.

[0014] In step S1, a continuous video of the remote classroom is acquired, and the continuous video is analyzed frame by frame to extract the set of facial feature points of the remote students in the continuous video.

[0015] It should be noted that the first step is to acquire continuous video from the remote classrooms. This continuous video refers to the continuous video stream data generated by video capture devices deployed in each remote classroom, which captures the student area in real time. This video stream data is acquired at a preset frame rate. For example, the preset frame rate can be 20 frames per second (fps) or 25 fps, determined based on the performance limits of the remote classroom video capture devices and the network transmission bandwidth. When device performance allows and network bandwidth is sufficient, 25 fps is preferred to obtain smoother time-series data; when network bandwidth is limited, 20 fps is used to reduce data transmission volume. To reduce network bandwidth usage, the video capture devices typically compress and encode the acquired raw video signal using video encoding standards when outputting the continuous video stream data, and then transmit the compressed and encoded video stream data to the system via the network communication link. After receiving the compressed and encoded video stream data, the system decodes it to restore it to frame-by-frame original image data for subsequent image processing and analysis.

[0016] Specifically, the continuous video is analyzed frame by frame to extract a set of facial feature points of the distant student in the continuous video, including: The continuous video is decoded into color image frames, the color image frames are converted to grayscale, and a grayscale image sequence is output. Histogram equalization is performed on each frame of grayscale image in the grayscale image sequence to obtain an equalized image sequence; Calculate the grayscale integral projection curves in the horizontal and vertical directions based on the equalized image sequence; The eye region and mouth region are located based on the grayscale integral projection curve, and edge feature points are extracted from the eye region and mouth region to obtain a set of facial feature points.

[0017] It should be noted that the color image frame refers to image frame data expressed in the RGB color space, where each pixel's color information is represented by the component values ​​of the red, green, and blue color channels. Since the continuous video acquired by the system is compressed video stream data, it cannot be directly processed for image processing; therefore, decoding is required first. Decoding involves calling a decoder corresponding to the encoding standard to parse the compressed bitstream according to the rules defined by that standard, extracting the luminance and chrominance values ​​of each pixel in each frame from the bitstream, and restoring the compressed video stream data to frame-by-frame original image data. The decoded image data is in RGB format, and then undergoes grayscale conversion to become a grayscale image. A grayscale image is a single-channel image where each pixel uses only one grayscale value to represent its luminance information, with the grayscale value ranging from 0 to 255. The purpose of grayscale conversion is to reduce the three-channel color image to a single-channel grayscale image, thereby reducing the computational complexity of subsequent processing.

[0018] Specifically, for each pixel, the component values ​​of its R, G, and B channels are multiplied by preset weights and then summed to obtain the grayscale value of that pixel. For example, the most commonly used grayscale conversion formula in academia is the ITU-R BT.601 standard, which sets the weight of the red channel to 0.299, the green channel to 0.587, and the blue channel to 0.114. After performing the above weighted summation operation on all pixels, a grayscale image is output.

[0019] In one implementation, a single video capture device may simultaneously capture images of multiple remote students. To accurately extract facial feature points of the target student, this embodiment performs face detection on each frame of grayscale image. The face detection employs a well-known deep learning-based face detection algorithm, such as a multi-task cascaded convolutional network or RetinaFace. The specific implementation process of such algorithms can be found in existing technical literature and will not be elaborated here. After determining the face region of each remote student through face detection, a unique identifier is assigned to each remote student, ensuring that the same face region detected in subsequent frames can be associated with the same identifier, thereby guaranteeing that facial feature point extraction always belongs to the same target remote student. After determining the face bounding box of the target remote student, the grayscale image of the target student's face region is cropped from the grayscale image based on the bounding box, and processed frame by frame to obtain a grayscale image sequence.

[0020] It should be noted that before triggering the histogram equalization, the global contrast variance of each frame of grayscale image is first detected. Specifically, firstly, the mean grayscale value of all pixels in the grayscale image is calculated, that is, the sum of the grayscale values ​​of each pixel is divided by the total number of pixels; then, the difference between the grayscale value of each pixel and the mean grayscale value is calculated and squared; finally, the squared differences of all pixels are summed and divided by the total number of pixels to obtain the global contrast variance. If the global contrast variance of the current frame is detected to be lower than a preset variance threshold, the current frame is determined to have low contrast, and histogram equalization is triggered for enhancement processing. If the global contrast variance is not lower than the variance threshold, the current frame is determined to have normal contrast, histogram equalization is not triggered, and the grayscale image of that frame is directly used as the equalized image for subsequent processing. The variance threshold is calculated by acquiring multiple frames of student facial images under normal lighting conditions in a remote classroom, calculating the global contrast variance of each frame, and taking the average of the variance values ​​of all frames as the threshold. When the lighting conditions of the actual teaching scene change significantly, the system administrator can re-acquire facial images under the current scene and recalculate the average value to update the threshold. For example, the variance threshold can be set to 45.0. Through the above contrast detection and conditional triggering mechanism, equalization processing is only performed when the image quality is indeed insufficient, avoiding the problem of brightness jumps between adjacent frames caused by unconditional equalization frame by frame.

[0021] In one implementation, histogram equalization involves non-linearly stretching the grayscale histogram of a grayscale image, remapping the pixel distribution originally concentrated in a narrow grayscale range to the entire grayscale range, thereby improving the global contrast of the image. In actual teaching scenarios in remote classrooms, due to uneven lighting conditions, backlighting, or local shadows, the acquired student facial images often suffer from low contrast. Histogram equalization can make facial edges and organ contours clearer and more discernible, providing high-quality image data for subsequent eye and mouth region localization.

[0022] Specifically, for an input grayscale image frame, the number of pixels contained in each grayscale level is first counted to obtain a grayscale histogram of the grayscale image, where the grayscale levels are integers from 0 to 255. The grayscale histogram records the frequency of each grayscale level in the image, reflecting the distribution of pixel grayscale in the image. Then, the cumulative distribution function of each grayscale level is calculated based on the grayscale histogram. The cumulative distribution function records the proportion of the cumulative number of pixels from grayscale level 0 to the current grayscale level to the total number of pixels. After obtaining the cumulative proportion value of each grayscale level, for each pixel with a grayscale value of k in the original image, the cumulative proportion value corresponding to grayscale level k is multiplied by 255 and rounded to obtain the new grayscale value of that pixel after equalization. Through the above mapping, the pixel grayscale values ​​originally concentrated in a narrow range are stretched to the entire grayscale range of 0 to 255, thereby improving the global contrast of the image. After processing all grayscale images in sequence, an equalized image sequence is obtained, which serves as the input for the grayscale integral projection calculation step.

[0023] It should be noted that the gray-scale integral projection curve refers to the curve formed by accumulating and statistically analyzing the gray-scale values ​​of pixels along the horizontal or vertical direction in an image. The horizontal gray-scale integral projection curve reflects the distribution pattern of the sum of the gray-scale values ​​of pixels in each row in the vertical direction, while the vertical gray-scale integral projection curve reflects the distribution pattern of the sum of the gray-scale values ​​of pixels in each column in the horizontal direction. The gray-scale value of the human eye region is significantly lower than that of the surrounding facial skin region, appearing as a low-value area below the sides in the horizontal projection; the mouth region appears as a band of low values ​​in the horizontal projection. Based on this characteristic, the eye region and mouth region can be located by analyzing the projection curves.

[0024] Specifically, for a frame of equalized image, the gray values ​​of each row of pixels are accumulated to obtain the projection value of that row, and all the projection values ​​of the rows are arranged in order from top to bottom to form a gray-level integral projection curve in the horizontal direction; the gray values ​​of each column of pixels are accumulated to obtain the projection value of that column, and all the projection values ​​of the columns are arranged in order from left to right to form a gray-level integral projection curve in the vertical direction.

[0025] In one implementation, the vertical position of the eye region is determined based on the local minimum value location in the horizontal gray-scale integral projection curve, and the vertical position range of the mouth region is determined based on the upper and lower boundaries of the low-value band in the curve. The horizontal positions of the eye and mouth regions are determined separately based on the local minimum value locations in the vertical gray-scale integral projection curve. The bounding boxes of the eye and mouth regions are determined by combining the analysis results of the horizontal and vertical projection curves.

[0026] Specifically, within the located eye region, the system scans pixel by pixel along both horizontal and vertical directions, calculating the grayscale difference between each pixel and its neighboring pixels. Pixels with grayscale differences exceeding a preset threshold are marked as eye edge points. Similarly, within the located mouth region, the system scans pixel by pixel along both horizontal and vertical directions, calculating the grayscale difference between each pixel and its neighboring pixels. Pixels with grayscale differences exceeding a preset threshold are marked as mouth edge points. For example, the preset threshold is determined based on the overall grayscale dynamic range of the current image, such as 30% of the maximum grayscale difference among all neighboring pixels in the current region. The eye edge points and mouth edge points are combined to output the set of facial feature points for the distant student.

[0027] In step S2, spatial coordinate points are extracted from the set of facial feature points, the offset of the pupil center relative to the eye contour and the degree of deformation of the mouth edge are determined, the offset is mapped by coordinates, a gaze direction vector is constructed, and the degree of deformation is processed by matrix transformation to obtain the expression numerical matrix.

[0028] The step of extracting spatial coordinate points from the set of facial feature points and determining the offset of the pupil center relative to the eye contour and the degree of deformation of the mouth edge includes: Extract the first spatial coordinates of the eye contour and the second spatial coordinates of the mouth edge from the set of facial feature points; Locate the pupil edge points from the first spatial coordinates, calculate the geometric center of the pupil region formed by the pupil edge points, obtain the pupil center coordinates, calculate the geometric center of the region enclosed by the eye contour based on the first spatial coordinates, and obtain the geometric centroid of the eye contour. Calculate the horizontal and vertical pixel differences between the pupil center coordinates and the geometric centroid to obtain the offset of the pupil center relative to the eye contour; The position difference of the corresponding coordinate points is calculated by performing the second spatial coordinates corresponding to the current frame and the starting frame of the continuous video, and the displacement of each coordinate point is obtained. The displacements are arranged in order of coordinate points to form a displacement sequence, thereby obtaining the degree of deformation of the mouth edge.

[0029] It should be noted that the first spatial coordinates of the eye contour and the second spatial coordinates of the mouth edge are extracted from the facial feature point set. Both the first and second spatial coordinates are the coordinate data corresponding to the eye edge points and mouth edge points in the facial feature point set output in step S1. Specifically, the first spatial coordinates record the position of the eye contour in the image coordinate system, and the second spatial coordinates record the position of the mouth edge in the image coordinate system.

[0030] It should be noted that, by locating the pupil edge points from the first spatial coordinates, calculating the geometric center of the pupil region formed by the pupil edge points, and obtaining the pupil center coordinates, the geometric center of the region enclosed by the eye contour is calculated based on the first spatial coordinates, thus obtaining the geometric centroid of the eye contour. Specifically, the Cannibal edge detection algorithm is used to process the image of the region enclosed by the first spatial coordinates of the eye contour, outputting a single-pixel width edge image. The Cannibal edge detection algorithm is a mature and well-known algorithm in the field, and its specific implementation process will not be elaborated here. The single-pixel width edge image refers to a binary image in which the width of each edge line in the image is a single pixel, where the pixel value of the edge point is 1, and the pixel value of the non-edge point is 0.

[0031] In one implementation, each edge pixel in the single-pixel width edge image is traversed, and consecutive edge pixels are connected to form edge segments. An edge segment with an approximately closed loop shape and a gray-scale mean lower than the surrounding area is selected, and the edge pixels on this segment are defined as pupil edge points. The area enclosed by these pupil edge points is the pupil region. The geometric center of the pupil region is the average position of all pixel coordinates within the area enclosed by the pupil edge points. The geometric centroid of the eye contour is the average position of all pixel coordinates within the area enclosed by the eye contour. The horizontal and vertical pixel differences between the pupil center coordinates and the geometric centroid are calculated, and these differences are combined to form the offset of the pupil center relative to the eye contour. This offset, expressed in pixels, characterizes the degree of deviation of the pupil center from the geometric center of the eye region.

[0032] It should be noted that the displacement of each coordinate point is obtained by calculating the position difference between the corresponding second spatial coordinates of the current frame and the starting frame of the continuous video. Specifically, the starting frame is selected by extracting the second spatial coordinates of the mouth edge from the first N frames of the continuous video. Since the first frame is the starting moment of video recording and has no preceding frame for calculating inter-frame changes, starting from the second frame, the absolute values ​​of the displacements of each coordinate point relative to the previous frame are calculated sequentially. The frame with the smallest sum of absolute displacement values ​​is determined as the starting frame. If the minimum sum of absolute displacement values ​​in the first N frames is still greater than a preset static threshold, the frame with the smallest sum of absolute displacement values ​​is still selected as the starting frame. The displacement offset of this frame relative to the previous frame is then deducted in subsequent deformation calculations to reduce the systematic bias caused by slight facial expressions in the starting frame affecting subsequent deformation calculations.

[0033] For example, N can be 30 frames, and the static threshold can be set to 10 pixels. The static threshold of 10 pixels is obtained by acquiring 30 consecutive frames of facial images of a remote student in a completely still and expressionless state, calculating the sum of the absolute values ​​of the displacement of each feature point of the mouth edge between adjacent frames, and taking the maximum value of the sum of the displacement of all frames as the static threshold. This value can be automatically calculated and generated during the system initialization phase.

[0034] Specifically, the second spatial coordinate system includes the coordinates of multiple feature points distributed along the edge of the mouth, for example, it may include 24 feature points distributed along the edges of the upper and lower lips and the corners of the mouth. For each feature point, the horizontal and vertical coordinate values ​​of the point in the current frame are subtracted from the horizontal and vertical coordinate values ​​of the corresponding point in the starting frame, respectively, to obtain the displacement components of the feature point in the horizontal and vertical directions, which are recorded as each displacement amount in the order of the coordinate points.

[0035] In one implementation, the displacements are arranged in order of coordinate points to form a displacement sequence, thereby obtaining the degree of deformation of the mouth edge. Each displacement in the displacement sequence records the magnitude and direction of the positional change of each mouth edge coordinate point relative to the reference frame in the current frame, reflecting the degree of deformation of the mouth opening and closing, the direction and magnitude of the pulling of the corners of the mouth. For example, when a student opens their mouth, the upper lip edge feature point moves upward and the lower lip edge feature point moves downward, with the magnitude of each displacement increasing and the direction being opposite; when the corners of the student's mouth are raised, the feature points at both corners of the mouth move upward and outward; when the corners of the student's mouth are lowered, the feature points at both corners of the mouth move downward and outward.

[0036] The process of mapping the offset to a coordinate plane, constructing a gaze direction vector, and performing matrix transformation on the degree of deformation to obtain an expression numerical matrix includes: Based on the horizontal and vertical pixel differences in the offset, and combined with the preset camera intrinsic parameter matrix, the horizontal and vertical angular offsets in the three-dimensional space corresponding to the two-dimensional pixel offset are calculated. The horizontal angular offset is used as the yaw angle, and the vertical angular offset is used as the pitch angle. The yaw angle and the pitch angle are combined into a line-of-sight vector. Each displacement in the degree of deformation is mapped to a preset matrix dimension according to the corresponding coordinate point position to construct a deformation feature matrix; The deformation feature matrix is ​​subjected to orthogonal transformation and dimensionality reduction to extract feature vectors reflecting mouth opening and closing and corner pulling, and an expression numerical matrix is ​​constructed.

[0037] In one implementation, based on the horizontal and vertical pixel differences in the offset, and combined with a preset camera intrinsic parameter matrix, the horizontal and vertical angular offsets in three-dimensional space corresponding to the two-dimensional pixel offset are calculated. The camera intrinsic parameter matrix is ​​a parameter matrix obtained in advance through a calibration process after the remote classroom video acquisition equipment is installed. The calibration process can employ the Zhang Zhengyou calibration method, placing the checkerboard calibration board within the field of view of the video acquisition equipment and acquiring multiple images of the calibration board in different poses; the pixel coordinates of the checkerboard corner points in each image are extracted. The three-dimensional coordinates of each corner point on the checkerboard calibration board in the world coordinate system are known. For example, the calibration board plane is set as the Z=0 plane, and each corner point has known X and Y coordinates. A mapping relationship is established between the world coordinates (X, Y, 0) of the corner point and its corresponding pixel coordinates (u, v) in the image, and a 3×3 matrix describing the mapping relationship from the calibration board plane to the image plane is solved. By simultaneously establishing the homography matrices corresponding to multiple calibration board images and utilizing the orthogonality constraint of the rotation matrix, the camera intrinsic parameter matrix is ​​solved. The camera intrinsic parameter matrix includes the horizontal focal length, vertical focal length, and the principal coordinates of the image plane center point of the video acquisition device.

[0038] It should be noted that, firstly, the spatial coordinates of key facial points such as the tip of the nose, the left and right corners of the eyes, and the left and right corners of the mouth are extracted from the set of facial feature points. Based on the correspondence between the spatial coordinates of these key facial points and the 3D coordinates of corresponding points in the standard 3D face model, the rotation matrix of the far-end student's head relative to the camera coordinate system in the current frame is calculated. The head yaw angle and head pitch angle are then decomposed from the rotation matrix. For example, the standard 3D face model can be a general 3D face averaging model, which records the 3D coordinate values ​​of key facial points such as the tip of the nose, the left and right corners of the eyes, and the left and right corners of the mouth under a standard frontal face pose. After obtaining the head yaw angle and head pitch angle, the pixel offset corresponding to the head pose is subtracted from the horizontal and vertical pixel differences in the offset, respectively, to obtain the horizontal and vertical pixel differences after head pose compensation. Then, the compensated horizontal and vertical pixel differences are divided by the horizontal and vertical focal lengths, respectively, to obtain the normalized horizontal offset ratio and the normalized vertical offset ratio. The arctangent function is then taken for each ratio to obtain the horizontal and vertical angular offsets. The horizontal angular offset is used as the yaw angle, and the vertical angular offset is used as the pitch angle. The yaw angle and the pitch angle are combined to form the line-of-sight vector.

[0039] In one implementation, each displacement quantity in the deformation degree is mapped to a preset matrix dimension according to its corresponding coordinate point position to construct a deformation feature matrix. The preset matrix dimension is determined based on the number of feature points on the mouth edge, such that the product of the number of rows and columns is greater than or equal to the number of feature points, and the number of rows does not exceed the number of columns, with a preference for combinations with similar row and column numbers. Then, the displacement quantities corresponding to each feature point are sequentially filled into the matrix row by row. If there is no corresponding displacement quantity in the remaining position of the matrix, a zero value is filled in. For example, when the number of mouth edge feature points is 24, the matrix dimension can be set to 4 rows and 6 columns, and the 24 displacement quantities are filled into the matrix row by row according to the coordinate point arrangement order to construct the deformation feature matrix.

[0040] In one implementation, the deformation feature matrix undergoes orthogonal transformation and dimensionality reduction to extract feature vectors reflecting mouth opening and closing and corner pulling, thus constructing an expression numerical matrix. The orthogonal transformation and dimensionality reduction remove redundant information from the original displacement matrix, preserving the main patterns of mouth deformation. Specifically, the deformation feature matrix undergoes singular value decomposition to obtain the product of a left singular vector matrix, a singular value diagonal matrix, and a right singular vector matrix. Calculate the cumulative contribution rate of each singular value, i.e., the proportion of the sum of the first n singular values ​​to the sum of all singular values. Select the smallest n value that makes the cumulative contribution rate reach a preset threshold as the number of dimensions after dimensionality reduction. Here, n represents the number of dimensions after dimensionality reduction. Singular values ​​are obtained by singular value decomposition and arranged in descending order. The preset threshold is obtained by collecting sample data containing multiple mouth deformation patterns, calculating the cumulative contribution rate curve of each principal component, and selecting the cumulative contribution rate value that can retain the main deformation information in the sample data while effectively reducing the feature dimension. This value can be fine-tuned by the system administrator after verifying the accuracy of the classification model in actual application. For example, the preset threshold can be set to 90%. Take the left and right singular vectors corresponding to the first n largest singular values, and calculate the feature matrix after dimensionality reduction. The value of each element in the feature matrix is ​​the projection coefficient of the displacement of the corresponding coordinate point in the first n main deformation directions. The feature matrix after dimensionality reduction is used as the expression value matrix.

[0041] In step S3, the gaze direction vector and the facial expression numerical matrix are time-series aligned to obtain a synchronized feature sequence, which is then interpolated and padded to obtain a complete feature sequence, including: Extract the first timestamp of the gaze direction vector and the second timestamp of the facial expression value matrix in the current frame, and calculate the time deviation between the first timestamp and the second timestamp; If the time deviation is greater than the preset alignment threshold, the values ​​of the gaze direction vector and the values ​​of the facial expression value matrix are paired in chronological order to obtain a synchronized feature sequence after time series alignment. If there are missing timestamps in the synchronization feature sequence, then by constructing a piecewise polynomial function, the gaze and expression values ​​of the missing timestamps are calculated and filled to obtain the complete feature sequence after interpolation and filling.

[0042] It should be noted that the acquisition time of the gaze direction vector and the acquisition time of the facial expression value matrix may differ due to sampling frequency differences or network transmission delays. The first timestamp is the acquisition time corresponding to the gaze direction vector, and the second timestamp is the acquisition time corresponding to the facial expression value matrix. When the time difference between the first and second timestamps is greater than a preset alignment threshold, it indicates that there is a misalignment between the two on the time axis, and a pairing operation is needed to match the same time or the time difference with the smallest one. The preset alignment threshold can be set according to the system sampling frequency. For example, when the sampling frequency is 20 frames per second, the alignment threshold can be set to the time interval between two adjacent frames, i.e., 50 milliseconds.

[0043] Specifically, the pairing operation is achieved by constructing a Euclidean distance matrix between the two sequences. Each time point in the gaze direction vector sequence is paired with each time point in the expression value matrix sequence. The Euclidean distance between the gaze direction vector and the expression value matrix at the same time in each pair is calculated. All the Euclidean distances of the pairs are arranged according to their corresponding time positions to form a distance matrix. In this distance matrix, a curved path with the minimum cumulative distance is searched from the starting point to the ending point. Each matrix position traversed by this path represents a pair of matching correspondences. The pair with the minimum cumulative distance in the path is taken as the optimal alignment mapping between the two sequences. According to this optimal alignment mapping, each time point in the gaze direction vector sequence is bound to the corresponding time point in the expression value matrix sequence, so that each data point in the paired synchronous feature sequence simultaneously contains both the gaze direction vector value and the expression value matrix value, thereby achieving alignment between the two in the time dimension.

[0044] In one implementation, video transmission may experience network jitter or data packet loss, leading to missing values ​​in the gaze direction vector or facial expression matrix for some timestamps in the synchronization feature sequence. This embodiment employs a piecewise polynomial function for fitting and filling. A preset number of valid data frames are selected forward and backward from the missing timestamp. The timestamps of these valid data frames are used as nodes, and the intervals between adjacent nodes constitute the segmented intervals. The preset number consists of 5 valid data frames before and after the missing timestamp, totaling 10 reference nodes. This is sufficient to construct a cubic spline interpolation function covering two segmented intervals before and after the missing interval. This ensures interpolation accuracy while avoiding excessive computational complexity due to too many reference nodes. The preset number can be increased as the actual data sampling frequency increases; for example, the preset number can be set to 5. For each segmented interval, a cubic polynomial function is constructed, with the timestamp as the independent variable and the gaze direction vector or facial expression matrix as the dependent variable. The polynomials in adjacent segmented intervals satisfy the continuity of function values, first derivatives, and second derivatives at shared nodes, ensuring a smooth connection of cubic polynomials across each segmented interval at the nodes, forming a continuously fitting curve. The missing timestamp is substituted into the cubic polynomial function of the segmented interval containing that missing timestamp to calculate the fitted value for that timestamp, which is then used as the imputation value for the missing data.

[0045] It is worth noting that if the number of valid data frames before the missing timestamp is less than a preset number but the number after it is sufficient, then a preset number of valid data frames after the missing timestamp are selected as reference nodes. A cubic polynomial function starting with the missing timestamp and ending with the valid data frames after it is constructed. This cubic polynomial function is then extended towards the missing timestamp, and the missing timestamp is substituted into the cubic polynomial function to calculate the fitted value. If the number of valid data frames after the missing timestamp is less than a preset number but the number before it is sufficient, then a preset number of valid data frames before the missing timestamp are selected as reference nodes. A cubic polynomial function starting with the valid data frames before it and ending with the missing timestamp is constructed. Similarly, extrapolation is used for filling. If the number of valid data frames before and after the missing timestamp is less than a preset number, a low-order polynomial function is constructed based on the actual number of valid data frames to fit and fill the missing timestamp. For example, if the total number of valid data frames before and after the missing timestamp is not less than 3, a quadratic polynomial function can be constructed for fitting. If the total number of valid data frames is less than 3, linear interpolation is used for filling, or the value of the nearest valid data frame is directly selected as the filling value. After performing the piecewise polynomial function construction and imputation operations on all missing timestamps sequentially, a complete feature sequence is obtained. Each timestamp in the complete feature sequence contains complete values ​​for the gaze direction vector and the facial expression matrix.

[0046] In one implementation, extracting the temporal dependency between gaze deviation and facial expression changes in the complete feature sequence to obtain a numerical behavior sequence includes: Extract the yaw and pitch angles of the line-of-sight vector at each moment in the complete feature sequence, as well as each element in the facial expression numerical matrix at each moment; The yaw angle and the pitch angle are calculated by comparing them with the yaw angle and pitch angle at the previous moment to obtain the changes in yaw angle and pitch angle at each moment. The difference between each element in the expression value matrix and the corresponding element in the expression value matrix at the previous moment is calculated to obtain the expression change matrix at each moment. The matrices of yaw angle change, pitch angle change, and facial expression change are arranged in chronological order to form a numerical behavior sequence.

[0047] It is worth noting that the change in yaw angle reflects the rate of deflection of the line of sight in the horizontal direction, the change in pitch angle reflects the rate of deflection of the line of sight in the vertical direction, and the values ​​of each element in the facial expression change matrix reflect the rate of change of each deformation pattern of the mouth over time. All of the above changes are calculated by subtracting the value from the value at the current moment from the value at the previous moment. When the time intervals between consecutive moments are equal, the result of the difference calculation is equivalent to the rate of change.

[0048] Specifically, for the yaw angle in the line-of-sight vector, the yaw angle value at the current moment is subtracted from the yaw angle value at the previous moment to obtain the change in yaw angle at the current moment; for the pitch angle in the line-of-sight vector, the pitch angle value at the current moment is subtracted from the pitch angle value at the previous moment to obtain the change in pitch angle at the current moment; for each element in the expression value matrix, each element in the expression value matrix at the current moment is subtracted from the corresponding element in the expression value matrix at the previous moment to obtain the expression change matrix at the current moment. For the first moment in the complete feature sequence, there is no previous moment, so the yaw angle change, pitch angle change, and expression change matrices for the first moment are all set to zero. By sequentially traversing each moment in the complete feature sequence, the changes corresponding to each moment are obtained, and arranged in chronological order to form a numerical behavior sequence. Each data point in the numerical behavior sequence is composed of matrices of yaw angle change, pitch angle change, and facial expression change, which comprehensively reflects the dynamic coupling relationship between gaze deflection and facial expression change, that is, whether the student's gaze deflection at a certain moment is accompanied by facial expression change, and whether the directions of the two changes are consistent.

[0049] In step S4, the numerical behavior sequence is input into a pre-trained classification model to obtain the probability value of the mapping relationship between the numerical behavior sequence and the preset cognitive state label. Based on the probability value of the mapping relationship, a threshold judgment is made to determine the learning state category of the remote student, including: The numerical behavior sequence is input into the pre-trained classification model, and the initial score values ​​of the numerical behavior sequence and each preset cognitive state label are calculated respectively. All the initial score values ​​are normalized, and each normalized initial score value is used as a probability value of the mapping relationship between the numerical behavior sequence and each preset cognitive state label. Extract the probability term with the largest value and its corresponding cognitive state label from the probability values ​​of the mapping relationship; If the probability item with the largest value is greater than a preset probability threshold, then the cognitive state label corresponding to the probability item with the largest value is determined as the learning state category of the remote student. If the probability term with the largest value is less than or equal to the probability threshold, the current recognition result is determined to be invalid, and the learning status category of the remote student is maintained as the judgment result of the previous moment.

[0050] In one implementation, the numerical behavior sequence is input into the pre-trained classification model, and initial scores for the numerical behavior sequence and each preset cognitive state label are calculated. The pre-trained classification model refers to a classifier obtained by pre-training with numerical behavior sequence samples labeled with cognitive state labels. Its input is the numerical behavior sequence, and its output is the initial score of the sequence belonging to each preset cognitive state label. For example, the numerical behavior sequence samples can be numerical behavior sequences containing student gaze deviation and facial expression changes obtained in the early stages, and the classifier can employ a one-dimensional convolutional neural network algorithm. The preset cognitive state labels include multiple pre-set learning state categories. For example, the preset cognitive state labels may include categories such as focus, fatigue, confusion, and distraction. Specific categories can be set and adjusted according to the needs of the actual teaching scenario. The labels are determined based on the basic classification of classroom learning states in the field of educational psychology. The classification criteria for each label refer to the descriptions of student classroom attention and cognitive participation in relevant educational psychology textbooks. Teaching administrators can add, delete, or merge and adjust the label set according to the actual grade level and subject characteristics of the remote classroom. The initial score reflects the degree of matching between the numerical behavior sequence and each preset cognitive state label. The higher the score, the more likely the sequence is to belong to the corresponding state category.

[0051] It should be noted that the structure of the one-dimensional convolutional neural network, from input to output, includes an input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, and a fully connected output layer. The input layer receives a numerical behavior sequence. Each data point in the sequence is composed of elements from matrices representing changes in yaw angle, pitch angle, and facial expression. For example, when the facial expression matrix is ​​reduced to 'a' dimensions, the dimension of each data point is 2+a. The input sequence is a two-dimensional data structure (time series length, 2+a), where the time series length corresponds to the number of sampling frames, and 2+a corresponds to the feature dimension at each time step. Since the convolution operation only slides along the time series length, transforming the 2+a-dimensional feature vector at each time step as a whole, a one-dimensional convolutional kernel is used to match the input data. The first and second convolutional layers each use a one-dimensional convolutional kernel of a preset size to perform sliding window scanning and local feature extraction on the input sequence data. The size and number of convolutional kernels are preset according to the dimension and complexity of the input data. For example, the convolutional kernel size can be set to 3, and the stride can be set to 1. The first and second pooling layers are used to downsample the features output by the convolutional layers, reducing the feature dimensionality and retaining the main feature information. For example, the pooling layers can use max pooling, and the pooling window size can be set to 2. The fully connected output layer is used to globally integrate and fuse all the local features output by the pooling layers, outputting the initial score value between the numerical behavior sequence and each preset cognitive state label. For example, the output dimension of the fully connected layer can be set to the number of categories of the preset cognitive state labels.

[0052] In one implementation, the training process of the pre-trained classification model first involves collecting a large number of classroom videos of remote students in different learning states, and extracting the corresponding numerical behavior sequences as samples according to steps S1 to S3. Multiple education experts independently evaluate the video segments corresponding to the samples, and the cognitive state labels are determined by combining the majority opinions. The labeled samples are then divided into training, validation, and test sets in a 7:1.5:1.5 ratio. The samples in the training set are input into the one-dimensional convolutional neural network, and the predicted probability distribution for each sample is obtained through forward propagation. The predicted probability distribution refers to the probability value vector corresponding to each preset cognitive state label output by the model. The cross-entropy loss function is used to calculate the loss value between the predicted probability distribution and the true label. The cross-entropy loss function measures the difference between the model's prediction result and the true label; the smaller the loss value, the more accurate the prediction. The network parameters are updated using the Adam optimizer, an adaptive learning rate optimization algorithm. The Adam optimizer independently adjusts the learning rate for each parameter by calculating the first and second moment estimates of the gradient. For example, the initial learning rate can be set to 0.001, the exponential decay rate of the first moment estimate can be set to 0.9, and the exponential decay rate of the second moment estimate can be set to 0.999. After each training epoch, the current model is evaluated using a validation set. Training stops when the classification accuracy on the validation set no longer improves after five consecutive training epochs, and the current model parameters are saved as the pre-trained classification model. After training, the pre-trained classification model can be directly used to calculate the initial score between the input numerical behavior sequence and each preset cognitive state label.

[0053] In one implementation, all initial score values ​​are normalized, and the normalized initial score values ​​are used as probability values ​​for the mapping relationship between the numerical behavior sequence and each preset cognitive state label. The normalization process scales each initial score value proportionally to a range where the sum is 1, converting each score value into a probability distribution. Specifically, a normalization exponential function, such as the Softmax function, is used to process the global feature tensor output by the fully connected layer. The normalization exponential function converts each initial score value into a probability value between 0 and 1, and the sum of all probability values ​​is 1. After normalization, each mapping relationship probability value is between 0 and 1 and sums to 1. Each mapping relationship probability value represents the probability that the numerical behavior sequence belongs to the corresponding cognitive state label.

[0054] It is worth noting that the probability term with the largest value and its corresponding cognitive state label are extracted from the probability values ​​of the mapping relationship. Specifically, each preset cognitive state label corresponding to the probability value of the mapping relationship is traversed, and the probability value corresponding to each label is compared one by one to find the probability term with the largest value and its corresponding cognitive state label. If the probability term with the largest value is greater than a preset probability threshold, then the cognitive state label corresponding to the probability term with the largest value is determined as the learning state category of the remote student. The preset probability threshold is a pre-set probability judgment threshold. When the maximum probability value is greater than the threshold, it indicates that the recognition result of the classification model has sufficient credibility, and the cognitive state label corresponding to the maximum probability value can be output as the learning state category of the remote student at the current moment. If the probability term with the largest value is less than or equal to the probability threshold, then the current recognition result is determined to be invalid, and the learning state category of the remote student is kept as the judgment result of the previous moment.

[0055] It should be noted that if the current frame is the first frame of the continuous video, there is no judgment result from the previous moment, and the learning state category judgment result can be temporarily withheld, waiting for the recognition result of the next frame. The preset probability threshold can be determined based on the confidence distribution of the model on the validation set. For example, the highest probability value of all correctly classified samples in the validation set is statistically analyzed, the mean of these highest probability values ​​is calculated, and the mean is multiplied by a preset coefficient to obtain the preset probability threshold. For example, the preset coefficient can be set to a value between 0.85 and 0.95, such as 0.9. For example, in this embodiment, the preset probability threshold can be set to 0.75.

[0056] In step S5, if the learning state category belongs to a preset negative state set, then a pre-established intervention rule base is retrieved according to the learning state category to generate teaching control instructions for the remote student, including: If the learning state category is in the negative state set, then the current continuous frame sequence is extracted from the continuous video of the remote student, and the proportion of frames in the continuous frame sequence whose facial expression changes exceed a preset range is calculated. The proportion of frames is used as the negative degree value. Using the learning state category as the search keyword, the regulatory instruction template associated with the learning state category is searched in the pre-established intervention rule base; The negative degree value is filled into the corresponding parameter position in the control instruction template to generate a teaching control instruction for the remote student.

[0057] In one implementation, if the learning state category is in the negative state set, then the current continuous frame sequence is extracted from the continuous video of the remote student, and the percentage of frames in the continuous frame sequence whose facial expression changes exceed a preset range is calculated. This percentage is then used as the negative degree value. The negative state set is a pre-constructed set containing negative learning state categories that require intervention. For example, the negative state set may include categories such as inattention and emotional agitation.

[0058] Specifically, the facial expression change amplitude is obtained by extracting the set of facial feature points from each frame in the continuous frame sequence, and separating the second spatial coordinates of the mouth edge from the set of facial feature points in each frame. For each frame in the continuous frame sequence except the first frame, the difference between the second spatial coordinates of the current frame and the second spatial coordinates of the previous frame is calculated to obtain the inter-frame displacement component of each coordinate point. The Euclidean distance of the inter-frame displacement component is calculated and averaged to obtain the facial expression change amplitude of the frame, which is different from the degree of deformation of the mouth edge calculated from the current frame to the starting frame. For example, the feature points involved in the calculation are all 24 mouth edge feature points in the second spatial coordinates, and the aggregation method is to average the Euclidean distance of the inter-frame displacement of each feature point. When the facial expression change amplitude of a certain frame exceeds a preset amplitude, the frame is marked as a changed frame. The proportion of the number of changed frames in the continuous frame sequence to the total number of frames in the sequence is counted, and this proportion is multiplied by 100 to obtain the negative degree value. The negative degree value is used to quantify the severity of the current negative learning state, and the value range is 0 to 100.

[0059] For example, the preset amplitude is calculated by sampling the video of the remote classroom under normal teaching conditions, calculating the amplitude of facial expression changes in each frame according to the above method, calculating the mean and standard deviation of the amplitude values ​​of all frames, and using the sum of the mean and twice the standard deviation as the preset amplitude. For example, when the mean amplitude value is 3 pixels and the standard deviation is 1 pixel, the preset amplitude is 5 pixels. This value can be adjusted according to the actual lighting conditions of different remote classrooms and the parameters of the video acquisition equipment.

[0060] In one implementation, the learning state category is used as the search keyword to search for a control instruction template associated with the learning state category in the pre-established intervention rule base. The negative degree value is then filled into the corresponding parameter position in the control instruction template to generate a teaching control instruction for the remote student. The intervention rule base is a pre-established rule base that stores the mapping relationship between different learning state categories and control strategies. In the intervention rule base, each learning state category corresponds to one or more control instruction templates, and each control instruction template has variable placeholders for filling in specific parameters. Specifically, the intervention rule base is pre-configured by teaching administrators according to teaching intervention strategies during the system deployment phase. For each preset cognitive state label, the teaching administrators set at least one corresponding control instruction template. Each control instruction template includes a guiding framework and at least one variable placeholder. When the same learning state category corresponds to multiple control instruction templates, the templates are arranged from high to low priority, and the system selects the template with the highest priority when generating control instructions. The currently determined learning state category is used as the search keyword to perform a matching search in the intervention rule base to extract the control instruction template associated with that learning state category. The control instruction template includes a preset guidance framework and parameter placeholders. After mapping the negative degree value to the corresponding guidance intensity parameter or duration parameter, the placeholders are filled in to generate a complete teaching control instruction.

[0061] For example, when the learning state category is emotional agitation, the control instruction template can be "Please follow the prompts to perform X minutes of deep breathing relaxation." The negative degree value is mapped to the corresponding relaxation duration and then filled into the X position, generating a customized teaching control instruction for the remote student. The mapping relationship between the negative degree value and the deep breathing duration is established by inviting several remote students to participate in a pre-experiment during the system deployment phase. Facial videos of the students in a natural state, a mildly fatigued state, and a significantly fatigued state are collected. The negative degree value for each state is calculated according to steps S1 to S5, and the self-reported relaxation needs of the students in each state are recorded, establishing a correspondence between the negative degree value and the required duration. The negative degree value is divided into three intervals: less than or equal to the mean minus the standard deviation, between the mean plus and minus the standard deviation, and above the mean plus the standard deviation, corresponding to 1 minute of short relaxation, 2 minutes of medium relaxation, and 3 minutes of long relaxation, respectively. For example, when the mean of the negative values ​​in the experimental data is 50 and the standard deviation is 20, the three intervals are 0 to 30, 31 to 70, and 71 to 100, respectively, which are mapped to 1 minute, 2 minutes, and 3 minutes. The above mapping relationship can be adjusted by teaching administrators based on actual classroom feedback.

[0062] It should be noted that both the negative state set and the intervention rule base are pre-established during the system deployment phase according to teaching management needs. The negative state set defines the range of negative learning state categories that require triggering teaching intervention. The intervention rule base stores control strategies corresponding to various negative learning state categories. Different severity levels correspond to different control contents. For example, mild negative states can correspond to gentle reminder-type control instructions, while severe negative states can correspond to strongly guiding control instructions. The variable placeholders in the control instruction template are used to fill in parameter values ​​calculated based on real-time status when generating specific instructions, so that the generated control instructions can adapt to the actual severity of the student's current state.

[0063] In one implementation, the step of sending the instruction to the display terminal in the main lecture hall includes: acquiring the teaching control instruction; using the urgency level indicator of the teaching control instruction as a weight parameter; constructing an instruction transmission queue; inserting the teaching control instruction at the head of the instruction transmission queue; and outputting the instruction to be sent. The instruction to be sent is then pushed to the display terminal in the main lecture hall via a pre-established full-duplex communication channel. The full-duplex communication channel is established based on the WebSocket protocol, and the instruction to be sent is encapsulated in JSON format. The JSON data structure includes the student's ID, learning status category, negative score, suggested intervention action, and urgency level indicator.

[0064] In summary, this invention discloses a collaborative teaching method for remote classrooms based on video interaction. The method includes acquiring continuous video from a remote classroom and analyzing it frame by frame to extract a set of facial feature points for the remote student; calculating the offset of the pupil center relative to the eye contour and the degree of deformation of the mouth edge based on the set of facial feature points to construct a gaze direction vector and an expression numerical matrix; sequentially aligning, interpolating, and extracting dependencies from the gaze direction vector and expression numerical matrix to obtain a numerical behavior sequence; inputting this sequence into a pre-trained classification model to obtain mapping probability values, and determining the learning state category of the remote student through threshold judgment; if the student belongs to a preset negative state set, generating a teaching control instruction and sending it to the main classroom display terminal. This invention achieves accurate perception and learning status recognition of remote students' nonverbal behavior by integrating the timing of gaze direction and facial expression changes. When a negative state is identified, it automatically generates intervention instructions and pushes them to the main teacher. This allows the main teacher to be aware of abnormal student states and intervene in a timely manner without interrupting the teaching rhythm. It realizes a real-time intervention closed loop for collaborative teaching in multiple classrooms in remote classrooms, effectively improving the teaching quality and interactive timeliness of remote classrooms.

[0065] The second embodiment of the present invention provides a video-interactive remote delivery classroom multi-classroom collaborative teaching system, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method described above.

[0066] It should be noted that the video-interactive remote delivery classroom multi-classroom collaborative teaching system provided in this embodiment of the invention is used to execute all the process steps of the video-interactive remote delivery classroom multi-classroom collaborative teaching method in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0067] It should be noted that the system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0068] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A remote collaborative teaching method for multiple classrooms based on video interaction, characterized in that, include: Acquire continuous video from a remote classroom, analyze the continuous video frame by frame, and extract the set of facial feature points of the remote students in the continuous video; Spatial coordinate points are extracted from the set of facial feature points. The offset of the pupil center relative to the eye contour and the degree of deformation of the mouth edge are determined. The offset is mapped by coordinates to construct a gaze direction vector. The degree of deformation is processed by matrix transformation to obtain the expression value matrix. The gaze direction vector and the facial expression numerical matrix are aligned in time series to obtain a synchronous feature sequence, which is then interpolated and filled to obtain a complete feature sequence. The temporal dependency between gaze deflection and facial expression change in the complete feature sequence is extracted to obtain a numerical behavior sequence. The numerical behavior sequence is input into a pre-trained classification model to obtain the mapping relationship probability value between the numerical behavior sequence and the preset cognitive state label. Based on the mapping relationship probability value, the learning state category of the remote student is determined by threshold judgment. If the learning state category belongs to a preset negative state set, then the pre-established intervention rule base is retrieved according to the learning state category, teaching control instructions for the remote student are generated, and sent to the display terminal of the main classroom.

2. The method for collaborative teaching in multiple classrooms via video-interactive remote delivery classrooms according to claim 1, characterized in that, The step of parsing the continuous video frame by frame to extract the set of facial feature points of the far-end student in the continuous video includes: The continuous video is decoded into color image frames, the color image frames are converted to grayscale, and a grayscale image sequence is output. Histogram equalization is performed on each frame of grayscale image in the grayscale image sequence to obtain an equalized image sequence; Calculate the grayscale integral projection curves in the horizontal and vertical directions based on the equalized image sequence; The eye region and mouth region are located based on the grayscale integral projection curve, and edge feature points are extracted from the eye region and mouth region to obtain a set of facial feature points.

3. The method for collaborative teaching across multiple classrooms in a remote delivery classroom based on video interaction as described in claim 1, characterized in that, The step of extracting spatial coordinate points from the set of facial feature points to determine the offset of the pupil center relative to the eye contour and the degree of deformation of the mouth edge includes: Extract the first spatial coordinates of the eye contour and the second spatial coordinates of the mouth edge from the set of facial feature points; Locate the pupil edge points from the first spatial coordinates, calculate the geometric center of the pupil region formed by the pupil edge points, obtain the pupil center coordinates, calculate the geometric center of the region enclosed by the eye contour based on the first spatial coordinates, and obtain the geometric centroid of the eye contour. Calculate the horizontal and vertical pixel differences between the pupil center coordinates and the geometric centroid to obtain the offset of the pupil center relative to the eye contour; The position difference of the corresponding coordinate points is calculated by performing the second spatial coordinates corresponding to the current frame and the starting frame of the continuous video, and the displacement of each coordinate point is obtained. The displacements are arranged in order of coordinate points to form a displacement sequence, thereby obtaining the degree of deformation of the mouth edge.

4. The method for collaborative teaching in multiple classrooms via video-interactive remote delivery classrooms according to claim 1, characterized in that, The process of mapping the offset to a coordinate plane, constructing a gaze direction vector, and performing matrix transformation on the degree of deformation to obtain an expression numerical matrix includes: Based on the horizontal and vertical pixel differences in the offset, and combined with the preset camera intrinsic parameter matrix, the horizontal and vertical angular offsets in the three-dimensional space corresponding to the two-dimensional pixel offset are calculated. The horizontal angular offset is used as the yaw angle, and the vertical angular offset is used as the pitch angle. The yaw angle and the pitch angle are combined into a line-of-sight vector. Each displacement in the degree of deformation is mapped to a preset matrix dimension according to the corresponding coordinate point position to construct a deformation feature matrix; The deformation feature matrix is ​​subjected to orthogonal transformation and dimensionality reduction to extract feature vectors reflecting mouth opening and closing and corner pulling, and an expression numerical matrix is ​​constructed.

5. The method for collaborative teaching across multiple classrooms in a remote delivery classroom based on video interaction as described in claim 1, characterized in that, The step of aligning the gaze direction vector and the facial expression numerical matrix over time to obtain a synchronized feature sequence, and then interpolating and padding to obtain a complete feature sequence, includes: Extract the first timestamp of the gaze direction vector and the second timestamp of the facial expression value matrix in the current frame, and calculate the time deviation between the first timestamp and the second timestamp; If the time deviation is greater than the preset alignment threshold, the values ​​of the gaze direction vector and the values ​​of the facial expression value matrix are paired in chronological order to obtain a synchronized feature sequence after time series alignment. If there are missing timestamps in the synchronization feature sequence, then by constructing a piecewise polynomial function, the gaze and expression values ​​of the missing timestamps are calculated and filled to obtain the complete feature sequence after interpolation and filling.

6. The method for collaborative teaching across multiple classrooms in a remote delivery classroom based on video interaction according to claim 1, characterized in that, The step of extracting the temporal dependency between gaze deviation and facial expression changes in the complete feature sequence to obtain a numerical behavior sequence includes: Extract the yaw and pitch angles of the line-of-sight vector at each moment in the complete feature sequence, as well as each element in the facial expression numerical matrix at each moment; The yaw angle and the pitch angle are calculated by comparing them with the yaw angle and pitch angle at the previous moment to obtain the changes in yaw angle and pitch angle at each moment. The difference between each element in the expression value matrix and the corresponding element in the expression value matrix at the previous moment is calculated to obtain the expression change matrix at each moment. The matrices of yaw angle change, pitch angle change, and facial expression change are arranged in chronological order to form a numerical behavior sequence.

7. The method for collaborative teaching across multiple classrooms in a remote delivery classroom based on video interaction as described in claim 1, characterized in that, The step of inputting the numerical behavior sequence into a pre-trained classification model to obtain the probability value of the mapping relationship between the numerical behavior sequence and the preset cognitive state label includes: The numerical behavior sequence is input into the pre-trained classification model, and the initial score values ​​of the numerical behavior sequence and each preset cognitive state label are calculated respectively. All the initial scores are normalized, and the normalized initial scores are used as the probability values ​​of the mapping relationship between the numerical behavior sequence and each preset cognitive state label.

8. The method for collaborative teaching in multiple classrooms via video-interactive remote delivery classroom as described in claim 1, characterized in that, The step of determining the learning status category of the remote student based on the probability value of the mapping relationship and through threshold judgment includes: Extract the probability term with the largest value and its corresponding cognitive state label from the probability values ​​of the mapping relationship; If the probability item with the largest value is greater than a preset probability threshold, then the cognitive state label corresponding to the probability item with the largest value is determined as the learning state category of the remote student. If the probability term with the largest value is less than or equal to the probability threshold, the current recognition result is determined to be invalid, and the learning status category of the remote student is maintained as the judgment result of the previous moment.

9. The method for collaborative teaching across multiple classrooms in a remote delivery classroom based on video interaction according to claim 1, characterized in that, If the learning state category belongs to a preset negative state set, then a pre-established intervention rule base is retrieved based on the learning state category to generate teaching control instructions for the remote student, including: If the learning state category is in the negative state set, then the current continuous frame sequence is extracted from the continuous video of the remote student, and the proportion of frames in the continuous frame sequence whose facial expression changes exceed a preset range is calculated. The proportion of frames is used as the negative degree value. Using the learning state category as the search keyword, the regulatory instruction template associated with the learning state category is searched in the pre-established intervention rule base; The negative degree value is filled into the corresponding parameter position in the control instruction template to generate a teaching control instruction for the remote student.

10. A video-interactive remote classroom collaborative teaching system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 9.

Images