A deep learning-based classroom video character segmentation method

Abstract

The application provides a classroom video character segmentation method based on deep learning, and belongs to the technical field of artificial intelligence, and comprises the following steps: acquiring classroom teaching panoramic video stream data; performing student-level mask segmentation processing on the classroom teaching panoramic video stream data based on a preset multi-scale student segmentation network model, to obtain a student-level mask segmentation prediction result output by the multi-scale student segmentation network model; wherein the multi-scale student segmentation network model is obtained based on deep learning model training of a multi-scale student segmentation network based on spatial semantic guidance on classroom video training data. The technical scheme can enable low-vision users to accurately segment student characters in a classroom video.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method for character segmentation in classroom videos based on deep learning. Background Technology

[0002] With the rapid development of artificial intelligence technology in recent years, deep learning-based video processing technology has made it possible to automatically segment students in classroom videos.

[0003] In related technologies, traditional segmentation methods based on Convolutional Neural Networks (CNNs) and segmentation methods based on the Transformer sequence modeling architecture with self-attention mechanisms both have shortcomings and cannot achieve high-precision student and person segmentation for classroom videos. This is because CNN models are limited by their local receptive fields, making it difficult to effectively model long-distance dependencies between large areas in the image. Relying solely on local features is insufficient to overcome the challenges posed by blurry edges and varied shapes of students. While Transformers can compensate for this deficiency by modeling long-distance dependencies through global attention, they often neglect local details crucial to the segmentation task during computation. Furthermore, the secondary computational complexity introduced by global attention further limits its application in this task. Emerging sequence modeling architectures based on the Selective State Space Model (SSM), such as Mamba, possess efficient sequence modeling capabilities and achieve a good balance between efficiency and accuracy. However, their inherent fixed scanning order makes it difficult to adaptively associate distant but semantically relevant human body regions when dealing with complex classroom scenarios. At the same time, the long-range attenuation problem also causes deep semantic information to gradually weaken during transmission, which seriously affects the final segmentation accuracy.

[0004] In summary, existing segmentation methods cannot achieve high-precision student and person segmentation in classroom videos. Summary of the Invention

[0005] The main objective of this application is to propose a deep learning-based method for segmenting students in classroom videos.

[0006] To achieve the above objectives, a first aspect of this application proposes a deep learning-based method for character segmentation in classroom videos, the method comprising: Acquire panoramic video stream data of classroom teaching; The classroom teaching panoramic video stream data is processed by mask segmentation at the student level based on a preset multi-scale student segmentation network model, and the mask segmentation prediction results at the student level output by the multi-scale student segmentation network model are obtained. The multi-scale student segmentation network model is obtained by training a deep learning model of a multi-scale student segmentation network guided by spatial semantics on classroom video training data.

[0007] In some embodiments, the multi-scale student segmentation network includes: The vectorized convolution module is used to perform multi-layer convolution operations on the feature image to gradually compress the spatial resolution and extract low-level semantic features while preserving the local spatial structure; the feature image is a classroom video image in the classroom video training data. A convolutional neural network is used to progressively extract fine-grained information from low-level texture to mid-level semantics from the low-level semantic features to obtain shallow high-resolution features of the feature image. A sequence modeling architecture based on a selective state-space model is used to perform global modeling of the feature image based on spatial semantic guidance to obtain the deep semantic features of the feature image. A multi-scale convolutional decoder is used to fuse the shallow high-resolution features and the deep semantic features to obtain a segmentation prediction at the original resolution of the feature image.

[0008] In some embodiments, the sequence modeling architecture based on a selective state-space model includes: A spatial semantic-guided scanning order generation module is used to generate a dynamic scanning order of the feature image based on spatial semantic guidance. An enhanced visual selection scanning module is used to select two-dimensional features of the feature image and rearrange them into a one-dimensional feature sequence according to the dynamic scanning order. The one-dimensional feature sequence is used for long-range dependency modeling.

[0009] In some embodiments, the method further includes: Acquire sample data of panoramic video streams of classroom teaching; Student character annotation is performed on single-frame classroom teaching video image samples in the classroom teaching panoramic video stream sample data to obtain annotated classroom teaching panoramic video stream sample data; the classroom video training data includes the annotated classroom teaching panoramic video stream sample data.

[0010] In some embodiments, the process of performing student-level masking segmentation on the panoramic video stream data of classroom teaching based on a preset multi-scale student segmentation network model includes: The single-frame classroom teaching video image from the panoramic video stream data is input into a preset multi-scale student segmentation network model. Based on the multi-scale student segmentation network model, each pixel of the single-frame classroom teaching video image is classified into student and background categories. Connected component partitioning is performed on the pixels classified into the student category to obtain and output the mask segmentation prediction results at the student level.

[0011] In some embodiments, after obtaining the mask segmentation prediction results for the student level output by the multi-scale student segmentation network model, the method further includes: Connectivity analysis is performed on the mask segmentation prediction results to obtain analysis results; the analysis results include: filtering results after filtering out regions with areas smaller than a preset threshold in the mask segmentation prediction results; Based on the video temporal sequence of the panoramic video stream data of the classroom teaching, the filtering results are smoothed to obtain the video character segmentation results.

[0012] In some embodiments, acquiring panoramic video stream data of classroom teaching includes: Based on pre-set multi-angle camera equipment, raw classroom video stream data is collected synchronously in the classroom teaching environment; The original classroom video stream data is aligned to eliminate perspective differences, resulting in panoramic classroom teaching video stream data.

[0013] To achieve the above objectives, a second aspect of this application proposes a deep learning-based classroom video character segmentation device, the device comprising: The acquisition module is used to acquire panoramic video stream data of classroom teaching. The model segmentation module is used to perform student-level mask segmentation processing on the panoramic video stream data of the classroom teaching based on a preset multi-scale student segmentation network model, and to obtain the student-level mask segmentation prediction results output by the multi-scale student segmentation network model. The multi-scale student segmentation network model is obtained by training a deep learning model of a multi-scale student segmentation network guided by spatial semantics on classroom video training data.

[0014] To achieve the above objectives, a third aspect of the present application provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the second aspect above.

[0015] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the second aspect above.

[0016] To achieve the above objectives, a fifth aspect of this application provides a computer program product storing a computer program that, when executed by a processor, implements the method described in the second aspect.

[0017] This application proposes a method, apparatus, computer device, computer-readable storage medium, and computer program product for classroom video character segmentation based on deep learning. The method involves acquiring panoramic video stream data of classroom teaching; performing student-level mask segmentation processing on the panoramic video stream data based on a preset multi-scale student segmentation network model; and obtaining the student-level mask segmentation prediction results output by the multi-scale student segmentation network model. The multi-scale student segmentation network model is obtained by training a deep learning model on a spatial semantic-guided multi-scale student segmentation network using classroom video training data.

[0018] Compared to traditional segmentation methods, this application's embodiments train a deep learning model on a spatial semantic-guided multi-scale student segmentation network using classroom video training data. This results in a trained multi-scale student segmentation network model. Upon acquiring panoramic video stream data of classroom teaching, this model can be used to perform student-level mask segmentation, yielding the student-level mask segmentation prediction results output by the multi-scale student segmentation network model. In other words, this application's embodiments overcome the limitation of traditional segmentation methods in achieving high-precision student segmentation in classroom videos, proposing a method capable of accurately segmenting students from complex classroom teaching scenarios. Attached Figure Description

[0019] Figure 1 A flowchart illustrating the steps of a deep learning-based classroom video character segmentation method provided in some embodiments of this application; Figure 2 for Figure 1 A detailed flowchart of step S101; Figure 3 A flowchart illustrating the steps of the deep learning-based classroom video character segmentation method provided in this application in some other embodiments; Figure 4The diagram shows the structure of the multi-scale student segmentation network involved in some embodiments of the deep learning-based classroom video character segmentation method provided in this application. Figure 5 The diagram shows the structure of the convolutional neural network involved in some embodiments of the deep learning-based classroom video character segmentation method provided in this application. Figure 6 A schematic diagram of the scanning order generation module involved in some other embodiments of the deep learning-based classroom video character segmentation method provided in this application; Figure 7 A schematic diagram of the structure of the enhanced visual selection scanning module involved in some embodiments of the deep learning-based classroom video character segmentation method provided in this application. Figure 8 for Figure 1 A detailed flowchart of step S102; Figure 9 This is a flowchart illustrating the steps of an embodiment of this application in other embodiments; Figure 10 A schematic diagram of the core process of a complete embodiment of a deep learning-based classroom video character segmentation method provided in this application; Figure 11 A schematic diagram of a deep learning-based classroom video character segmentation device provided in an embodiment of this application; Figure 12 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0023] First, the overall concept of the embodiments of this application will be explained.

[0024] In smart education and classroom behavior analysis, accurate student segmentation in classroom videos is a crucial step in subsequent tasks such as behavior recognition, attention analysis, and interaction detection. High-quality student segmentation helps the system further analyze student posture, concentration levels, and interaction frequency, thus providing quantifiable support for teaching diagnosis and classroom feedback, and enabling personalized teaching services for students. With the rapid development of artificial intelligence technology in recent years, deep learning-based video processing technology has made automatic student segmentation in classroom videos a reality.

[0025] However, the high complexity of classroom video scenes makes automatic student segmentation in these videos still challenging. First, due to complex classroom lighting, desks and textbooks, and occlusion between students, student edges often appear blurred with low contrast, making differentiation difficult based on local information alone. Second, the significant scale difference between students in the front and back rows, coupled with the diverse postures of students in the classroom leading to substantial changes in their physical appearance, further complicates the automatic segmentation of students.

[0026] Faced with the aforementioned challenges, traditional segmentation methods based on Convolutional Neural Networks (CNNs) or Transformers have shortcomings and cannot achieve high-precision segmentation of people in classroom videos. CNN models are limited by their local receptive field, making it difficult to effectively model long-distance dependencies between large areas in the image. Relying solely on local features is insufficient to overcome the challenges posed by blurry edges and varied shapes of students. While Transformers can compensate for this deficiency by modeling long-distance dependencies through global attention, they often neglect crucial local details during computation, and the secondary complexity introduced by global attention further limits their application in this task. Emerging sequence modeling architectures based on selective state-space models, such as Mamba, possess efficient sequence modeling capabilities, achieving a good balance between efficiency and accuracy. However, their inherent fixed scanning order makes it difficult to adaptively associate distant but semantically relevant human body regions when dealing with complex classroom scenarios. Furthermore, the long-range attenuation problem causes deep semantic information to gradually weaken during transmission, severely affecting the final segmentation accuracy. In summary, existing segmentation methods cannot achieve high-precision student and person segmentation in classroom videos.

[0027] To address this, this application proposes a method, apparatus, computer device, computer-readable storage medium, and computer program product for classroom video character segmentation based on deep learning. This method acquires panoramic video stream data of classroom teaching; performs student-level mask segmentation processing on the panoramic video stream data based on a preset multi-scale student segmentation network model, obtaining the student-level mask segmentation prediction results output by the multi-scale student segmentation network model; wherein the multi-scale student segmentation network model is obtained by training a deep learning model on a spatial semantic-guided multi-scale student segmentation network using classroom video training data.

[0028] Compared to traditional segmentation methods, this application's embodiments train a deep learning model on a spatial semantic-guided multi-scale student segmentation network using classroom video training data. This results in a trained multi-scale student segmentation network model. Upon acquiring panoramic video stream data of classroom teaching, this model can be used to perform student-level mask segmentation, yielding the student-level mask segmentation prediction results output by the multi-scale student segmentation network model. In other words, this application's embodiments overcome the limitation of traditional segmentation methods in achieving high-precision student segmentation in classroom videos, proposing a method capable of accurately segmenting students from complex classroom teaching scenarios.

[0029] In some embodiments, this application proposes a student character segmentation algorithm for classroom videos based on Spatial-Semantic-Guided Scanning Mamba (SSGS-Mamba). This algorithm leverages a CNN-Mamba hybrid architecture to efficiently extract multi-scale information, addressing the challenges of significant scale differences and blurred boundaries in student character morphology. To alleviate the insufficient global perception capability caused by the fixed scanning order of the original Mamba architecture, this algorithm designs a spatially guided adaptive scanning order generation module. This module efficiently aggregates spatial information using an attention mechanism to generate the optimal scanning order for Mamba.

[0030] In some embodiments, this application also proposes a method for collecting and labeling a complete dataset of student character segmentation in teaching videos. A complete dataset of student character segmentation in teaching videos collected and labeled by this method can be used for training the aforementioned end-to-end model SSGS-Mamba.

[0031] Based on the overall concept of the embodiments of this application described above, specific embodiments of a method, apparatus, computer device, computer-readable storage medium, and computer program product for classroom video character segmentation based on deep learning, as provided in the embodiments of this application, are proposed. First, various specific embodiments of the deep learning-based classroom video character segmentation method in the embodiments of this application are described in detail.

[0032] It should be noted that the embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0033] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0034] Furthermore, in various specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Moreover, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. Additionally, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after explicitly obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of these embodiments acquired.

[0035] Furthermore, the deep learning-based classroom video character segmentation method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a terminal device in the field of computer vision and educational scene video analysis technology, such as related management devices, smartphones, tablets, laptops, desktop computers, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing a deep learning-based classroom video character segmentation method, but is not limited to the above forms.

[0036] Alternatively, the deep learning-based classroom video character segmentation method provided in this application can also be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0037] For ease of understanding and explanation, the following descriptions will use a deep learning-based classroom video character segmentation method provided in the embodiments of this application as an example for detailed explanation. The implementation of the deep learning-based classroom video character segmentation method provided in the embodiments of this application using any of the above-described themes can refer to the process described below for applying the deep learning-based classroom video character segmentation method to a terminal device.

[0038] Please refer to Figure 1 , Figure 1 This document illustrates the steps of a deep learning-based classroom video character segmentation method provided in some embodiments of this application. It should be understood that, although... Figure 1The figure shows the execution order of some method steps, but based on the different design needs of practical applications, the deep learning-based classroom video character segmentation method provided in this application embodiment can of course adopt a different execution order of method steps than that shown in the figure. That is, Figure 1 The order of the method steps shown does not constitute a limitation on the execution logic order of the deep learning-based classroom video character segmentation method provided in this application embodiment. Any other method based on... Figure 1 Reasonable changes to the order of steps shown should be included within the protection scope of the deep learning-based classroom video character segmentation method provided in this application embodiment.

[0039] like Figure 1 As shown, in some embodiments, the classroom video character segmentation method based on deep learning provided in this application may include, but is not limited to, steps S101 and S102.

[0040] Step S101: Obtain panoramic video stream data of classroom teaching; Step S102: Perform student-level mask segmentation processing on the panoramic video stream data of the classroom teaching based on the preset multi-scale student segmentation network model to obtain the student-level mask segmentation prediction results output by the multi-scale student segmentation network model; wherein, the multi-scale student segmentation network model is obtained by training a deep learning model of a multi-scale student segmentation network based on spatial semantic guidance on the classroom video training data.

[0041] The terminal device pre-trains a deep learning model of a multi-scale student segmentation network guided by spatial semantics on classroom video training data, resulting in a trained multi-scale student segmentation network model. Thus, when performing student segmentation on videos in the complex scenario of classroom teaching, the terminal device can first acquire panoramic video stream data of the classroom teaching session. Then, based on the pre-trained multi-scale student segmentation network model, it performs student-level mask segmentation processing on the panoramic video stream data, thereby obtaining the student-level mask segmentation prediction results output by the multi-scale student segmentation network model.

[0042] Compared to traditional segmentation methods, this application's embodiments train a deep learning model on a spatial semantic-guided multi-scale student segmentation network using classroom video training data. This results in a trained multi-scale student segmentation network model. Upon acquiring panoramic video stream data of classroom teaching, this model can be used to perform student-level mask segmentation, yielding the student-level mask segmentation prediction results output by the multi-scale student segmentation network model. In other words, this application's embodiments overcome the limitation of traditional segmentation methods in achieving high-precision student segmentation in classroom videos, proposing a method capable of accurately segmenting students from complex classroom teaching scenarios.

[0043] In some embodiments, the terminal device can synchronously acquire panoramic video streams of classroom teaching based on multi-angle high-definition camera equipment, thereby obtaining panoramic video stream data of classroom teaching to be segmented into students.

[0044] Based on this, please refer to Figure 2 , Figure 2 for Figure 1 A detailed flowchart of step S101.

[0045] like Figure 2 As shown, in some embodiments, step S101 above: acquiring panoramic video stream data of classroom teaching may include steps S201 and S202 as shown below.

[0046] Step S201: Based on the preset multi-angle camera equipment, synchronously collect the original classroom video stream data in the classroom teaching environment.

[0047] It should be noted that multi-angle camera equipment can be pre-deployed in a standardized classroom environment. This equipment can achieve millisecond-level time synchronization via GPS timing modules or the NTP protocol, ensuring precise alignment of the video stream's built-in timestamps. Furthermore, the process of simultaneously capturing panoramic video streams of classroom teaching using multi-angle high-definition cameras strictly adheres to privacy protection regulations, with non-teaching areas undergoing real-time mosaic processing.

[0048] When performing the task of segmenting students and figures in a video of a complex classroom teaching scenario, the terminal device can synchronously acquire panoramic video streams of classroom teaching at a fixed frame rate (e.g., 30fps) based on pre-deployed multi-angle high-definition camera equipment to obtain raw classroom video stream data.

[0049] In some embodiments, the terminal device can control the multi-angle camera device to set the video capture resolution to 1920×1080 (full HD) and cover a dynamic range of more than 80dB to adapt to changes in classroom lighting, while using H.264 encoding compression to reduce storage overhead.

[0050] In some embodiments, the terminal device can control the multi-angle camera device to bind classroom metadata (such as course number, class information, teaching time, etc.) through digital watermarking technology during the process of acquiring panoramic video streams of classroom teaching, and attach a synchronously generated timestamp during acquisition, so as to facilitate accurate positioning of video segments in subsequent work.

[0051] Step S202: Align the original classroom video stream data to eliminate perspective differences and obtain panoramic classroom teaching video stream data.

[0052] After acquiring the original classroom video stream data, the terminal device can perform alignment processing on the original classroom video stream data to eliminate the differences in perspectives from multiple cameras and obtain panoramic classroom teaching video stream data for student and human segmentation.

[0053] In some embodiments, the terminal device may acquire sample data of the panoramic video stream of classroom teaching in the same way as the above-described acquisition of panoramic video stream data of classroom teaching, and then construct classroom video training data based on the sample data of the panoramic video stream of classroom teaching to train a deep learning model of a multi-scale student segmentation network guided by spatial semantics, thereby obtaining the trained multi-scale student segmentation network model.

[0054] Based on this, please refer to Figure 3 , Figure 3 The flowchart of the deep learning-based classroom video character segmentation method provided in this application is illustrated in some other embodiments.

[0055] like Figure 3 As shown, in some embodiments, the deep learning-based classroom video character segmentation method provided in this application may further include steps S301 and S302 as shown below.

[0056] Step S301: Obtain sample data of the panoramic video stream of classroom teaching; Step S302: Perform student character annotation processing on the single-frame classroom teaching video image samples in the classroom teaching panoramic video stream sample data to obtain annotated classroom teaching panoramic video stream sample data; the classroom video training data includes the annotated classroom teaching panoramic video stream sample data.

[0057] When training a deep learning model for a multi-scale student segmentation network guided by spatial semantics using classroom video training data, the terminal device can simultaneously acquire panoramic video streams of classroom teaching at a fixed frame rate (e.g., 30fps) based on pre-deployed multi-angle high-definition camera equipment. Then, the acquired classroom video stream data is aligned to eliminate the differences in the perspectives of multiple cameras, thus obtaining sample data of the panoramic video stream of classroom teaching.

[0058] After acquiring the panoramic video stream sample data of classroom teaching, the terminal device processes the sequentially continuous video stream into single-frame slices according to the sampling rate (i.e., 30fps), obtaining single-frame classroom teaching video image samples. Furthermore, it performs student character annotation processing on these single-frame classroom teaching video image samples, obtaining annotated panoramic video stream sample data. In this way, the terminal device can use the annotated panoramic video stream sample data as training data for training a deep learning model on a multi-scale student segmentation network.

[0059] In some embodiments, when the terminal device performs student character annotation processing on a single-frame classroom teaching video image sample, it can perform pixel-by-pixel standardization on each student character in the single-frame classroom teaching video image sample. For example, the terminal device first predicts the student character segmentation mask using the open-source general video segmentation model SAM2 (Segment Anything Model2) to obtain coarse segmentation annotations, and then obtains the exact student character segmentation mask through manual calibration, thereby obtaining annotated classroom teaching panoramic video stream sample data for subsequent model training and evaluation.

[0060] In some embodiments, after constructing the classroom video training data, the terminal device can further preprocess the data to improve its quality and diversity, thereby enhancing the accuracy and generalization of the training model. For example, the terminal device can improve graph quality and alleviate model overfitting through histogram equalization and normalization, where the equalization and normalization formulas are as follows:

[0061] in, The input image is H=W=256. Indicates the pixel position of the image The grayscale value; L is the number of grayscale levels (e.g., L=256). Let k be the probability distribution of gray level k. This represents its cumulative distribution function (CDF). The normalization process linearly maps pixel values ​​to the [0,1] interval, where I min with Imax These represent the minimum and maximum pixel values ​​of the image, respectively. Additionally, random horizontal / vertical flipping is used to process the training set images to increase image diversity during the training phase, thereby improving the model's generalization performance on unknown data and accelerating the model's convergence speed.

[0062] In some embodiments, the multi-scale student segmentation network includes: The vectorized convolution module is used to perform multi-layer convolution operations on the feature image to gradually compress the spatial resolution and extract low-level semantic features while preserving the local spatial structure; the feature image is a classroom video image in the classroom video training data. A convolutional neural network is used to progressively extract fine-grained information from low-level texture to mid-level semantics from the low-level semantic features to obtain shallow high-resolution features of the feature image. A sequence modeling architecture based on a selective state-space model is used to perform global modeling of the feature image based on spatial semantic guidance to obtain the deep semantic features of the feature image. A multi-scale convolutional decoder is used to fuse the shallow high-resolution features and the deep semantic features to obtain a segmentation prediction at the original resolution of the feature image.

[0063] Terminal devices can preprocess classroom video training data to obtain preprocessed subject video images, and then design a multi-scale student segmentation network based on spatial semantic guidance.

[0064] Please refer to Figure 4 , Figure 4 The diagram shows the structure of the multi-scale student segmentation network involved in some embodiments of the deep learning-based classroom video character segmentation method provided in this application.

[0065] like Figure 4As shown, the multi-scale student segmentation network can be composed of an image vectorization stage corresponding to the vectorized convolutional module, a convolutional local feature encoding stage corresponding to the convolutional neural network, a Mamba global modeling stage corresponding to the sequence modeling architecture based on the selective state space model, and a multi-scale convolutional decoding stage corresponding to the multi-scale convolutional decoder. Specifically, the multi-scale student segmentation network includes a vectorized convolutional module, used to perform multi-layer convolutional operations on the feature image during the image vectorization stage, progressively compressing the spatial resolution and extracting low-level semantic features while maintaining the local spatial structure; the feature image is a classroom video image from the classroom video training data. The convolutional neural network is used to progressively extract fine-grained information from low-level texture to mid-level semantics during the convolutional local feature encoding stage, obtaining shallow high-resolution features of the feature image; the sequence modeling architecture based on the selective state space model is used to perform global modeling of the feature image based on spatial semantic guidance during the Mamba global modeling stage, obtaining deep semantic features of the feature image; and the multi-scale convolutional decoder is used to fuse the shallow high-resolution features and deep semantic features during the multi-scale convolutional decoding stage, obtaining a segmentation prediction at the original resolution of the feature image.

[0066] In some embodiments, the image vectorization stage involves directly inputting the preprocessed image into a vectorized convolutional module composed of convolutional layers (Conv) and depthwise separable convolutional layers (DSConv). Through multiple convolutional operations, the spatial resolution is gradually compressed and low-level semantic features are extracted while preserving the local spatial structure. The formula can be as follows: , Where X represents the input feature map with dimensions C0*H0*W0, and S represents the output feature map with dimensions C2*H2*W2. Conv(X) represents a regular convolution operation with a kernel of... The bias is b (1) The stride is s1. BN1 and BN2 correspond to the batch normalization operations of the first and second layers, respectively, and ReLU() is the element-wise activation function ReLU(x) = max(0,x). The depthwise convolution DW(Z) in the DSConv module performs convolution independently on each channel, and its convolution kernel is... Pointwise convolution Use 1x1 convolution kernel Map the number of channels to C2. k1 and k2 are the kernel sizes, s1 and s2 are the strides of the corresponding convolutional layers, and C0, C1, and C2 are the number of input, intermediate, and output channels, respectively.

[0067] Conventional convolutions capture basic contour, edge, and texture information within the original image space, while depthwise separable convolutions combine depthwise and pointwise convolutions to enhance the expressive power of feature channels at a lower computational cost. As convolutional layers are stacked, the spatial size of the feature map is continuously downsampled, reducing the output feature map to one-quarter the size of the original image in terms of spatial dimension, while simultaneously expanding the channel dimension to a specified number suitable for subsequent modules. This vectorized convolutional module achieves continuous vectorized representation of features through convolution operations without explicit patching, thereby generating initial feature maps that can be used for subsequent local feature extraction and global modeling based on the scan sequence.

[0068] In some embodiments, to extract fine-grained information for student task segmentation, the terminal device can, during the convolutional local feature encoding stage, feed the enhanced single-frame image of the classroom video, after vectorization, into a convolutional neural network to extract local features. The initial features are fed into MBConv (Mobile Inverted Bottleneck Convolution) to progressively extract fine-grained information from low-level texture to mid-level semantics, with the structure as follows: Figure 5 As shown.

[0069] The module works as follows: Primary convolutional layers capture pixel-level edge and texture features. Then, Depthwise Conv (element-wise depthwise convolution) is used to expand the receptive field at low cost and improve the ability to separate details within channels. Next, the dilatation and compression structure of MBConv is used to enhance the non-linear representation of local patterns. To ensure the boundary sensitivity of features during downsampling, the module's built-in bottleneck and residual connections maintain information flow and gradient stability. This ensures that the output of this stage retains sufficient local boundary details for accurate segmentation of the student's body contour while also transmitting information to the subsequent global modeling module in a compact channel dimension. The specific formula is shown below:

[0070] in, For the input feature map, This represents a convolution operation with channel expansion capability, used to expand the channel dimension from C to 4C; denoted as 3x3 depthwise convolution, which performs channel-wise spatial convolution while maintaining the same number of channels; U is the intermediate feature map after dilated convolution and depthwise convolution, with dimensions H*W*4C. The Squeeze-and-Excitation channel attention module is computed as follows:

[0071] Global average pooling The features are compressed into channel vectors, then passed through fully connected layer weights W1 and W2 for dimensionality reduction and dimensionality increase, and finally activated by the ReLU function. Channel attention weights are obtained using the sigmoid activation function. (Symbol) This represents a channel-wise element-wise multiplication with the input feature U, achieving channel-level adaptive reweighting. For projective convolution, the number of channels is compressed from 4C back to C. Finally, element-wise addition with the input yields the output feature map Y, which is the residual connection structure.

[0072] It should be noted that Mamba, as a state-space model, is generally considered a continuous linear time-invariant system, which uses state vectors... Input stimulus Mapped to output signal Where t, D, and N represent the time step, the number of signal channels, and the state dimension, respectively. This type of model can be formally represented by a linear ordinary differential equation (ODE) (e.g., Eq. ~\ref{eq:1}), whose parameters include... , , and jump connections .

[0073] ; .

[0074] To achieve computational efficiency, discretization is typically required to ensure the model maintains the same sampling rate as the signal in the input data, thus converting the ODE into a discrete function. The zero-order hold method is commonly used for ODE discretization, and by setting a specific sampling time... Convert continuous variables into their corresponding discrete parameters: , , , .

[0075] In practical applications, first-order Taylor series expansions are used to further improve the understanding of... Approximate accuracy: .

[0076] Unlike traditional methods that store information through finite states and linear time-invariant (LTI), Mamba introduces a dynamic mechanism for selectively filtering input to form sequential states. Specifically, to enhance context modeling capabilities, parameters... , and Directly from the input sequence , (in The results are obtained through linear projection. Subsequently, the output sequence can be calculated using these input-adaptive discretization parameters. : ; .

[0077] In some embodiments, the sequence modeling architecture based on a selective state-space model includes: A spatial semantic-guided scanning order generation module is used to generate a dynamic scanning order of the feature image based on spatial semantic guidance. An enhanced visual selection scanning module is used to select two-dimensional features of the feature image and rearrange them into a one-dimensional feature sequence according to the dynamic scanning order. The one-dimensional feature sequence is used for long-range dependency modeling.

[0078] During the Mamba global modeling phase, the terminal device first generates feature images based on a spatial semantic-guided scanning order generation module. Then, through an enhanced visual selection scanning module, the two-dimensional features of the feature images are rearranged into a one-dimensional feature sequence according to the dynamic scanning order. This one-dimensional feature sequence can then be used for long-range dependency modeling.

[0079] In some embodiments, although the Mamba architecture described above can efficiently extract global information, it can only process one-dimensional information and cannot directly process two-dimensional data such as images. Traditional Mamba-based algorithms typically flatten images into a one-dimensional sequence by manually designing a fixed scanning order before processing them using Mamba. However, this approach struggles to adaptively associate distant but semantically relevant human body regions when dealing with complex classroom scenarios; furthermore, the long-range attenuation problem causes deep semantic information to gradually weaken during transmission, severely impacting the final segmentation accuracy. Therefore, this embodiment designs a Spatial Semantic Guidance Scanning Block (SSGS Block) to generate a scanning order based on spatial language guidance.

[0080] Please refer to Figure 6 , Figure 6The diagram below shows the structure of the scanning order generation module in some embodiments of the deep learning-based classroom video character segmentation method provided in this application.

[0081] like Figure 6 As shown, specifically, given an input feature map The module first performs Global Average Pooling (GAP) and Global Max Pooling (GMP) in parallel to obtain two global channel descriptions:

[0082] The two are mapped via a shared multilayer perceptron (MLP) and activated by a sigmoid function to obtain initial spatial semantic weights. These weights are then used to weight and harmonicize the original input, aggregating global information, enhancing key channels, and suppressing irrelevant channels. .

[0083] Subsequently, the input features are simultaneously subjected to spatial average pooling (AvgPool) and spatial max pooling (MaxPool), and then concatenated along the channel dimension:

[0084] The concatenated result is passed through a convolutional layer and activated by a sigmoid function to generate a spatial attention map.

[0085] Finally, channel attention and spatial attention are combined to apply to the input features, resulting in a semantic saliency map that can be used to guide the dynamic scanning order:

[0086] in, This represents element-wise multiplication. The saliency map numerically characterizes the importance of each image patch relative to the target segmentation task. Based on this saliency map, a ranking function is used to generate a scan sequence, prioritizing image patches with high saliency scores and semantic relevance in the sequence. This transforms semantically related but spatially distant regions in two-dimensional space into a continuous one-dimensional sequence, thus achieving efficient dynamic scan order generation.

[0087] In some embodiments, after generating a dynamic scanning sequence, the terminal device further uses an Enhanced Visual Selective Scanning Block (EVSS Block) to rearrange the two-dimensional features into a one-dimensional token sequence using the dynamically generated scanning sequence. The sequence is then input into a Mamba (selective state space) module for long-range dependency modeling. The state space module uses a branching structure supported by linear layers, gated activation, and deep convolution or DW-Conv to perform temporal propagation and feature interaction of the sequence information, and combines multiplicative gating between branches to achieve channel filtering and information fusion. This combination allows the model to significantly enhance the interaction between semantically related but spatially dispersed regions while preserving local details, thereby improving the overall structural consistency and boundary integrity in complex scenarios (such as multiple people occluding each other or changing postures in a classroom). Specifically, as shown... Figure 7 As shown.

[0088] This module first processes the input feature sequence. (Where L is the sequence length and C is the number of channels) Perform layer normalization:

[0089] Subsequently, the main path performs feature transformation through linear layers, depthwise convolutions, and the SiLU activation function:

[0090] Next, the SSGS-SSM module uses the dynamic scan sequence S to guide state space modeling, thereby achieving feature enhancement for spatial semantic awareness:

[0091] Meanwhile, the bypass branch generates adaptive gating signals through a linear layer and SiLU:

[0092] The two are multiplicative-gated fusion m through element-wise multiplication, and then processed through a linear layer:

[0093] Finally, the output is obtained through residual connection:

[0094] This architecture effectively enhances the long-range interaction capabilities of semantically related regions (such as occluded human bodies) through state-space modeling guided by dynamic scanning order and branch gating mechanism, and significantly improves structural consistency and boundary segmentation accuracy in complex scenarios.

[0095] In some embodiments, the multi-scale convolutional decoder can be a lightweight multi-scale convolutional decoder. The deep semantic features obtained after global modeling and the shallow high-resolution features output by the previous convolutional module are aligned and fused by scale during the decoding stage to restore the original resolution segmentation prediction. Specifically, the decoder first aligns the spatial dimensions of the features from the shallow and deep layers through convolution and upsampling operations. Then, a lightweight multi-scale fusion strategy is used to perform channel-wise or pixel-wise weighted fusion of information at different scales. The fused features are then processed in one step by several MBConv units to alleviate the semantic differences between the encoder and decoder and enhance edge details. After completing multi-level upsampling and convolutional rectification, the decoder maps the features to a single-channel segmentation probability map or multi-channel category prediction through the convolutional layer at the end. Finally, the final segmentation map of the student region is obtained through Softmax, as shown in the following formula:

[0096] in, Let represent the i-th element in the input vector (i.e., the unnormalized logit of the model output), and K be the number of classes. To perform an exponential mapping on logit to ensure that the output is non-negative, This is a normalization factor that ensures the output satisfies probability constraints. The final result is... Let represent the predicted probability of class i, and satisfy . as well as .

[0097] The decoder is designed to minimize computational burden while ensuring high reconstruction quality, in order to adapt to the deployment requirements of real-time or near real-time classroom videos.

[0098] In some embodiments, when training a deep learning model of a spatial semantic-guided multi-scale student segmentation network, to achieve efficient model training, the terminal device can perform all training processes on a workstation equipped with an NVIDIA RTX A6000 GPU (48GB VRAM), using the Adam optimizer (momentum coefficient 0.9, weight decay 1e-4) and cosine annealing learning rate scheduling (initial learning rate 5e-4). The model training consists of 400 epochs (batch size 16), and early stopping (patience value 10 epochs) is employed to prevent overfitting. Input images are uniformly scaled to 256x256 pixels for efficient model training.

[0099] Please refer to Figure 8 , Figure 8 for Figure 1 A detailed flowchart of step S102.

[0100] like Figure 8As shown, in some embodiments, the step of “performing student-level mask segmentation processing on the panoramic video stream data of classroom teaching based on a preset multi-scale student segmentation network model” in step S102 above may include steps S801 and S802 as shown below.

[0101] Step S801: Input the single-frame classroom teaching video image from the panoramic video stream data into a preset multi-scale student segmentation network model; Step S802: Based on the multi-scale student segmentation network model, classify each pixel of the single-frame classroom teaching video image into student and background categories, and perform connected component partitioning on the pixels classified as student categories to obtain and output the mask segmentation prediction results at the student level.

[0102] When a terminal device uses a trained multi-scale student segmentation network model to perform student-level mask segmentation on panoramic classroom video stream data, it can use this model to perform frame-by-frame evaluation of the classroom video. Specifically, a single frame of the classroom video stream is input into the multi-scale student segmentation network model. Based on this model, each pixel of the input single frame is classified into two categories: "student" and "background," resulting in student and background categories. Subsequently, connected component partitioning is performed on the pixels of the student category to obtain and output the student-level mask segmentation prediction result, thus achieving student-level mask segmentation prediction. This embodiment, through accurate and efficient student segmentation in teaching videos, provides a solid foundation for subsequent important downstream tasks in educational intelligence, such as behavior recognition, attention analysis, and interaction detection.

[0103] Please refer to Figure 9 , Figure 9 This is a flowchart illustrating the steps of some embodiments of this application.

[0104] like Figure 9 As shown, in some embodiments, the deep learning-based classroom video character segmentation method provided in this application, after step S102: performing student-level mask segmentation processing on the classroom teaching panoramic video stream data based on a preset multi-scale student segmentation network model to obtain the student-level mask segmentation prediction result output by the multi-scale student segmentation network model, may further include steps S901 and S902 as shown below.

[0105] Step S901: Perform connected component analysis on the mask segmentation prediction result to obtain the analysis result; the analysis result includes: the filtering result after filtering out regions with areas smaller than a preset threshold in the mask segmentation prediction result; Step S902: Based on the video temporal sequence of the classroom teaching panoramic video stream data, smooth the filtering result to obtain the video character segmentation result.

[0106] The terminal device uses a multi-scale student segmentation network model to perform student-level mask segmentation on the panoramic video stream data of classroom teaching. After obtaining the student-level mask segmentation prediction results, it can further perform connected component analysis on the mask segmentation prediction results, filtering out regions with areas smaller than a preset threshold, and using the filtered results as the analysis results. Furthermore, the terminal device also smooths the filtering results based on the video temporal sequence of the panoramic video stream data of classroom teaching, thereby obtaining the video person segmentation results.

[0107] In some embodiments, when post-processing the segmentation mask predicted by the model, the terminal device may first process the binary mask of the current frame. Perform connected component analysis and remove components with areas smaller than threshold A. min The filtered results are obtained from the specified area. , CC(·) represents the connected component filtering operation.

[0108] Subsequently, the terminal device utilizes the video temporal consistency smoothing result: for the current frame's connected component set C t , and the post-processing result of the previous frame Calculate the maximum intersection-union ratio (LoU) and apply a threshold. : .

[0109] Among them, C t-1 The set of connected components from the previous frame. This is an indicator function.

[0110] This embodiment can effectively distinguish different student instances and smooth inter-frame prediction through this non-learning method, significantly improving the continuity and completeness of segmentation results in occluded scenes.

[0111] Next, a complete embodiment of the deep learning-based classroom video character segmentation method provided in this application is presented.

[0112] like Figure 10 As shown in the embodiment of this application, the deep learning-based classroom video character segmentation method uses an automatic student character segmentation algorithm based on Spatial-Semantic-Guided Scanning Mamba (SSGS-Mamba). Its core process includes the following steps: Step 1: Classroom video recording.

[0113] Multi-angle high-definition cameras are deployed in a standardized classroom environment to synchronously capture panoramic video streams of classroom teaching at a fixed frame rate (preferably 30fps). Each camera achieves millisecond-level time synchronization via GPS timing modules or the NTP protocol, ensuring precise alignment of the built-in time in the video stream. The video capture resolution is set to 1920x1080 (Full HD), with a dynamic range exceeding 80dB to adapt to changes in classroom lighting. H.264 encoding compression is used to reduce storage overhead. The acquisition process strictly adheres to privacy protection guidelines, with real-time mosaic processing applied to non-teaching areas. Digital watermarking technology is used to bind classroom metadata (including course number, class information, and course search time), and a synchronously generated timestamp is attached during acquisition for accurate location of video segments in subsequent work.

[0114] Step 2: Data labeling.

[0115] The original classroom video is aligned to eliminate differences in perspective from multiple cameras. The temporally continuous video stream is processed into single-slice segments at a sampling rate of 30fps, and each student figure is standardized pixel by pixel. This invention first uses the open-source general video segmentation model SAM2 (Segment Anything Model2) to predict student figure segmentation masks to obtain coarse segmentation labels, and then obtains accurate student figure segmentation masks through manual calibration, which are used for subsequent model training and evaluation.

[0116] Step 3: Data preprocessing.

[0117] The collected data was preprocessed to improve its quality and diversity, thereby further enhancing the accuracy and generalization of the trained model.

[0118] Step 4: Student semantic segmentation network guided by spatial semantics.

[0119] After obtaining the preprocessed video images of the subject, this embodiment designs a multi-scale student segmentation network based on spatial semantic guidance, which consists of an image vectorization stage, a convolutional local feature encoding stage, a Mamba global modeling stage, and a multi-scale convolutional decoder stage.

[0120] Step 5: Training.

[0121] To achieve efficient model training, all training processes were performed on a workstation equipped with an NVIDIA RTX A6000 GPU (48GB VRAM), using the Adam optimizer (momentum coefficient 0.9, weight decay 1e-4) and cosine annealing learning rate scheduling (initial learning rate 5e-4). The model training consisted of 400 epochs (batch size 16), with early stopping (10 epochs of patience) employed to prevent overfitting. Input images were uniformly scaled to 256x256 pixels for efficient model training.

[0122] Step 6: Reasoning.

[0123] The trained model is used to evaluate and infer each frame of the classroom teaching video, classifying each pixel of the video into two categories: "student" and "background". Subsequently, mask segmentation prediction at the student level is achieved through connected component partitioning. The masks obtained through accurate and efficient student segmentation in the teaching video provide a solid foundation for subsequent important downstream tasks in educational intelligence, such as behavior recognition, attention analysis, and interaction detection.

[0124] Step 7: Post-processing of prediction results Post-processing of the segmentation mask predicted by the model includes: first, performing connected component analysis on the current binary mask to remove regions with areas smaller than a threshold, obtaining the filtered result; then, smoothing the result using video temporal consistency: for the connected component set of the current frame and the post-processing result of the previous frame, calculating the maximum intersection-union ratio (MUR) and applying the threshold. This non-learning method effectively distinguishes different student instances and smooths inter-frame predictions, significantly improving the continuity and completeness of segmentation results in occluded scenes.

[0125] Step 8: Extract student morphological parameters by integrating time series information.

[0126] Compared to traditional segmentation methods, the SSGS-Mamba approach designed in this embodiment achieves efficient and accurate segmentation of student tasks in classroom videos. It can obtain high-precision segmentation masks for downstream tasks such as project status assessment. Furthermore, addressing the challenges of overlapping figures, uneven lighting leading to blurred boundaries, and diverse task scales in classrooms, this framework achieves a balance between detail perception and global modeling through the efficient fusion of CNN and Mamba. CNN captures fine-grained local details of figures in the early stages of feature extraction, while Mamba understands long-distance dependencies across regions in the later stages. This hybrid design is more lightweight than Transformer models and possesses stronger global semantic perception capabilities than pure CNN models. Moreover, existing Mamba methods rely too heavily on manually designed scanning sequences, which not only fails to overcome the long-range decay problem of Mamba but also makes it difficult for the algorithm to adaptively associate distant but semantically relevant human body regions, resulting in insufficient flexibility in complex classroom scenarios. Based on this, this embodiment proposes a spatial semantic-guided scanning mechanism to enhance Mamba's global perception capabilities. By comprehensively utilizing global person semantics, channel importance, and local contextual information to generate a spatial saliency map, and dynamically generating scanning paths for the Mamba module accordingly, it prioritizes processing key areas of people, thereby effectively improving segmentation accuracy in complex scenes. Finally, the network structure involved in this embodiment still possesses strong scalability. Whether it's the varying sizes of classrooms, the differences in student numbers due to different teaching methods, or the scene complexity caused by multi-camera shooting in practice, this algorithm can be quickly deployed and implemented.

[0127] Please refer to Figure 11 , Figure 11 This is a schematic diagram of a deep learning-based classroom video character segmentation device provided in an embodiment of this application.

[0128] like Figure 11 As shown, in some embodiments, the deep learning-based classroom video character segmentation device provided in this application may include: The acquisition module is used to acquire panoramic video stream data of classroom teaching. The model segmentation module is used to perform student-level mask segmentation processing on the panoramic video stream data of the classroom teaching based on a preset multi-scale student segmentation network model, and to obtain the student-level mask segmentation prediction results output by the multi-scale student segmentation network model. The multi-scale student segmentation network model is obtained by training a deep learning model of a multi-scale student segmentation network guided by spatial semantics on classroom video training data.

[0129] In some embodiments, the multi-scale student segmentation network includes: The vectorized convolution module is used to perform multi-layer convolution operations on the feature image to gradually compress the spatial resolution and extract low-level semantic features while preserving the local spatial structure; the feature image is a classroom video image in the classroom video training data. A convolutional neural network is used to progressively extract fine-grained information from low-level texture to mid-level semantics from the low-level semantic features to obtain shallow high-resolution features of the feature image. A sequence modeling architecture based on a selective state-space model is used to perform global modeling of the feature image based on spatial semantic guidance to obtain the deep semantic features of the feature image. A multi-scale convolutional decoder is used to fuse the shallow high-resolution features and the deep semantic features to obtain a segmentation prediction at the original resolution of the feature image.

[0130] In some embodiments, the sequence modeling architecture based on a selective state-space model includes: A spatial semantic-guided scanning order generation module is used to generate a dynamic scanning order of the feature image based on spatial semantic guidance. An enhanced visual selection scanning module is used to select two-dimensional features of the feature image and rearrange them into a one-dimensional feature sequence according to the dynamic scanning order. The one-dimensional feature sequence is used for long-range dependency modeling.

[0131] In some embodiments, the deep learning-based classroom video character segmentation apparatus provided in this application may further include: The model training module is used to acquire sample data of panoramic video streams of classroom teaching; to perform student character annotation processing on single-frame classroom teaching video image samples in the sample data of panoramic video streams of classroom teaching, so as to obtain annotated sample data of panoramic video streams of classroom teaching; the classroom video training data includes the annotated sample data of panoramic video streams of classroom teaching.

[0132] In some embodiments, the model segmentation module is further configured to input a single-frame classroom teaching video image from the panoramic classroom teaching video stream data into a preset multi-scale student segmentation network model; based on the multi-scale student segmentation network model, classify each pixel of the single-frame classroom teaching video image into a student category and a background category, and perform connected component partitioning on the pixels classified into the student category to obtain and output the mask segmentation prediction results at the student level.

[0133] In some embodiments, the deep learning-based classroom video character segmentation apparatus provided in this application may further include: The post-processing module is used to perform connected component analysis on the mask segmentation prediction results to obtain analysis results; the analysis results include: filtering results after filtering out regions with areas smaller than a preset threshold in the mask segmentation prediction results; and smoothing the filtering results based on the video temporal sequence of the classroom teaching panoramic video stream data to obtain video character segmentation results.

[0134] In some embodiments, the acquisition module is further configured to synchronously acquire original classroom video stream data in a classroom teaching environment based on a preset multi-angle camera device; and to perform alignment processing on the original classroom video stream data to eliminate perspective differences, thereby obtaining panoramic video stream data of classroom teaching.

[0135] It should be noted that the deep learning-based classroom video character segmentation device provided in this application implements the aforementioned deep learning-based classroom video character segmentation method during operation.

[0136] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned deep learning-based classroom video character segmentation method. This computer device can be an AR device, an AR device control and management device, a smartphone, tablet computer, laptop computer, desktop computer, or other terminal device.

[0137] Please see Figure 12 , Figure 12 This illustration shows the hardware structure of a computer device according to one embodiment. The computer device includes: The processor 1201 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1202 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1202 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and called and executed by the processor 1201 to implement a deep learning-based classroom video character segmentation method according to an embodiment of this application. The input / output interface 1203 is used to implement information input and output; The communication interface 1204 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1205 transmits information between various components of the device (e.g., processor 1201, memory 1202, input / output interface 1203, and communication interface 1204); The processor 1201, memory 1202, input / output interface 1203 and communication interface 1204 are connected to each other within the device via bus 1205.

[0138] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned deep learning-based classroom video character segmentation method.

[0139] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0140] This application also provides a computer program product that stores a computer program that, when executed by a processor, implements the aforementioned deep learning-based classroom video character segmentation method.

[0141] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0142] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0143] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0144] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0145] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0146] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0148] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0150] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for character segmentation in classroom videos based on deep learning, characterized in that, The method includes: Acquire panoramic video stream data of classroom teaching; The classroom teaching panoramic video stream data is processed by mask segmentation at the student level based on a preset multi-scale student segmentation network model, and the mask segmentation prediction results at the student level output by the multi-scale student segmentation network model are obtained. The multi-scale student segmentation network model is obtained by training a deep learning model of a multi-scale student segmentation network guided by spatial semantics on classroom video training data.

2. The method according to claim 1, characterized in that, The multi-scale student segmentation network includes: The vectorized convolution module is used to perform multi-layer convolution operations on the feature image to gradually compress the spatial resolution and extract low-level semantic features while preserving the local spatial structure; the feature image is a classroom video image in the classroom video training data. A convolutional neural network is used to progressively extract fine-grained information from low-level texture to mid-level semantics from the low-level semantic features to obtain shallow high-resolution features of the feature image. A sequence modeling architecture based on a selective state-space model is used to perform global modeling of the feature image based on spatial semantic guidance to obtain the deep semantic features of the feature image. A multi-scale convolutional decoder is used to fuse the shallow high-resolution features and the deep semantic features to obtain a segmentation prediction at the original resolution of the feature image.

3. The method according to claim 2, characterized in that, The sequence modeling architecture based on the selective state-space model includes: A spatial semantic-guided scanning order generation module is used to generate a dynamic scanning order of the feature image based on spatial semantic guidance. An enhanced visual selection scanning module is used to select two-dimensional features of the feature image and rearrange them into a one-dimensional feature sequence according to the dynamic scanning order. The one-dimensional feature sequence is used for long-range dependency modeling.

4. The method according to claim 1, characterized in that, The method further includes: Acquire sample data of panoramic video streams of classroom teaching; Student character annotation is performed on single-frame classroom teaching video image samples in the classroom teaching panoramic video stream sample data to obtain annotated classroom teaching panoramic video stream sample data; the classroom video training data includes the annotated classroom teaching panoramic video stream sample data.

5. The method according to claim 1, characterized in that, The process of performing student-level masking segmentation on the panoramic video stream data of classroom teaching based on a preset multi-scale student segmentation network model includes: The single-frame classroom teaching video image from the panoramic video stream data is input into a preset multi-scale student segmentation network model. Based on the multi-scale student segmentation network model, each pixel of the single-frame classroom teaching video image is classified into student and background categories. Connected component partitioning is performed on the pixels classified into the student category to obtain and output the mask segmentation prediction results at the student level.

6. The method according to claim 1, characterized in that, After obtaining the mask segmentation prediction results for the student level output by the multi-scale student segmentation network model, the method further includes: Connectivity analysis is performed on the mask segmentation prediction results to obtain analysis results; the analysis results include: filtering results after filtering out regions with areas smaller than a preset threshold in the mask segmentation prediction results; Based on the video temporal sequence of the panoramic video stream data of the classroom teaching, the filtering results are smoothed to obtain the video character segmentation results.

7. The method according to claim 1, characterized in that, The acquisition of panoramic video stream data for classroom teaching includes: Based on pre-set multi-angle camera equipment, raw classroom video stream data is collected synchronously in the classroom teaching environment; The original classroom video stream data is aligned to eliminate perspective differences, resulting in panoramic classroom teaching video stream data.

8. A deep learning-based classroom video character segmentation device, characterized in that, The device includes: The acquisition module is used to acquire panoramic video stream data of classroom teaching. The model segmentation module is used to perform student-level mask segmentation processing on the panoramic video stream data of the classroom teaching based on a preset multi-scale student segmentation network model, and to obtain the student-level mask segmentation prediction results output by the multi-scale student segmentation network model. The multi-scale student segmentation network model is obtained by training a deep learning model of a multi-scale student segmentation network guided by spatial semantics on classroom video training data.

9. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the deep learning-based classroom video character segmentation method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a deep learning-based method for class video character segmentation as described in any one of claims 1 to 7.

Images