A classroom sitting posture detection and identity recognition method and system based on multi-modal fusion and feature engineering

Abstract

The application discloses a classroom sitting posture detection and identity recognition method and system based on multi-modal fusion and feature engineering and belongs to the technical field of artificial intelligence and educational informatization. In order to solve the problem of identity tracking interruption caused by shielding, bowing and the like and the problem of single sitting posture evaluation dimension in a complex classroom scene, the application constructs a complete monitoring link from multi-path video acquisition, AI inference to Web visualized interaction based on a Sophon BM1684X edge computing device. The method core comprises the following steps: a three-level dynamic identity fusion strategy of "face first - ReID relay - temporary ID bottom-up" is adopted to realize continuous identity tracking in a shielding scene; based on 11 upper body key points, a hierarchical 19-dimensional geometric feature vector normalized by shoulder width is constructed, and a light-weight classification model is combined to realize multi-dimensional sitting posture discrimination; a class level and a student level double queue structure is adopted, a Redis cache and an MQTT asynchronous uploading mechanism are combined, and closed-loop data management without data loss in a disconnected network is realized. The system supports multi-path video stream concurrent processing on a single processor through FP16 quantization and a SAIL inference engine. Experiments show that the identity recognition accuracy is 99.2%, and the sitting posture judgment accuracy is 97.2%.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, computer vision, and educational informatization technology. Specifically, it relates to a method and system for detecting sitting posture of multiple video streams in a classroom setting based on an edge computing device with a Sophon BM1684X AI accelerator processor, as well as its user interaction page. Background Technology

[0002] In modern educational environments, students' posture and health are receiving increasing attention. Poor posture can not only affect students' learning efficiency, but also lead to a series of health problems such as scoliosis and vision loss in the long run. Traditional methods of posture monitoring mainly rely on teachers' manual inspection and reminders, which are subjective, inefficient, and difficult to achieve continuous monitoring and data management on a large scale around the clock.

[0003] Several solutions exist in the prior art that attempt to address the aforementioned problems, but all have significant shortcomings. For example, prior art document 1 (CN202411690577.X) discloses an IoT-based smart teaching management system that uses a 3D scanner to acquire students' 3D data for location monitoring. This solution lacks vision-based real-time posture detection, failing to meet the monitoring needs of daily teaching scenarios; it also lacks a face recognition module, resulting in inaccurate student identification and hindering personalized management; furthermore, its reliance on 3D scanning equipment leads to high costs and complex deployment.

[0004] Comparative document 2 (CN202510754605.8) discloses a real-time sitting posture detection method based on a Kalman filter fusion algorithm, which uses MediaPipe posture estimation and Kalman filtering to identify hunchback. Although this scheme can detect sitting posture, it lacks student identification and sitting posture data statistical analysis functions, making it unable to achieve complete teaching management; moreover, its recognition accuracy is limited in complex classroom scenarios, and it lacks the ability to monitor multiple students simultaneously, especially lacking the ability to perform concurrent processing of multiple video streams on edge computing devices.

[0005] Therefore, existing technologies cannot simultaneously achieve high-precision identity recognition, continuous identity tracking, multi-dimensional posture assessment, and data management in complex classroom scenarios, nor can they process large-scale multi-channel video streams on edge computing devices. This invention aims to solve these problems. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] The present invention aims to overcome the shortcomings of the prior art and provide a method and system that can achieve concurrent processing of multiple video streams on edge computing devices and maintain continuous identity tracking and accurate multi-dimensional posture assessment in complex classroom scenarios such as occlusion and head-down posture.

[0008] (II) Technical Solution

[0009] A classroom sitting posture detection and identity recognition method based on multimodal fusion and feature engineering includes the following steps:

[0010] S1. Construct a classroom posture monitoring edge inference system to acquire image frames from at least one classroom video stream and use the hardware decoding capability of the Sophon BM1684X AI acceleration processor to process multiple RTSP video streams;

[0011] S2. Perform human key point detection and full-image face detection in parallel on the acquired image frames, extract the coordinates of 11 key points of the upper body and the face region features respectively, and extract the appearance features of human re-identification (ReID) based on the human body region.

[0012] S3. Implement a three-level dynamic identity fusion strategy of "face priority - ReID relay - temporary ID backup", which integrates face recognition results with ReID feature matching results to determine the identity of each detected person;

[0013] S4. Based on the 11 key points of the upper body, construct a hierarchical 19-dimensional geometric feature vector with normalized shoulder width, input it into a lightweight sitting posture classification model, and output the judgment result of normal or abnormal sitting posture.

[0014] S5. Write the posture judgment results of the associated identity into the class-level record queue and the student-level historical trajectory queue respectively. Through the Redis-based dual-queue caching and MQTT asynchronous upload mechanism, a closed-loop data management system is achieved to ensure that data is not lost when the network is disconnected.

[0015] Furthermore, step S2 includes the following sub-steps:

[0016] S21. Input the image frame into the YOLOv8-pose human joint detection model and output the human bounding box and the coordinates of 11 upper body key points, including: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist.

[0017] S22. Input the image frame into the RetinaFace face detection model, set the detection threshold to 0.5, and use the ArcFace model to extract 512-dimensional face feature vectors for identity comparison of the detected face regions;

[0018] S23. Calculate the bounding boxes of the head and chest regions based on the detected key points of the upper body, adjust the region image to 256×128 pixels, input it into the OSNet person re-identification model to extract 512-dimensional ReID feature vectors, and perform L2 normalization processing.

[0019] Furthermore, step S3 includes the following sub-steps:

[0020] S31. Face-first strategy: When the confidence level of the detected face is not lower than the first threshold of 0.5, the face recognition result is directly used to determine the identity, and the ReID feature extracted in the current frame is associated with the identity and updated to the historical feature library;

[0021] S32. ReID relay strategy: When the face confidence is lower than the first threshold or no face box is detected, calculate the cosine similarity between the ReID feature vector of the current frame and each feature in the historical feature library: similarity = 1 - cos(f1, f2). If the highest similarity is not lower than the second threshold of 0.05, then the identity identifier corresponding to the historical feature is used to achieve continuous identity tracking in occluded scenarios.

[0022] S33. Temporary ID fallback strategy: When both face recognition and ReID matching fail, a temporary identity identifier unknow_n is automatically assigned to the current person, which is then associated with the skeletal key points and sitting posture results and visually annotated in the current frame; in subsequent frames, when the person's face is successfully recognized, the data associated with the temporary ID is automatically merged into the correct identity;

[0023] S34. The ReID feature library is updated online using an exponential moving average strategy. The update formula is: f_updated = α · f_old + (1-α) · f_new, where the attenuation factor α = 0.9 to suppress feature fluctuations caused by illumination changes, occlusion, and detection noise.

[0024] Furthermore, step S4 includes the following sub-steps:

[0025] S41. Construct a normalized coordinate system for shoulder width, with the midpoint between the key points of the left and right shoulders as the origin and the Euclidean distance between the two shoulders as the normalized scale factor. Translate and scale the coordinates of all 11 upper body key points to obtain key point representations with unchanged scale and position.

[0026] S42. Construct a hierarchical 19-dimensional geometric feature vector, and extract geometric quantities with physical meaning according to five structural layers:

[0027] Head stability layer: including the height difference between the left and right eyes, the height difference between the left and right ears, and the horizontal deviation of the eyes and nose;

[0028] Shoulder balance layer: including the height difference between the left and right shoulders, the horizontal distance between the shoulders, the vertical distance between the head and shoulders, and the tilt angle of the shoulder line;

[0029] Arm support layer: including the bending angle of the left and right arms, the symmetry of the arm angle, the height of the wrist relative to the shoulder, the height difference between the hands, the degree of elbow extension, and the symmetry of arm opening;

[0030] Mid-axis stabilization layer: includes the horizontal offset of the head relative to the body's mid-axis;

[0031] Confidence fusion and noise suppression layer: including average confidence of key points and proportion of low confidence points;

[0032] S43. Input the 19-dimensional feature vector into a lightweight sitting posture classification model composed of StandardScaler normalization preprocessing and Logistic regression classifier, and output the binary classification results of normal sitting posture and poor sitting posture; the classification model is trained based on labeled samples collected in real classroom scenarios, and the test accuracy reaches 97.2%.

[0033] Furthermore, step S5 includes the following sub-steps:

[0034] S51. Class-level record writing: When a student with abnormal sitting posture is detected in a class, a class record is generated containing the camera number, grade and class, number of students with warnings, total number of students with abnormal sitting posture, number of students not identified, and screenshot path. This record is written to the Redis class record queue via LPUSH operation. When the queue accumulates to the first preset threshold, the batch records are packaged and a unique request identifier is generated. The records are then asynchronously uploaded to the server via the MQTT protocol. After successful upload, the corresponding cache is automatically cleared.

[0035] S52. Student-level record writing: Maintain an abnormal posture count for each student. When the number of times the same student is judged to have incorrect posture reaches the second preset threshold, generate a personal warning record and write it to the Redis student record queue through the LPUSH operation. This counting mechanism is used to filter out false judgments caused by occasional posture jitter.

[0036] S53. In the event of a network interruption, the dual queues continuously cache records locally based on Redis, and automatically resume transmission after the network is restored, thus ensuring no data loss during network outages.

[0037] Furthermore, the human joint detection model, face detection model, face recognition model, and person re-identification model mentioned in step S2, as well as the sitting posture classification model mentioned in step S4, are all pre-converted into FP16 quantized BModel format using the Sophon Toolkit2 model conversion tool to adapt to the TPU instruction set of the Sophon BM1684X AI accelerator processor, thereby achieving low latency and high throughput edge inference performance.

[0038] This invention also provides a classroom sitting posture detection and identity recognition system based on multimodal fusion and feature engineering, comprising:

[0039] Video capture and decoding module: used to access multiple RTSP network camera video streams and acquire image frames using the hardware decoding capability of the BM1684X processor;

[0040] AI inference module: Deployed on the Sophon BM1684X edge computing device, including human joint detection model, face detection and recognition model, people re-identification model and sitting posture classification model;

[0041] Multimodal identity fusion module: used to execute the three-level dynamic identity fusion strategy, perform fusion decision on face recognition and ReID matching results, and output continuous identity identifiers for each person;

[0042] Sitting posture discrimination module: used to construct a hierarchical 19-dimensional geometric feature vector with shoulder width normalized based on the 11 upper body key points, and call a lightweight classification model to output the sitting posture discrimination result;

[0043] Data closed-loop management module: Based on the class-level record queue and student-level historical trajectory queue, it uses Redis to implement data caching, statistics and network disconnection buffering, and uses the MQTT protocol to implement asynchronous uploading and cleaning;

[0044] Web Visual Interaction Module: Provides a front-end visual interface for real-time monitoring, historical trajectory query, face database management, and system configuration.

[0045] (III) Beneficial Effects

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. High robustness of identity tracking: Through a three-level fusion strategy of face priority, ReID relay, and temporary ID as a backup, it can maintain continuous identity ID even in scenarios such as looking down or face occlusion, avoiding identity interruption or confusion.

[0048] 2. High accuracy in posture assessment: It adopts a shoulder width normalized coordinate system and hierarchical geometric feature encoding, which eliminates the influence of body shape differences and camera angle, and can effectively identify a variety of complex poor sitting postures.

[0049] 3. High concurrency and low latency at the edge: Through model FP16 quantization and SAIL inference engine, multiple video streams can be processed concurrently on a single BM1684X processor to meet the actual deployment needs of classrooms.

[0050] 4. Complete and reliable data management: The innovative dual-queue management mechanism and MQTT asynchronous upload support local caching when the network is disconnected and automatic resume transmission after reconnection, building a full-dimensional data closed loop from individuals to classes. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the overall architecture of the board-end system according to Embodiment 1 of the present invention.

[0052] Figure 2 This is a schematic diagram of the multimodal fusion engine workflow in Embodiment 2 of the present invention.

[0053] Figure 3 This is a multi-threaded processing architecture diagram of Embodiment 5 of the present invention.

[0054] Figure 4 This is a schematic diagram of the actual deployment of Embodiment 7 of the present invention.

[0055] Figure 5 This is a flowchart of model conversion and quantization in Embodiment 7 of the present invention.

[0056] Figure 6 This is a schematic diagram of the single-frame processing and display of Embodiment 8 of the present invention.

[0057] Figure 7 This is a front-end interface display diagram of Embodiment 9 of the present invention. Detailed Implementation

[0058] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0059] Overall Process Overview

[0060] The overall processing flow of the method of this invention is as follows:

[0061] Step 1, Data Acquisition: Acquire classroom video streams from multiple RTSP network cameras, and use the hardware decoding capability of the Sophon BM1684XAI accelerator processor to decode the video streams and obtain continuous image frames.

[0062] Step 2, Human Joint Detection: Scale the image frame to 640×640 pixels, input the YOLOv8-pose human joint detection model, and output the human bounding box and coordinates of 11 upper body key points, including: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist. This model is pre-converted to FP16 quantized BModel format using the Sophon Toolkit2 tool.

[0063] Step 3, Full-Image Face Recognition: A two-stage approach is adopted. First, the RetinaFace face detection model is used to detect faces in the entire image, with a detection threshold of 0.5. For the detected face regions, the ArcFace model is used to extract 512-dimensional face feature vectors for subsequent identity comparison. Both models are pre-converted to FP16 quantized BModel format.

[0064] Step 4, Person Re-identification: Based on the detected upper body key points, calculate the bounding boxes for the head and chest regions. Adjust the image of this region to a standard size of 256×128 pixels, input it into the OSNet person re-identification model to extract a 512-dimensional ReID feature vector, perform L2 normalization on the feature vector, and save it to memory. This model is also pre-converted to FP16 quantized BModel format.

[0065] Step 5, Multimodal Identity Fusion: Execute a three-level dynamic fusion strategy of "face priority - ReID relay - temporary ID fallback". Level 1: When the detected face confidence score is not lower than 0.5, the face recognition result is directly used to determine the identity identifier, and the ReID feature associated with that identity identifier is updated. Level 2: When the face confidence score is lower than 0.5 or no face bounding box is detected, the cosine similarity between the ReID feature of the current frame and each feature in the historical ReID feature library is calculated (the formula is similarity = 1 - cos(f1, f2)). If the highest similarity is not lower than 0.05, the identity identifier corresponding to that historical feature is used. Level 3: When both face recognition and ReID matching fail, a temporary identity identifier "unknow_n" is automatically assigned to the current person, associated with the skeletal key points and sitting posture results, and visually annotated in the current frame.

[0066] Step 6, Sitting Posture Classification: Based on the 11 upper body key points of the identified individuals, construct a hierarchical 19-dimensional geometric feature vector with normalized shoulder width, input it into a lightweight sitting posture classification model composed of StandardScaler normalization preprocessing and Logistic regression classifier, and output the binary classification result of normal or poor sitting posture.

[0067] Step 7, Data Management: Write the posture judgment results associated with the student's identity into the class-level record queue and the student-level historical trajectory queue, respectively. In the class-level record queue, when 20 records are accumulated, they are packaged and uploaded. In the student-level record queue, if the same student is judged to have incorrect posture three times consecutively, a personal warning record is generated, and when 20 records are accumulated, they are packaged and uploaded. Uploads are performed asynchronously via the MQTT protocol, and the corresponding Redis cache is cleared after a successful upload. In the event of a network interruption, both queues are continuously cached locally based on Redis, and uploads are automatically resumed once the network is restored.

[0068] Step 8, Feature Update: The ReID feature library is updated online using an exponential moving average strategy. The update formula is f_updated = α · f_old + (1-α) · f_new, where the decay factor α is 0.9 to suppress feature fluctuations caused by illumination changes, occlusion, and detection noise, thereby improving the continuity and robustness of identity tracking.

[0069] All the aforementioned AI models were pre-converted to FP16 quantized BModel format using the Sophon Toolkit2 model conversion tool to fully adapt to the TPU instruction set of the BM1684X processor, achieving low-latency, high-throughput edge inference performance. The system uses a parameter configuration module to uniformly manage adjustable variables such as system operating parameters, face database update parameters, camera configuration parameters, and discrimination thresholds.

[0070] Example 1: System Overall Architecture and Workflow

[0071] Reference Figure 1 The overall system architecture of this invention includes five functional layers: perception layer, analysis layer, fusion decision layer, attitude discrimination layer, and data closed loop layer.

[0072] The perception layer is responsible for collecting raw data from the physical world, including video stream acquisition, image frame preprocessing, and synchronous management of multimodal raw data, providing a stable and unified data input channel for subsequent analysis modules.

[0073] The analysis layer includes a human detection and pose estimation module, a full-image face recognition module, and a person re-identification module. All models are converted to FP16 quantized BModel format based on Sophon Toolkit2 to fully adapt to the TPU instruction set and parallel computing characteristics of the BM1684X processor, achieving high throughput and low latency edge inference performance.

[0074] The fusion decision layer is one of the core innovations of this invention, responsible for associating, fusing, and making decisions based on multimodal features from the analysis layer. (See reference...) Figure 2 This layer proposes a three-level dynamic fusion strategy of "face priority, ReID relay, and temporary ID as a backup", which can achieve continuous, robust and consistent identity tracking across frames in complex scenarios such as occlusion and lighting changes.

[0075] The posture discrimination layer constructs a 19-dimensional geometric feature vector using 11 upper body key points of the identified individual output by the fusion decision layer, and inputs it into a pre-trained classifier to achieve accurate discrimination of sitting posture.

[0076] The data closed-loop layer is responsible for organizing instantaneous detection results into long-term valuable data assets. This layer includes a dual-queue data management module, a multi-dimensional statistical engine, and a data visualization interface, which can realize identity trajectory recording, posture behavior trend analysis, and visualization output for teaching management, thus constructing a complete data closed-loop system from collection and recognition to feedback optimization.

[0077] Example 2: Technical Details of Multimodal Fusion Engine

[0078] Reference Figure 2The multimodal fusion engine is the key innovation of this invention, which achieves high-precision and continuous identity recognition through a three-level dynamic hybrid strategy.

[0079] The first level is a face-priority strategy. When the confidence level of a detected face is not lower than 0.5, the system prioritizes using the face recognition result to determine the identity and updates the person's ReID features associated with that identity. This strategy ensures that the most accurate identity information can be obtained when the face is clearly visible.

[0080] The second level is the ReID relay strategy. When the face confidence score is below 0.5 or a human skeleton is detected but no face is recognized, the system activates the person re-identification module. It calculates the cosine similarity between the person's ReID features in the current frame and each feature in the historical ReID feature library, using the formula: similarity = 1 - cos(f1, f2), where f1 is the current frame feature and f2 is the historical frame feature. If the highest similarity score is not lower than a preset threshold of 0.05, the identity identifier corresponding to that historical feature is retained. This strategy solves the problem of continuous identity tracking in complex scenarios such as students looking down or facial occlusion.

[0081] The third level is a temporary ID fallback strategy. When both face recognition and ReID matching fail to identify the person, the system automatically generates a temporary identity identifier for the currently detected person, denoted as unknow_n. This temporary identifier is recorded along with the person's skeletal key point information and pose determination results, and is visually annotated in the current video frame. In this way, even if facial features cannot be obtained, the system can still manage the person by number, achieving a one-to-one correspondence between skeletal detection results and pose states, ensuring uninterrupted person differentiation and tracking in multi-target scenarios. In subsequent frames, when the person's face is successfully recognized, the system automatically merges the data associated with the temporary ID into the correct identity.

[0082] Example 3: Seated Posture Feature Construction and Autonomous Lightweight Classification Model

[0083] Embodiment 3 of this invention proposes an independently developed upper body posture feature construction algorithm (UP-19) and a lightweight sitting posture classification network (SPC-Net) for edge deployment. This algorithm system is designed entirely based on a comprehensive analysis of the statistical patterns, geometric features, and edge computing power limitations of sitting posture behavior in classroom scenarios. It can achieve high-precision, low-latency sitting posture judgment on the BM1684X processor. Unlike existing solutions based on deep networks or simple geometric features, the UP-19 and SPC-Net proposed in this invention belong to an independently designed algorithm framework from feature system to classifier structure, making it particularly suitable for real classroom scenarios with multiple students, frequent occlusion, and significant key point jitter.

[0084] I. UP-19, an independently developed upper body posture geometric feature model

[0085] This invention systematically analyzes the geometric relationships of 11 key points of the human upper body and proposes a self-constructed 19-dimensional posture geometric feature model. Its innovations include the following three aspects.

[0086] (I) Shoulder Width Normalized Human Coordinate System. To address the incomparability of sitting posture characteristics due to different heights and body types among students, this invention proposes a shoulder width normalized coordinate system based on left and right shoulder points. Specifically, the midpoint between the coordinates of the left and right shoulder key points is taken as the origin of the coordinate system. The Euclidean distance between the two shoulders is used as the normalization scale factor, and the coordinates of all 11 upper body key points are translated and scaled. This operation ensures that all geometric features are scale-invariant and position-invariant, forming the basis of the algorithm system of this invention. After normalization, the same sitting posture of students with different body types will be mapped to a similar feature space, eliminating the interference of individual differences on the judgment.

[0087] (ii) Hierarchical geometric relationship encoding. This invention is the first to break down sitting posture into five geometric structure layers, extracting geometric quantities with clear physical meaning from each layer to achieve structured modeling of sitting posture.

[0088] The head stability layer is used to characterize the degree of head tilt and forward tilt. The extracted features include: the height difference between the left and right eyes (the larger the value, the greater the angle of head tilt), the height difference between the left and right ears (to further verify the degree of head tilt), and the horizontal offset between the center of the eyes and the nose (reflecting the forward tilt state of the head).

[0089] The shoulder balance layer is used to characterize the balance of the shoulders and the tendency of the trunk to lean forward. The extracted features include: the height difference between the left and right shoulders (reflecting the degree of shoulder imbalance), the horizontal distance between the shoulders (this value will narrow when leaning forward), the vertical distance between the head and the shoulders (negative values ​​indicate that the head is lower than the shoulders), and the tilt angle of the shoulder line relative to the horizontal line.

[0090] The arm support layer is used to characterize the support mode and shape of the upper limbs. The extracted features include: the bending angle of the left and right arms, the difference in the angle between the left and right arms, the height of the wrist relative to the shoulder, the difference in the height of the left and right wrists, the horizontal spread of the elbow relative to the shoulder (the arms are closer to the body when lying on the table), and the difference in the degree of the left and right arms opening.

[0091] The axial stabilization layer is used to characterize the degree of scoliosis or tilt, and the extracted feature is the horizontal offset of the head relative to the body's axial axis.

[0092] The confidence fusion and noise suppression layer is used to automatically suppress the interference of low-quality keypoints on the discrimination results. The extracted features include: the average confidence of 11 keypoints and the proportion of low-confidence keypoints. The introduction of noise perception statistics is an original improvement of the inventors, which enables the subsequent classifier to automatically model the uncertainty of the detection results.

[0093] The 19 features mentioned above are not simply stacked, but organized in five levels to form a complete structured description of sitting posture from head to torso, from balance to support, from geometry to detection quality.

[0094] (III) Adaptive Attitude Angle Encoding. To reduce misjudgments of sitting posture caused by differences in camera installation angles, this invention constructs stable relative angles by establishing a reference relationship with the natural vertical direction of the human body. Specifically, when calculating the arm bending angle, the angle between the upper arm direction vector and the forearm direction vector is used as the arm bending angle; when calculating the shoulder line tilt, the angle between the shoulder line vector and the horizontal direction vector is used as the tilt angle. This method makes angle calculation independent of the camera's pitch and yaw angles, significantly improving the consistency of attitude discrimination in multi-camera deployment scenarios.

[0095] II. Training and Deployment of a Lightweight Sitting Posture Classification Model

[0096] (I) Classifier Algorithm Type and Network Structure. The posture classifier adopts a machine learning pipeline combining standardized preprocessing and logistic regression, and implements model transformation through the PyTorch deep learning framework. Its core architecture consists of three layers: the preprocessing layer uses StandardScaler standardization to normalize the 19-dimensional geometric features of the input using Z-score, converting each feature dimension into a standard distribution with a mean of 0 and a standard deviation of 1; the classification layer uses a single-layer logistic regression model, which performs a linear transformation on the normalized features and then uses the Sigmoid activation function to map the output to probability values ​​between 0 and 1, representing the probability of poor posture; the output layer outputs probability values ​​for two categories, namely the probability of good posture and the probability of poor posture, the sum of which is 1. In the PyTorch model implementation, the mean and standard deviation parameters of StandardScaler are extracted as learnable tensors of the model, and the weight matrix (shape 1×19) and bias term (scalar) of Logistic regression are extracted as model parameters. During forward propagation, standardization, linear transformation, and Sigmoid activation are performed sequentially.

[0097] (II) Model Training Process. The training dataset consists of 2,000 labeled samples extracted from 64 video streams collected from actual classroom scenes, covering student data in different sitting postures. Data labeling was done manually by education experts according to sitting posture health standards, categorized into "correct sitting posture" and "incorrect sitting posture," and stored in separate folders. For each labeled sample, the UP-19 algorithm was used to extract a 19-dimensional geometric feature vector from 11 key points of the student's upper body. All sample feature vectors and corresponding labels were saved as feature vector files, containing a feature matrix X and a label vector y.

[0098] The training process is as follows: First, load the feature matrix X and label vector y; then, divide the dataset into 80% training set and 20% test set, using stratified sampling to maintain consistent class distribution across the training and test sets; construct a machine learning pipeline consisting of StandardScaler standardization and a Logistic regression classifier; finally, fit this pipeline to the training set. Key training parameters are: L-BFGS optimizer for optimization, L2 regularization for regularization to prevent overfitting, a maximum of 2000 iterations with an early stopping mechanism, and balanced mode for class balancing to automatically adjust the weights of different classes.

[0099] (III) Model Conversion and Deployment. The trained PyTorch model is converted to ONNX format to support cross-platform deployment. During conversion, the input and output names are specified, the computation set version is set to 11, constant folding optimization is enabled, and dynamic axes for input and output are declared to support variable batch inference. After the ONNX model is generated, it is further converted to the BModel format specific to the BM1684X processor using the Sophon Toolkit2 tool. Taking a trained classification model as an example, the conversion is performed in two steps: First, the ONNX model is converted to the MLIR intermediate representation format using a model conversion tool, specifying the model name, input shape, and other information, and outputting test results for accuracy verification; second, the MLIR model is converted to BModel format using a model deployment tool, specifying FP16 quantization precision and the target processor model as BM1684X, ultimately generating a quantized model file that can be directly loaded and run on edge devices.

[0100] (iv) Model Performance Evaluation. The trained classifier was evaluated on the test set. The main metrics included: accuracy, achieving a sitting posture judgment accuracy of 97.2% on the test set; precision and recall, with the precision, recall, and F1 score for each category (correct sitting posture / incorrect sitting posture) displayed in the classification report; and confusion matrix, which visualized the distribution of the number of correct and incorrect classifications across different categories to facilitate further analysis and optimization.

[0101] Example 4: Implementation of the Model Inference Engine

[0102] The system uses SAIL (Sophon AI Library) for model inference. SAIL is the official AI inference interface library for the Sophon BM1684X processor. Through this interface, the system can fully utilize the FP16 computing power of the BM1684X accelerator processor to achieve efficient model inference.

[0103] The specific workflow of the inference engine is as follows: During the initialization phase, a SAIL inference engine instance is created, the target device number is specified, and a pre-converted BModel format quantized model is loaded. After loading the model, the engine automatically obtains the network graph name and input tensor names to prepare for subsequent inference. During the inference phase, the preprocessed input data is organized into a dictionary format according to the model input names. The inference engine's processing interface is called, passing in the network graph name and the input data dictionary. The engine automatically performs forward computation on the TPU and returns the output results. Throughout the inference process, the engine automatically manages TPU memory allocation and data transfer without manual intervention. This implementation supports FP16 precision computation, significantly improving inference throughput while ensuring model accuracy, meeting the real-time requirements of concurrent processing of multiple video streams.

[0104] Example 5: Multi-stream concurrent processing architecture

[0105] Reference Figure 3 The system supports concurrent processing of multiple cameras. The core design principle of this architecture is "independent instance for each camera," which means creating an independent video decoding unit, model inference unit, and identity tracking unit for each video stream, so that the processing of each data stream is completely isolated from each other, avoiding resource contention and state interference.

[0106] The specific workflow for multi-stream video processing is as follows: First, obtain the list of RTSP addresses for all cameras from the camera configuration; then, create an independent video stream object for each RTSP address, responsible for decoding and frame acquisition of that video stream; next, create an independent pose recognition and identity tracker instance for each video stream name, which internally maintains a dedicated identity feature library and tracking status for that video stream; finally, process each video stream concurrently using a thread pool, with the maximum number of worker threads in the thread pool equal to the total number of cameras, and each thread independently handles the entire process of decoding, inference, and tracking for one video stream.

[0107] This architecture ensures that when a network fluctuation or image anomaly occurs in one camera channel, only that channel's processing is affected, while the processing of other cameras remains completely unaffected. Each recognizer instance maintains an independent identity feature database, avoiding the problem of identity confusion for the same student across different cameras.

[0108] Example 6: Implementation Details of Personnel Tracking

[0109] The system implements personnel tracking functionality through a personnel tracker class, which mainly includes three core functional modules: feature extraction, feature matching, and feature update.

[0110] The workflow of the feature extraction module is as follows: Calculate the minimum bounding box of the upper body based on the detected human skeleton points, specifically determining the bounding boxes of the head and chest regions based on key points of the upper body such as the nose, eyes, ears, shoulders, elbows, and wrists; adjust the image of this region to the standard input size of 256×128 pixels; extract 512-dimensional ReID feature vectors using the OSNet model; and perform L2 normalization on the feature vectors to make the feature vector magnitude 1, so as to eliminate the influence of feature magnitude on subsequent similarity calculations.

[0111] The feature matching module works as follows: it calculates the cosine similarity between the feature vector extracted from the current frame and each feature vector stored in the historical feature database. The similarity calculation formula is similarity = 1 - cos(f1, f2), where f1 is the feature vector of the current frame and f2 is the feature vector of the historical frame. The cosine similarity value ranges from -1 to 1, where 1 indicates that the directions are completely consistent and -1 indicates that the directions are completely opposite. After the formula conversion, the similarity value ranges from 0 to 2, with smaller values ​​indicating greater similarity between the two features. When the highest similarity is not lower than a set threshold, the current person and the historical person are considered to be the same person.

[0112] The feature update module works as follows: Tracking features are updated using an exponential moving average strategy, with the update formula being f_updated = α · f_old + (1-α) · f_new, where the decay factor α is 0.9. This strategy weights and fuses historical features with 90% weight and current frame features with 10% weight. When α is large, the feature update is more conservative, with historically accumulated appearance information dominating, suitable for stable tracking after identity verification. When rapid adaptation to appearance changes is needed, the α value can be appropriately reduced. This strategy effectively mitigates feature fluctuations caused by lighting changes, brief occlusions, and single-frame detection noise, improving the continuity and stability of identity tracking.

[0113] Example 7: Deployment Scheme and Performance Optimization

[0114] Reference Figure 4 and Figure 5 In one specific implementation, a complete deployment scheme for the system is shown.

[0115] Several network cameras are installed in the classroom. The installation location and number are determined based on the classroom area and seating layout, ensuring that all seating areas are covered by at least one camera. These cameras are connected to a switch via Ethernet cables, and the BM1684X edge computing device is also connected to this switch. The cameras transmit video streams to the edge computing device via the RTSP protocol. All the aforementioned processing modules are deployed on the edge computing device, including a human keypoint detection model, a face detection model, a face recognition model, a person re-identification model, and a posture classification model. Hardware decoding and model inference of multiple video streams are achieved through the SAIL interface. A single BM1684X device can support concurrent processing of multiple cameras.

[0116] The system employs the Sophon Toolkit2 toolchain for model conversion and optimization, fully leveraging the computing power of the BM1684X AI accelerator to achieve low-latency, high-efficiency inference performance. Through FP16 quantization technology, the precision of model parameters is reduced from 32-bit floating-point numbers to 16-bit floating-point numbers, significantly improving inference speed and reducing memory usage while maintaining model accuracy. Figure 5 The process steps for converting the model from ONNX format to the format required for BM1684X inference are demonstrated.

[0117] The specific steps for model conversion and optimization are as follows: Step 1: Convert the trained ONNX model to MLIR intermediate representation format. Use the model conversion tool, specifying the model name, ONNX model definition file path, input tensor shape, output MLIR file path, and providing test input data and expected output results for subsequent accuracy verification. Step 2: Convert the MLIR intermediate representation to BModel format executable by the BM1684X processor. Use the model deployment tool, specifying the input MLIR file path, quantization precision of F16, target processor model of BM1684X, and providing test input data and reference output results. This ultimately generates a quantized model file that can be directly loaded onto edge devices. After this conversion process, the model inference latency is significantly reduced, meeting the real-time requirements of concurrent processing of multiple video streams.

[0118] Example 8: Visualization and Data Management

[0119] Reference Figure 6The system displays the processing frame results from a single camera. In the processing frame, the system visually labels each detected student. For identified students, their name or student ID and current sitting posture (normal or abnormal) are displayed above the human body bounding box. For unidentified students, a temporary number is used, and their sitting posture is also displayed. The system simultaneously saves three types of screenshots: the first type is the original camera image, saved in a directory named after the camera number; the second type is the screenshot after the detection results are labeled, also saved in a directory named after the camera number; the third type is a close-up screenshot of the student, capturing only the student's body area and saved in a directory named after the student's name or student ID. For the third type of screenshot, the faces of other students in the image are blurred to protect their privacy.

[0120] The system's innovative dual-queue data closed-loop management system enables comprehensive data tracking and statistical analysis from individual behavior to class-wide data. The class record queue stores overall class posture statistics within a preset time period, while the student record queue stores each student's historical posture trajectory. Both queues are implemented using Redis caching. Redis, as an in-memory database, boasts extremely high read and write speeds, meeting the demands of real-time data processing, while also providing data persistence to prevent data loss due to process restarts. Data uploads are performed asynchronously via the MQTT protocol, a lightweight publish-subscribe messaging protocol suitable for data transmission in IoT scenarios. Upon successful upload, the system automatically clears the corresponding Redis cache record; if the upload fails (e.g., due to network interruption), the record remains in Redis for retry after network recovery, ensuring data integrity even during network outages.

[0121] The data management module employs two queue structures. In the class record queue, each record contains the following fields: camera serial number (identifying the data source); camera IP address; associated screenshot file name; grade information; class information; number of students receiving warnings; total number of students with abnormal posture; number of students whose identities were not verified; record generation timestamp; and screenshot file storage path. The latest record is written to the head of the Redis list via an LPUSH operation, ensuring processing in a first-in, first-out (FIFO) order. When the queue accumulates 20 records, the system automatically performs batch packaging and uploading.

[0122] Each record in the student record queue contains the following fields: camera serial number; camera IP address; associated screenshot file name; grade information; class information; student name and student ID; student ID; and record generation timestamp. The system maintains a posture error counter for each student. A personal warning record is generated and added to the student record queue only when the same student is judged to have incorrect posture three times in consecutive frames. This continuous counting mechanism effectively filters out false positives caused by occasional actions (such as briefly stretching or picking something up). The student record queue also uses an LPUSH write mechanism, with batch uploads occurring when 20 records are accumulated.

[0123] Example 9: Data Visualization and User Interface

[0124] Reference Figure 7 This invention provides a web-based front-end visual interface, employing a front-end and back-end separation architecture. The front-end interface is built using the Vue.js 3.0 framework and utilizes the Element Plus UI component library, exhibiting excellent responsiveness and cross-platform compatibility. It allows teachers and administrators to access and view data using a browser from any terminal (such as a computer or tablet) within the same local area network.

[0125] The interface mainly includes the following seven functional modules.

[0126] System dashboard interface: Provides the main system interface and serves as the entry point to other functional modules, including navigation links for parameter settings, MQTT settings, camera settings, face database updates, reference posture acquisition, and algorithm output.

[0127] Real-time monitoring interface: The processed video stream is dynamically displayed against the background of the camera's live feed. It supports simultaneous display of multiple camera feeds, provides control functions for running or stopping the face recognition algorithm, and displays the location and current sitting posture of each detected student in real time.

[0128] Individual Historical Trajectory Query Interface: Provides an interface for querying by student name or student ID, supports single-day or multi-day trajectory queries, displays each occurrence record of a specified student within a specific time period in time axis format, and also displays basic information such as the student's grade and class. Supports data export function for further analysis.

[0129] Algorithm output display interface: The algorithm processing results images are organized and displayed in folder format, supporting paginated loading and browsing of images, and providing image viewing and management functions.

[0130] System Information Monitoring Interface: Displays the configuration information of various parameters of the current algorithm, lists the names, IP addresses and online status of all connected cameras, and provides management and control functions such as system restart and algorithm restart.

[0131] Face database management interface: Supports batch uploading of new students' facial photos, automatically extracts and stores facial feature vectors, and provides functions for updating and maintaining facial features.

[0132] System settings interface: Includes configuration management functions such as parameter settings, MQTT server connection settings, and camera RTSP address settings, allowing administrators to adjust system operating parameters and connection information according to the actual deployment environment.

[0133] The front-end interface communicates with the back-end services deployed on the BM1684X edge device via an HTTP / RESTful API. The back-end services, implemented using the Flask framework, are responsible for handling various requests from the front-end, exchanging data with the edge computing device, managing detection results and statistics in the Redis database, and communicating with the cloud server via the MQTT protocol. The interface employs asynchronous loading technology to ensure real-time data updates and smooth user interaction.

[0134] Example 10: Experimental Verification

[0135] To verify the technical effects of the present invention, the following comparative experiments were conducted.

[0136] Experimental Design: To verify the effectiveness of the technical solution proposed in this invention, three sets of comparative experiments were designed. Control group 1 used a self-developed posture judgment algorithm but did not include a ReID tracking mechanism. This solution relied solely on single-modality face recognition for identity recognition. Control group 2 used multimodal fusion identity recognition (i.e., face recognition plus ReID tracking) but did not include a self-developed posture judgment algorithm. This solution integrated face recognition and ReID tracking technologies in the identity recognition stage, but still used traditional methods for posture judgment, namely comparing the similarity between the detected skeleton lines and the standard sitting posture skeleton lines to determine the correctness of the posture. The experimental group used the complete technical solution proposed in this invention, integrating face recognition and ReID tracking technologies in the identity recognition stage and using a self-developed lightweight posture classification algorithm based on machine learning in the posture judgment stage, forming a complete multimodal fusion recognition and intelligent posture judgment system. By comparing the performance of the three sets of experiments in terms of identity recognition accuracy, posture judgment precision, and system robustness, the superiority of the technical solution of this invention was verified.

[0137] Test scenario: The test was conducted in a simulated classroom environment with 4 students under a single camera. The students participating in the test exhibited different sitting postures, including normal sitting posture, writing with head down, resting chin on one hand, tilting head and leaning on the table, and other postures.

[0138] Evaluation metrics are defined as follows: Identity recognition accuracy is defined as the percentage of people who are correctly identified out of the total number of people; Posture judgment accuracy is defined as the percentage of people whose posture is correctly judged out of the total number of people.

[0139] Experimental results:

[0140] Identity recognition accuracy: Control group 1: 82.5%, Control group 2: 98.5%, Experimental group: 99.2%.

[0141] The accuracy rate of posture judgment was 97.0% in control group 1, 69.3% in control group 2, and 97.2% in the experimental group.

[0142] Experimental results show that the multimodal fusion scheme proposed in this invention (experimental group and control group 2) significantly improves the identity recognition accuracy compared to the single-modal scheme (control group 1), increasing from 82.5% to over 98.5%, fully demonstrating the effectiveness of the ReID relay mechanism in maintaining identity continuity in occluded scenarios. Regarding posture judgment accuracy, the self-developed posture judgment algorithm (experimental group and control group 1) significantly improves the accuracy compared to the traditional skeleton connection comparison method (control group 2), increasing from 69.3% to over 97.0%, proving that the scheme using hierarchical 19-dimensional geometric features and a lightweight classifier is far superior to the simple similarity comparison method. The complete scheme in the experimental group achieved optimal results in both identity recognition accuracy and posture judgment accuracy, at 99.2% and 97.2% respectively, fully demonstrating the effectiveness and advancement of the proposed technical solution.

Claims

1. A classroom sitting posture detection and identity recognition method based on multimodal fusion and feature engineering, characterized in that, Includes the following steps: S1. Obtain image frames from at least one classroom video stream, and extract key points of the upper body using a human joint detection model deployed on an edge computing device; S2. Perform face detection and recognition on the image frame, extract face features for identity matching; and simultaneously extract ReID features based on the detected human body regions. S3. A three-level dynamic fusion strategy of face priority, ReID relay, and temporary identifier as a backup is adopted to determine the identity of each inspection personnel; S4. Based on the key points of the upper body, construct a 19-dimensional normalized feature vector containing hierarchical geometric relationships of head stability, shoulder balance, arm support and central axis stability, and input it into the sitting posture classification model to determine whether the sitting posture is correct. S5. Write the identity identifier and posture judgment result into the class-level record queue and the student-level historical trajectory queue respectively. The dual-queue structure enables continuous data statistics, offline caching, and asynchronous uploading.

2. The method according to claim 1, characterized in that, Step S3 includes: S31. When the confidence level of the detected face is not lower than the first threshold, the face recognition result is used to determine the identity and the ReID feature associated with the identity is updated; S32. When the face confidence score is lower than the first threshold or no face is detected, calculate the cosine similarity between the ReID feature of the current frame and the features in the historical ReID feature library. If the highest similarity score is not lower than the second threshold, then the identity identifier corresponding to the historical feature is used. S33. When both face recognition and ReID matching fail, assign a temporary unique identifier unknown_n to the current person, associate it with the skeletal key points and pose results, and perform visual annotation in the current frame.

3. The method according to claim 1, characterized in that, Step S4 includes: S41. Define a shoulder width normalized coordinate system, translate the key points of the upper body relative to the midpoints of the left and right shoulders and normalize them with the shoulder width scale to obtain the coordinates of the key points with unchanged scale and position; S42. Calculate hierarchical geometric features, including the height difference between the left and right eyes, the height difference between the left and right ears, and the horizontal offset of the eyes and nose in the head stability layer; the height difference between the left and right shoulders, the horizontal distance between the shoulders, the vertical distance between the head and shoulders, and the shoulder line inclination in the shoulder balance layer; the angle between the left and right arms, the symmetry of the arm angle, the relative height of the wrist to the shoulder, the height difference between the hands, the degree of elbow extension and the symmetry of arm opening in the arm support layer; the body mid-axis offset in the mid-axis stability layer; and statistical features reflecting the confidence of key points. S43. The hierarchical geometric features are concatenated into a 19-dimensional feature vector, which is then input into a lightweight sitting posture classification model that has undergone standardized preprocessing and logistic regression classification. The output is a judgment result indicating whether the sitting posture is normal or abnormal.

4. The method according to claim 1, characterized in that, Step S5 includes: S51. The number of people with abnormal postures and the number of unidentified people in the same class are detected and compiled into class records, which are then pushed into the class record queue. When the cumulative number of class records reaches the first preset number, they are packaged and uploaded to the server asynchronously via a message protocol, and the corresponding images are uploaded at the same time. S52. For each individual student, when incorrect sitting posture is detected multiple times consecutively, a personal warning record is generated and added to the student record queue. When the cumulative number of student records reaches the second preset number, they are packaged and uploaded. S53. Both queues are implemented based on in-memory databases, supporting first-in-first-out order, local caching when the network is interrupted, and automatic cleanup after successful upload, so as to ensure that no data is lost when the network is disconnected.

5. The method according to claim 1, characterized in that, The human joint detection model, face detection and recognition model, people re-identification model, and sitting posture classification model are all pre-converted into a quantized model format adapted to the edge AI accelerator using a model conversion tool. Among them, the sitting posture classification model is converted from a trained logistic regression pipeline to ONNX format via PyTorch, and then generated into a BModel file dedicated to the target processor through model compilation and FP16 quantization, and deployed to run on the Sophon BM1684X AI accelerator processor.

6. A classroom sitting posture detection and identity recognition system based on multimodal fusion and feature engineering, characterized in that, The system is used to implement the method according to any one of claims 1 to 5; the system comprises: The video acquisition and decoding module is used to acquire and decode multiple RTSP video streams; The AI ​​inference module, deployed on an edge AI accelerator, includes a human key point detection model, a face detection and recognition model, a person re-identification model, and a sitting posture classification model. The multimodal fusion engine executes the three-level dynamic fusion strategy and outputs a continuous identity identifier for each person. The sitting posture recognition engine outputs the sitting posture state based on the 19-dimensional geometric feature vector and a lightweight classifier. The data closed-loop management module is based on a dual-queue structure of class queues and student queues, and uses an in-memory database to realize data caching, statistics and asynchronous uploading; The web-based visual interaction module provides a front-end interface for real-time monitoring, historical trajectory query, and system configuration.

Images