A method and system for synchronous control of paper screen in smart classroom

Abstract

The present invention proposes a method and system for synchronous control of paper screens in smart classrooms, the method comprising: arranging sensing units in the classroom, collecting student data in real time for data fusion, and obtaining multimodal data; performing format standardization processing on the multimodal data, then extracting key features from the standardized multimodal data, and performing feature fusion and dimensionality reduction processing on the key features; designing a paper screen synchronization algorithm; creating a virtual 3D interactive space, designing the integration and synchronous control of holographic projection; constructing an intelligent situation prediction model, generating personalized teaching content and suggestions; designing an interactive feedback mechanism; encrypting data transmission, designing a fault recovery mechanism to automatically perform fault switching and recovery when a fault occurs; and designing a friendly user interface and interactive mechanism. The present invention not only overcomes the shortcomings of traditional classroom paper screen technology in terms of synchronization delay, recognition accuracy, and interactivity, but also further improves the interactivity and personalization of classroom teaching.

Description

Technical Field

[0001] The present invention belongs to the technical field of classroom paper-screen synchronization, and in particular, relates to a smart classroom paper-screen synchronization control method and system. Background Art

[0002] With the development of educational informatization, classroom teaching models are gradually moving towards intelligence and digitization, and the synchronous control of paper notes and electronic screen content has become a major demand in modern classrooms. In the traditional teaching method, teachers write on the blackboard or display teaching content on the electronic screen, while students take notes on paper. This method has many shortcomings, such as limited interaction between teachers and students, the inability to share students' notes in real time, and difficulty in providing effective personalized tutoring and feedback in class. In recent years, although some smart pens, electronic paper and other products have appeared on the market, most of these products can only achieve one-way note recording and digital conversion, and lack the ability of real-time synchronization and two-way interaction. For example, a smart pen can convert paper notes into electronic text, but it cannot synchronize the content on the electronic screen to paper in real time; electronic paper can display electronic text, but the writing experience is different from traditional paper notes, and the price is relatively expensive, which is not suitable for large-scale popularization.

[0003] In addition, although some existing smart classroom systems can achieve paper-screen synchronization to a certain extent, they often rely on a single technical means, such as handwriting recognition technology based on optical recognition or simple wireless transmission technology. These systems face many challenges in practical applications, including low handwriting recognition accuracy, large synchronization delay, unstable data transmission, etc. These technical defects lead to poor classroom interaction and cannot truly achieve seamless synchronization and efficient interaction between paper notes and electronic screen content.

[0004] Therefore, the existing technology urgently needs a more intelligent, accurate and stable solution to achieve real-time synchronization and two-way interaction between paper notes and electronic screen content in classroom teaching, so as to improve teaching effectiveness and students' learning experience. Summary of the invention

[0005] The purpose of the present invention is to design a smart classroom paper screen synchronization control method and system. Through the organic combination of multimodal perception technology, holographic projection technology and intelligent situation prediction algorithm, the present invention not only overcomes the shortcomings of traditional technology in synchronization delay, recognition accuracy and interactivity, but also further improves the interactivity and personalization of classroom teaching.

[0006] In order to achieve the above object, a first aspect of the present invention provides a method for controlling paper-screen synchronization in a smart classroom, the method comprising the following steps:

[0007] Arrange a perception unit in the classroom to collect student data in real time, pre-process the student data, and perform data fusion on the pre-processed data to obtain multimodal data; the perception unit includes a visual perception unit, a speech perception unit, and a tactile perception unit;

[0008] Standardize the format of multimodal data, extract key features from the standardized multimodal data, and perform feature fusion and dimensionality reduction on the key features;

[0009] Design a paper-screen synchronization algorithm to update paper note content to the electronic screen;

[0010] Based on the paper-screen synchronization algorithm, a virtual 3D interactive space is created, and the integration and synchronization control of holographic projection is designed;

[0011] Building an intelligent context prediction model based on machine learning algorithms, which dynamically predicts students' learning contexts based on multimodal data and historical data, and generates personalized teaching content and suggestions;

[0012] Design an interactive feedback mechanism, based on which the teacher annotates and explains the students' notes through holographic projection;

[0013] Encryption technology is used to encrypt data transmission, and identity authentication access and alarm devices are set up to ensure the security of data transmission. Fault recovery mechanisms are designed to automatically perform fault switching and recovery when a fault occurs;

[0014] At the same time, a friendly user interface and interactive mechanism are designed to optimize the system based on user feedback;

[0015] Wherein, the intelligent situation prediction model includes:

[0016] Establish a state space model and construct the state transfer equation, which is expressed as follows:

[0017] x t+1 =Ax t +Bu t +w t

[0018] Among them, x t+1 represents the updated state vector, x t represents the state vector, A represents the state transfer matrix, B represents the control matrix, u t represents the control vector, w t represents process noise;

[0019] The observation equation is expressed as follows:

[0020] y t =Cx t+v t

[0021] Among them, y t represents the observation vector, C represents the observation matrix, v t represents the observation noise;

[0022] Use the Kalman filter to predict the situation and update the state estimate, which is expressed as follows:

[0023]

[0024] P t|t-1 =AP t-1|t-1 A T +Q

[0025] in, represents the predicted state, P t|t-1 represents the prediction covariance, Q represents the process noise covariance matrix, represents the predicted state at the previous moment, P t-1|t-1 represents the prediction covariance of the previous moment, and T represents transposition;

[0026] Updated as follows:

[0027] K t =P t|t-1 C T (CP t|t-1 C T +R) -1

[0028]

[0029] P t|t =(IK t C)P t|t-1

[0030] Among them, K t represents the Kalman gain, R represents the observation noise covariance matrix, I represents the identity matrix, represents the updated prediction state, P t|t represents the updated forecast covariance.

[0031] In the above scheme, a visual perception unit is installed in the classroom to collect visual data and pre-process the visual data; a voice perception unit is installed in the classroom to collect voice data and pre-process the voice data; a tactile perception unit is installed on the student side to collect tactile data and pre-process the tactile data; the pre-processed visual data, voice data and tactile data are fused to obtain fused data;

[0032] Wherein, the preprocessing of visual data includes image correction, image segmentation and image enhancement;

[0033] The preprocessing of the speech data includes noise reduction and enhancement;

[0034] The preprocessing of the tactile data includes pressure calibration and writing speed calculation;

[0035] The fused data obtained by performing data fusion is expressed as follows:

[0036] D fusion (t) = αI(t) + βS(t) + γP(t)

[0037] Among them, D fusion (t) represents the fused data, I(t) represents the image data, S(t) represents the speech data, P(t) represents the pressure data, and α, β, and γ represent the fusion weight coefficients.

[0038] In the above scheme, the format standardization processing includes formatting and time aligning the visual data, the voice data and the tactile data;

[0039] The visual data is represented as a two-dimensional matrix I ij (t), where i, j represents pixel positions and t represents the time frame;

[0040] Speech data is represented as a one-dimensional vector S k (t), where k represents the sampling point and t represents the time frame;

[0041] Tactile data is represented as a one-dimensional vector P m (t), where m represents the pressure sensor reading and t represents the time frame;

[0042] The key features extracted from the standardized multimodal data include:

[0043] Extracting the outline of the handwriting from the visual data using edge detection and morphological transformation, and extracting straight line features as image features using Hough transformation;

[0044] Extracting spectral features from the speech data using short-time Fourier transform, and then extracting Mel-frequency cepstral coefficients as speech features;

[0045] From the tactile data, calculating the mean and standard deviation of the pressure signal, and the mean and standard deviation of the writing speed as tactile features;

[0046] The feature fusion and dimensionality reduction processing of the key features includes:

[0047] Let the image feature be F I , the speech feature is F S , the tactile characteristic is F P, then the fused multimodal feature vector F is expressed as:

[0048] F c =[F I ,F S ,F P ]

[0049] Among them, F I ,F S ,F P Represent the feature sub-vectors extracted from image, speech and tactile data respectively;

[0050] Assume that the fused feature vector matrix is ​​X and its covariance matrix is ​​R:

[0051]

[0052] Among them, μ x represents the mean of the eigenvector;

[0053] Calculate the eigenvalues ​​and eigenvectors of R, and select the eigenvectors corresponding to the first k largest eigenvalues ​​to form the projection matrix W:

[0054] W=[w1,w2,…,w k ]

[0055] The feature representation after dimensionality reduction is:

[0056] F PCA =XW

[0057] Among them, F PCA Represents the feature representation after dimensionality reduction.

[0058] In the above solution, the paper-screen synchronization algorithm includes:

[0059] Determine the data transmission path and nodes, and use the network time protocol to synchronize the time of each node to ensure clock consistency;

[0060] Design data transmission protocols to ensure low-latency and real-time data transmission between nodes, and use the User Datagram Protocol for data transmission to reduce transmission delays;

[0061] A ring buffer is designed for data caching, and data enters the buffer in FIFO mode; wherein the management algorithm of the ring buffer is as follows:

[0062] Buffer[i]=Dataif0≤i <L

[0063] Where Buffer[i] represents the data at the i-th buffer position;

[0064] A synchronization control algorithm based on a physical model is designed, and a proportional-integral-differential control algorithm is used to adjust data synchronization; the proportional-integral-differential control algorithm is expressed as follows:

[0065]

[0066] Among them, u(t) represents the control amount, e(t) represents the error, K p ,K i ,K d They represent proportional, integral and differential coefficients respectively, dt represents the derivative, and de(t) represents the differential of the error e(t);

[0067] Perform timestamp correction on the received data packets to ensure data synchronization, and display the corrected data in timestamp order to ensure real-time synchronization between paper notes and electronic screen content;

[0068] The synchronization error is detected regularly and adjusted using a synchronization error correction algorithm; wherein the synchronization error correction algorithm is expressed as follows:

[0069] E s =T c -T d

[0070] Among them, T c Indicates the current timestamp, T d Indicates the expected timestamp, E s Indicates synchronization error;

[0071] Monitor data transmission and synchronization status in real time and handle abnormalities promptly.

[0072] In the above scheme, the integration and synchronous control of the holographic projection includes:

[0073] Select high-resolution holographic projection equipment to be installed in the classroom, and generate holographic images based on the principles of physical optics;

[0074] The fused multimodal feature vector F is converted into the input parameter of the holographic image, and the corresponding image and text content are generated according to the feature vector. Then, the image contrast is improved by using histogram equalization, and the text content is generated according to the content of the feature vector and rendered on the image.

[0075] The projection content is adjusted in real time according to the teacher's explanation and the students' notes, and the proportional integral differential control algorithm is used to adjust the projection content, which is expressed as follows:

[0076] e(t)=r(t)-y(t)

[0077]

[0078] Among them, e(t) represents the error, r(t) represents the reference input, y(t) represents the actual output, u(t) represents the control amount, K p ,K i ,K d denote the proportional, integral and differential coefficients respectively;

[0079] Synchronize paper notes with holographic projection content, and provide real-time feedback through the holographic projection device.

[0080] In the above scheme, the intelligent context prediction model dynamically predicts the student's learning context based on multimodal data and historical data, and generates personalized teaching content and suggestions, including:

[0081] The multimodal data is standardized so that it can be analyzed under the same dimension. Then the multimodal data is segmented according to a fixed time window, and each segment is processed as a time series data block, which is expressed as follows:

[0082] D k ={X t |kT≤t<(k+1)T}

[0083] Among them, D k represents the data in the kth time window, X t represents the data at time t;

[0084] Use the time series pattern recognition model to identify time series patterns and extract key features, which are expressed as follows:

[0085]

[0086] Where R(τ) represents the autocorrelation function of lag τ, N represents the data length, μ represents the data mean, and X t+τ represents the data mean;

[0087] The time domain data is then converted to frequency domain data and the frequency components are identified, which is represented as follows:

[0088]

[0089] Where F(f) represents the Fourier transform result of frequency f, e -i2πft represents the complex exponential function, which is the basis function in Fourier transform;

[0090] The features extracted by temporal pattern recognition are fused with the original multimodal features to form a comprehensive feature vector;

[0091] The comprehensive feature vector is input into the intelligent situation prediction model, and a situation-based rule library is established. According to the situation prediction results, personalized teaching suggestions are generated and provided to teachers and students.

[0092] In the above solution, the interactive feedback includes visual feedback, voice feedback and tactile feedback.

[0093] In the above solution, the interactive feedback mechanism generates personalized feedback information based on the multimodal data collected in real time, including:

[0094] The visual feedback provides real-time suggestions for improving students’ notes based on the content and structure of their notes, including:

[0095] Use OCR technology to recognize text and formulas in student notes and generate note scores, as shown below:

[0096] S note =α1P+α2L+α3F

[0097] Among them, S note represents the note score, P represents completeness, L represents logic, F represents formula correctness, α1, α2, α3 represent weight coefficients;

[0098] The voice feedback uses voice recognition technology to transcribe the voice information in the classroom and generates a note score based on the voice information, which is expressed as follows:

[0099] S voice =β1q+β2r+β3i

[0100] Among them, S voice represents the speech score, q represents clarity, r represents speech rate, i represents interaction frequency, and β1, β2, and β3 represent weight coefficients;

[0101] The tactile feedback analyzes the student's writing behavior through sensor data, identifies the student's writing posture, and generates a pressure evaluation index, which is expressed as follows:

[0102] S pressure =γ1μ P +γ2σ P

[0103] Among them, S pressure represents the stress score, μ P represents the mean pressure, σ P represents the pressure standard deviation, γ1 and γ2 represent the weight coefficients.

[0104] In the above scheme, the method of encrypting data transmission by using encryption technology includes:

[0105] In the classroom network environment, an adaptive optimal path algorithm is used to dynamically select the data transmission path, as shown below:

[0106]

[0107] Among them, d i represents the distance of the path segment, l j represents the delay of the path segment, P opt represents the optimal path, m represents the number of factors of path segment delay, n represents the number of path segments, It indicates that the path P is selected so that the total distance and delay of the path are minimized;

[0108] Use distributed storage systems at key nodes to provide redundant storage and verification of data, and set up a scheduled backup mechanism to regularly back up data to cloud storage;

[0109] During data transmission, the data is encrypted using an encryption algorithm, as shown below:

[0110] C k =E k (P)

[0111] Among them, C k represents ciphertext, P represents plaintext, E k represents the encryption function, and k represents the key.

[0112] In a second aspect of the present invention, a smart classroom paper-screen synchronization control system is provided, the system comprising:

[0113] A sensor deployment unit is used to arrange perception units in the classroom, collect student data in real time, pre-process the student data, and perform data fusion on the pre-processed data to obtain multimodal data; the perception units include a visual perception unit, a speech perception unit, and a tactile perception unit;

[0114] A data processing unit is used to perform format standardization processing on the multimodal data, then extract key features from the standardized multimodal data, and perform feature fusion and dimensionality reduction processing on the key features;

[0115] The data integration unit is used to design a paper-screen synchronization algorithm to update the paper note content to the electronic screen; based on the paper-screen synchronization algorithm, a virtual 3D interactive space is created, and the integration and synchronization control of the holographic projection is designed;

[0116] A scenario prediction unit, which is used to build an intelligent scenario prediction model based on a machine learning algorithm. The intelligent scenario prediction model dynamically predicts students' learning scenarios based on multimodal data and historical data, and generates personalized teaching content and suggestions;

[0117] An interactive feedback unit, used to design an interactive feedback mechanism, according to which the teacher annotates and explains the student's notes through holographic projection;

[0118] Data transmission encryption unit, used to encrypt data transmission using encryption technology, and set identity authentication access and alarm devices to ensure the security of data transmission, and design a fault recovery mechanism to automatically perform fault switching and recovery when a fault occurs;

[0119] System update unit, which is responsible for designing user-friendly interfaces and interactive mechanisms to optimize the system based on user feedback;

[0120] Wherein, the intelligent situation prediction model includes:

[0121] Establish a state space model and construct the state transfer equation, which is expressed as follows:

[0122] x t+1 =Ax t +Bu t +w t

[0123] Among them, x t+1 represents the updated state vector, x t represents the state vector, A represents the state transfer matrix, B represents the control matrix, u t represents the control vector, w t represents process noise;

[0124] The observation equation is expressed as follows:

[0125] y t =Cx t +v t

[0126] Among them, y t represents the observation vector, C represents the observation matrix, v t represents the observation noise;

[0127] Use the Kalman filter to predict the situation and update the state estimate, which is expressed as follows:

[0128]

[0129] P t|t-1 =AP t-1|t-1 A T +Q

[0130] in, represents the predicted state, P t|t-1 represents the prediction covariance, Q represents the process noise covariance matrix, represents the predicted state at the previous moment, P t-1|t-1 represents the prediction covariance of the previous moment, and T represents transposition;

[0131] Updated as follows:

[0132] K t =P t|t-1 C T (CP t|t-1 C T +R) -1

[0133]

[0134] P t|t =(IK t C)P t|t-1

[0135] Among them, K t represents the Kalman gain, R represents the observation noise covariance matrix, I represents the identity matrix, represents the updated prediction state, P t|t represents the updated forecast covariance.

[0136] The beneficial technical effects of the present invention are at least as follows:

[0137] (1) The present invention uses the comprehensive perception and analysis of multimodal data such as vision, speech and touch to achieve high-precision synchronization of paper notes and electronic screen content. Compared with traditional single technical means, the multimodal perception technology of the present invention integrates multiple data sources through a deep learning model (such as a multimodal Transformer model), significantly improving the accuracy of handwritten content recognition and the stability of synchronization, overcoming the problems of low recognition accuracy and unstable data transmission in the prior art.

[0138] (2) The present invention introduces holographic projection equipment in the classroom to create a virtual 3D interactive space. Through holographic projection technology, the content of paper notes can be projected on the virtual screen in real time, and all students and teachers can intuitively see the synchronized note content. This immersive interactive experience not only enhances the fun of the class, but also solves the problem that students' notes cannot be shared in real time in traditional classrooms, thereby improving classroom interactivity.

[0139] (3) The present invention predicts students' learning situations and needs based on multimodal data and historical data through machine learning algorithms (such as LSTM, Transformer). The intelligent situation prediction module can dynamically adjust the display method and interaction strategy of the paper screen content, and provide personalized learning suggestions and feedback. Compared with the fixed content display method in the prior art, the intelligent situation prediction of the present invention not only improves the personalization of classroom teaching, but can also be adjusted according to students' real-time feedback, solving the problem of lack of personalized guidance and feedback in classroom teaching.

[0140] (4) The present invention designs a low-latency data transmission protocol and an efficient synchronization algorithm to ensure real-time synchronization and two-way interaction between paper notes and electronic screen content. Compared with the synchronization delay problem in traditional technologies, the present invention achieves instant updating and interaction of paper screen content through an optimized synchronization control system, significantly improving the efficiency and effectiveness of classroom teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0141] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0142] Figure 1 This is a flow chart of a smart classroom paper-screen synchronization control method of the present invention.

[0143] Figure 2 This is a framework diagram of a smart classroom paper-screen synchronization control system of the present invention. DETAILED DESCRIPTION

[0144] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0145] In one or more embodiments, Figure 1 As shown, a method for controlling paper-screen synchronization in a smart classroom is disclosed, and the method comprises the following steps S1-S8:

[0146] S1. Arrange a perception unit in the classroom to collect student data in real time, pre-process the student data, and fuse the pre-processed data to obtain multimodal data; the perception unit includes a visual perception unit, a speech perception unit, and a tactile perception unit.

[0147] In this embodiment, a visual perception unit (high-resolution camera), a voice perception unit (microphone array) and a tactile perception unit (smart pen or touch screen) are arranged in the classroom. These devices collect students' note content, classroom voice and writing behavior data in real time.

[0148] Preferably, a high-resolution camera (such as 1080p or 4K resolution) is installed in the classroom. The camera should be installed in the four corners of the classroom to ensure that all students' writing areas are covered and blind spots are avoided. The camera continuously collects image data at a fixed frame rate (such as 30fps) and transmits the image data stream to the preprocessing unit. Through the OpenCV library, the image stream of the camera is obtained in real time and converted into a grayscale image to reduce the amount of calculation.

[0149] Specifically, image correction uses the perspective transformation formula to correct the image and eliminate the image distortion caused by the camera position. Let the original image coordinates be (x, y), the corrected image coordinates be (x′, y′), and the perspective transformation matrix H be:

[0150]

[0151] Correction formula:

[0152]

[0153] Image segmentation uses the Canny edge detection algorithm and Hough transform to detect and segment the paper boundaries in the image. The Canny edge detection formula is:

[0154]

[0155] Among them, G x and G y are the gradients of the image in the x and y directions, respectively.

[0156] Hough transform formula:

[0157] ρ=xcosθ+ysinθ (4)

[0158] Detect the line (ρ,θ) and determine the bounding box of the paper.

[0159] Image enhancement uses contrast stretching and histogram equalization to enhance the image to improve the visibility of handwritten content. The contrast stretching formula is:

[0160]

[0161] Among them, I is the original pixel value, I min and I max are the minimum and maximum pixel values ​​of the image, respectively.

[0162] Specifically, the voice data design is based on the arrangement of omnidirectional microphone arrays in the classroom to ensure coverage of all areas where students and teachers speak. The microphone array collects voice signals in real time and converts analog signals into digital signals through an analog-to-digital converter (ADC). The sampling rate is set to 44.1kHz to ensure high fidelity of the voice signal.

[0163] Preferably, the noise reduction and enhancement is to use an adaptive filter (LMS algorithm) and a noise suppression algorithm to perform noise reduction processing on the speech signal to improve the clarity of the speech signal. The LMS adaptive filter formula is:

[0164] y(n)=w T (n)x(n) (6)

[0165] e(n)=d(n)-y(n) (7)

[0166] w(n+1)=w(n)+μe(n)x(n) (8)

[0167] Among them, y(n) is the filter output, e(n) is the error, w(n) is the filter weight, x(n) is the input signal, and μ is the learning rate.

[0168] Specifically, tactile data collection is done by students using a smart pen with a pressure sensor to record the pressure changes and writing speed during the writing process. The smart pen transmits the writing pressure and speed data to the preprocessing unit in real time via Bluetooth or wireless network. The data sampling frequency is set to 100Hz to ensure the continuity and accuracy of the collection.

[0169] Preferably, the pressure calibration uses a linear regression method to calibrate the reading of the pressure sensor to ensure the accuracy of the data. The linear regression model is:

[0170] P=β0+β1V (9)

[0171] Where P is the calibrated pressure value, V is the original reading of the sensor, and β0 and β1 are regression coefficients.

[0172] The writing speed is calculated based on the displacement data of the sensor. The writing speed formula is:

[0173]

[0174] Among them, v(t) is the writing speed, x(t) and y(t) are the horizontal and vertical coordinates of the pen tip respectively, and t is time.

[0175] Further, data fusion and storage: First, the visual, voice and tactile data are synchronized using timestamps to ensure the consistency of multimodal data. A unified UTC timestamp format is used to ensure time synchronization of different data sources. Then the pre-processed multimodal data is fused to provide input for subsequent analysis. Data fusion formula:

[0176] D fusion (t)=αI(t)+βS(t)+γP(t) (11)

[0177] Among them, D fusion (t) is the fused data, I(t) is the image data, S(t) is the speech data, P(t) is the pressure data, and α, β, γ are the fusion weight coefficients.

[0178] Data storage is to store the fused multimodal data in a database, using a NoSQL database (such as MongoDB) to support efficient storage and retrieval of large-scale data.

[0179] S2. Standardize the format of multimodal data, extract key features from the standardized multimodal data, and perform feature fusion and dimensionality reduction on the key features.

[0180] In this embodiment, a multimodal Transformer model is used to integrate visual, speech and tactile data for comprehensive analysis and processing. The model can effectively identify the content of paper notes and extract valuable synchronization control signals.

[0181] Specifically, data format standardization and alignment is to standardize the data from different sensors (vision, voice, touch) into a unified format and timeline. Then the same data format:

[0182] Image data: 2D matrix I ij (t), where i, j represent pixel positions and t represents the time frame.

[0183] Speech data: one-dimensional vector S k (t), where k represents the sampling point and t represents the time frame.

[0184] Pressure data: one-dimensional vector P m (t), where m represents the pressure sensor reading and t represents the time frame.

[0185] Linear interpolation is used to time-align data of different frequencies so that all data have the same timestamp at the same time point.

[0186] t i =t0+iΔt

[0187] Among them, t i is the timestamp of the i-th time frame, Δt is the time interval, and t0 is the initial time.

[0188] Furthermore, multimodal feature extraction is to extract key features from the standardized data for subsequent fusion and analysis.

[0189] Specifically, image feature extraction uses edge detection and morphological transformation to extract the outline of handwritten notes:

[0190]

[0191] Among them, G x and G y are the gradients of the image in the x and y directions, respectively.

[0192] Then use Hough transform to extract straight line features:

[0193] ρ=xcosθ+ysinθ

[0194] Where ρ is the distance from the origin to the line, and θ is the angle of the line.

[0195] Speech feature extraction uses short-time Fourier transform (STFT) to extract spectral features:

[0196]

[0197] Among them, X(f,t) is the spectrum, x(n) is the speech signal, w(nt) is the window function, f is the frequency, and t is the time.

[0198] Extract Mel-frequency cepstral coefficients (MFCC) as features:

[0199]

[0200] Tactile feature extraction is to calculate the mean and standard deviation of the pressure signal:

[0201]

[0202] Calculate the mean and standard deviation of writing speed.

[0203]

[0204] Furthermore, multimodal feature fusion combines the features of different modalities to form a comprehensive feature vector, including:

[0205] Let the image feature be F I , the speech feature is F S , the tactile characteristic is F P , then the fused feature vector F is expressed as:

[0206] F=[F I ,F S ,F P ]

[0207] Among them, F I ,F S ,F P are feature sub-vectors extracted from image, speech and tactile data respectively.

[0208] Furthermore, principal component analysis (PCA) is used to reduce the dimension of the fused feature vector, retain the main information and reduce the computational complexity: Let the fused feature vector matrix be X and its covariance matrix be R:

[0209]

[0210] Among them, μ X is the mean of the eigenvectors.

[0211] Calculate the eigenvalues ​​and eigenvectors of C, and select the eigenvectors corresponding to the first k largest eigenvalues ​​to form the projection matrix W:

[0212] W=[w1,w2,…,w k ]

[0213] The feature representation after dimensionality reduction is:

[0214] F PCA =XW

[0215] Through the current operation, efficient fusion and analysis of multimodal data is achieved. Each step is closely combined with the specific needs of the patent, from data standardization, feature extraction, feature fusion to feature dimensionality reduction, forming an organic whole, providing high-quality comprehensive feature vectors for subsequent real-time synchronous control, ensuring the efficiency and accuracy of the system.

[0216] S3. Design a paper-screen synchronization algorithm to update the paper note content to the electronic screen.

[0217] In this embodiment, the optimized data transmission protocol and synchronization algorithm ensure the instant synchronization and update of the content.

[0218] Specifically, the data transmission path and nodes are first determined, including data acquisition equipment, preprocessing units, feature fusion modules and display terminals. The time synchronization of each node uses the Network Time Protocol (NTP) to ensure the clock consistency of the system.

[0219] Then, a low-latency data transmission protocol is designed to ensure that data between nodes can be transmitted in real time. The User Datagram Protocol (UDP) is used for data transmission to reduce transmission delay. UDP header format:

[0220]

[0221] Define the transfer packet structure, which includes timestamp, data type and data payload:

[0222] Packet=[Timestamp Data Type Data Payload] (13)

[0223] Furthermore, a circular buffer is designed for data caching to ensure the stability of the data flow. The length of the circular buffer is L, and data enters the buffer in FIFO (first in first out) mode. Buffer management algorithm:

[0224] Buffer[i]=Data if 0≤i <L (14)

[0225] Among them, Buffer[i] represents the data at the i-th buffer position.

[0226] Preferably, a synchronization control algorithm based on a physical model is designed to ensure the synchronous display of different data sources. A proportional integral differential (PID) control algorithm is used to adjust data synchronization. The PID controller formula is:

[0227]

[0228] Among them, u(t) is the control amount, e(t) is the error (the difference between the actual timestamp and the expected timestamp), K p ,K i ,K d are the proportional, integral and differential coefficients respectively, dt represents the derivative, and de(t) represents the differential of the error e(t).

[0229] Perform timestamp correction on the received data packets to ensure data synchronization. Timestamp correction formula:

[0230] Let the receiving time be T r , the sending time is T s , the network delay is ΔT, and the corrected timestamp is T c :

[0231] T c =T r -ΔT (16)

[0232] Among them, ΔT can be obtained by measuring network transmission delay.

[0233] Furthermore, the corrected data is displayed in the order of timestamps to ensure real-time synchronization between paper notes and electronic screen content. Display buffer management: Assume that the display buffer length is M, and data enters the display buffer in the order of timestamps:

[0234] Display Buffer[j]=Calibrated Data if 0≤j <M (17)

[0235] The synchronization error is detected regularly and adjusted using the synchronization error correction algorithm. Synchronization error detection formula:

[0236] Let the current timestamp be Tc , the expected timestamp is T d , the synchronization error is E s :

[0237] E s =T c -T d (18)

[0238] The control parameters are adjusted using the synchronization error correction algorithm to ensure that the synchronization error is within an acceptable range.

[0239] Furthermore, a system monitoring module is designed to monitor data transmission and synchronization status in real time, and handle abnormalities in a timely manner. The fault handling process is as follows:

[0240] Assume the fault detection time is T f , the fault handling time is T h , the system recovery time is T r :

[0241] T r =T f +T h (19)

[0242] Different recovery operations are performed according to the fault type, such as retransmitting data packets, resynchronizing timestamps, etc.

[0243] S4. Create a virtual 3D interactive space based on the paper-screen synchronization algorithm and design the integration and synchronization control of holographic projection.

[0244] In this embodiment, a holographic projection device is installed in the classroom to create a virtual 3D interactive space. The paper note content is displayed on the virtual screen in real time through holographic projection technology for students and teachers to view and interact.

[0245] Specifically, first of all, the selection and installation of holographic projection equipment: choose high-resolution holographic projection equipment to ensure that its resolution and refresh rate can meet the needs of classroom teaching. The resolution must be at least 1080p to ensure the clarity of the projected content. The refresh rate must be at least 60Hz to ensure the smoothness of the projection.

[0246] Preferably, the holographic projection equipment should be installed in the front of the classroom or other appropriate location to ensure that all students can clearly see the projected content. The specific location should be adjusted according to the classroom layout to avoid blocking the projected content. Adjust the projection area according to the size of the classroom to ensure that the projected content is clearly visible and covers the entire classroom.

[0247] It is understandable that holographic image generation is based on the principles of physical optics to ensure the depth and clarity of the image. Among them, the light wave function is generated:

[0248] U(x,y)=A(x,y)exp(iφ(x,y)) (20)

[0249] Among them, A(x,y) is the amplitude of the light wave, and φ(x,y) is the phase of the light wave.

[0250] Interference pattern recording:

[0251] I(x,y)=|U r (x,y)+U o (x,y)| 2 (twenty one)

[0252] Among them, U r (x,y) is the reference light wave, U o (x,y) is the object light wave.

[0253] Holographic image reconstruction:

[0254]

[0255] in, represents Fourier transform.

[0256] Correspondingly, the fused multimodal feature vector F is converted into the input parameter of the holographic image, and the corresponding image and text content are generated according to the feature vector. The image processing includes image enhancement and text rendering, including:

[0257] Image enhancement: Increase image contrast using histogram equalization.

[0258]

[0259] Among them, I′(x,y) is the enhanced image, I(x,y) is the original image, and I min and I max are the minimum and maximum values ​​of the image respectively.

[0260] Text rendering: Generate corresponding text information based on the content of the feature vector and render it onto the image.

[0261] Furthermore, the projection content is adjusted in real time according to the teacher's explanation and the students' notes to ensure that the projection content is synchronized with the teaching content. The projection content is adjusted using the PID control algorithm to ensure that the projection content is synchronized with the actual teaching content.

[0262] e(t)=r(t)-y(t) (24)

[0263]

[0264] Among them, e(t) is the error, r(t) is the reference input (expected content), y(t) is the actual output (current projection content), u(t) is the control amount, K p ,K i ,K d are the proportional, integral and differential coefficients respectively.

[0265] Realize two-way synchronization between paper note content and holographic projection content, ensuring that students' writing on paper notes can be reflected in real time on the holographic projection, and vice versa.

[0266] Paper note collection is to use a smart pen to collect students' written content and send it to the holographic projection system in real time via wireless transmission.

[0267] Content synchronization is to convert the collected note content into holographic projection input parameters and update the projection content in real time. The teacher's explanation content and other auxiliary information are displayed through the holographic projection device, and students can record and mark on paper:

[0268] Sync(t)=αPen(t)+βProject(t) (26)

[0269] Among them, Sync(t) is the synchronization content, Pen(t) is the paper note content, Project(t) is the projection content, and α and β are synchronization weight coefficients.

[0270] It is understandable that real-time feedback can be provided through holographic projection equipment to enhance the interaction between teachers and students. Teachers can view the content of students' notes in real time through holographic projection equipment, and make annotations and instructions. Students can view teachers' annotations and explanations in real time through holographic projection equipment to enhance learning effects.

[0271] S5. Construct an intelligent context prediction model based on a machine learning algorithm, which dynamically predicts students' learning contexts based on multimodal data and historical data, and generates personalized teaching content and suggestions.

[0272] In this embodiment, based on multimodal data and historical data, machine learning algorithms (such as LSTM and Transformer) are used to develop an intelligent context prediction module. It can dynamically predict students’ learning contexts and generate personalized teaching content and suggestions.

[0273] The step S5 comprises the following steps:

[0274] S1. Multimodal data preprocessing:

[0275] Standardize the multimodal data (visual, speech, and tactile) so that they can be analyzed in the same dimension. Let the original data be X and the standardized data be X′:

[0276]

[0277] Among them, μ is the mean of the data and σ is the standard deviation of the data.

[0278] S2. Time series feature extraction:

[0279] The multimodal data is segmented according to a fixed time window, and each segment is processed as a time series data block. Assuming the time window length is T, the data segmentation formula is:

[0280] D k ={X t |kT≤t<(k+1)T} (28)

[0281] Among them, D k is the data in the kth time window, X t is the data at time t.

[0282] S3, timing pattern recognition:

[0283] Use the time series pattern recognition model to identify time series patterns and extract key features. Calculate the autocorrelation function of the data and identify periodic features in the data:

[0284]

[0285] Where R(τ) is the autocorrelation function of lag τ, N is the data length, μ represents the data mean, and X t+τ Represents data at a time lag of τ.

[0286] Convert the time domain data to the frequency domain and identify the main frequency components:

[0287]

[0288] Among them, F(f) is the Fourier transform result of frequency f, e -i2πft Represents the complex exponential function, which is the basis function in Fourier transform.

[0289] S4. Feature fusion and dimensionality reduction:

[0290] The features extracted by temporal pattern recognition are fused with the original multimodal features to form a comprehensive feature vector. Let the temporal feature be F T , the multimodal feature is F M , the fused feature vector is F:

[0291] F=[F T ,F M ] (31)

[0292] Use principal component analysis (PCA) to reduce the dimension of the fused feature vector to reduce the computational complexity. PCA dimension reduction formula:

[0293] X new =XW (32)

[0294] Among them, X is the fusion feature matrix, W is the projection matrix, X new is the feature matrix after dimensionality reduction.

[0295] S5. Intelligent situation prediction:

[0296] Based on the fusion feature vector, the context prediction model is constructed to predict the student's learning context and provide personalized teaching suggestions, including:

[0297] State space model: Establish a state space model to describe the dynamic changes of the learning situation. State transition equation:

[0298] x t+1 =Ax t +Bu t +w t (33)

[0299] Among them, x t is the state vector, A is the state transfer matrix, B is the control matrix, u t is the control vector, w t is the process noise.

[0300] Observation equation:

[0301] y t =Cx t +v t (34)

[0302] Among them, y t is the observation vector, C is the observation matrix, v t is the observation noise.

[0303] Use Kalman filter to predict the situation and update the state estimate. Prediction steps:

[0304]

[0305] P t|t-1 =AP t-1|t-1 A T +Q (36)

[0306] in, represents the predicted state, P t|t-1 represents the prediction covariance, Q represents the process noise covariance matrix, represents the predicted state at the previous moment, P t-1|t-1represents the prediction covariance of the previous moment, and T represents the transpose.

[0307] Update steps:

[0308] K t =P t|t-1 C T (CP t|t-1 C T +R) -1 (37)

[0309]

[0310] P t|t =(IK t C)P t|t-1 (39)

[0311] Among them, K t represents the Kalman gain, R represents the observation noise covariance matrix, I represents the identity matrix, represents the updated prediction state, P t|t represents the updated forecast covariance.

[0312] S6. Generation of personalized teaching suggestions:

[0313] Generate personalized teaching suggestions based on the situation prediction results and provide them to teachers and students. Establish a situation-based rule library to match the corresponding teaching strategies and resources. Generate personalized suggestions based on the situation prediction results and rule library, and display them to teachers and students through the system interface.

[0314] S6. Design an interactive feedback mechanism, based on which the teacher annotates and explains the students' notes through holographic projection.

[0315] In this embodiment, an interactive feedback mechanism is designed to allow teachers to annotate and explain students' notes through holographic projection. Students can also interact with the system through smart pens or touch screens to obtain real-time feedback and suggestions.

[0316] Specifically, it integrates visual, voice and tactile perception devices to collect multimodal data in the classroom in real time to ensure comprehensive coverage and high quality of the data.

[0317] Visual data is captured in real time using high-resolution cameras to capture students’ notes and classroom environment.

[0318] Voice data is collected in real time by using microphone arrays to collect voice information of teachers and students.

[0319] Tactile data records students’ writing behavior and pressure changes in real time using a smart pen or touch screen.

[0320] Furthermore, the Network Time Protocol (NTP) is used to synchronize the collected multimodal data to ensure the consistency of different data sources.

[0321] Assume that the timestamps of different data sources are T v (Visual), T s (Voice) t (tactile), the synchronized timestamp is T s :

[0322]

[0323] It is understandable that personalized feedback information is generated based on the multimodal data collected in real time to ensure the timeliness and pertinence of the feedback.

[0324] The visual feedback provides real-time optimization suggestions based on the content and structure of the student's notes. Use OCR technology to recognize the text and formulas in the student's notes. Generate evaluation indicators based on the completeness and logical structure of the notes:

[0325] S note =α1P+α2L+α3F (41)

[0326] Among them, S note Score the notes, P for completeness, L for logic, F for formula correctness, α1, α2, α3 for weight coefficients.

[0327] The voice feedback provides evaluation of teaching quality and interaction by analyzing the voice information in the classroom in real time. The voice information in the classroom is transcribed using voice recognition technology. Evaluation indicators are generated based on the clarity, speed and interaction frequency of the voice:

[0328] S voice =β1q+β2r+β3i (42)

[0329] Among them, S voice is the speech score, q is the clarity, r is the speaking rate, i is the interaction frequency, and β1, β2, and β3 are weight coefficients.

[0330] The tactile feedback provides optimization suggestions for writing posture and pressure by analyzing the student's writing behavior. Through sensor data, the student's writing posture is identified and whether there is a bad posture. Based on the pressure data recorded by the smart pen, the uniformity and stability of the writing pressure are evaluated:

[0331] S pressure =γ1μ P +γ2σ P (43)

[0332] Among them, S pressureis the stress score, μ P is the mean pressure, σ P is the pressure standard deviation, γ1, γ2 are weight coefficients.

[0333] Preferably, feedback information is displayed in real time through holographic projection or electronic screen for teachers and students to view, ensuring that the feedback information is intuitive and easy to understand, including:

[0334] Visual feedback display: Note scores and optimization suggestions are projected onto electronic screens for students to view in real time.

[0335] Voice feedback display: Voice scores and interaction situations are displayed through holographic projection, so teachers can adjust teaching strategies in real time.

[0336] Tactile feedback display: The writing posture and pressure analysis results are displayed on the electronic screen, and students can adjust the writing posture and strength in real time.

[0337] Preferably, the feedback mechanism is continuously optimized based on actual usage and user feedback to ensure the accuracy and practicality of the feedback information.

[0338] The user feedback collection is to collect feedback from teachers and students through the system interface.

[0339] The feedback mechanism improvement is to adjust the feedback algorithm and the weight coefficients of the evaluation indicators according to user feedback to ensure that the feedback information better meets the actual needs.

[0340] S7. Encrypt data transmission using encryption technology, set up identity authentication access and alarm devices to ensure the security of data transmission, and design a fault recovery mechanism to automatically perform fault switching and recovery when a fault occurs.

[0341] In this embodiment, the system architecture is optimized, the stability of data transmission is enhanced, and advanced encryption technology is used to ensure data security. The system should have high fault tolerance and be able to operate stably in a complex classroom environment.

[0342] Specifically, in order to ensure the stability of classroom data during transmission, a hybrid transmission protocol is used, combining the advantages of UDP and TCP to ensure low latency and high reliability of data transmission. For critical data (such as notes and teaching instructions), the TCP protocol is used to ensure reliable data transmission. For data with high real-time requirements (such as voice and video streams), the UDP protocol is used to reduce transmission delays.

[0343] In the classroom network environment, an adaptive optimal path algorithm is used to dynamically select data transmission paths, reduce network congestion, increase data transmission speed, and ensure real-time synchronization of classroom data. The optimal path selection formula is:

[0344]

[0345] Among them, d i is the distance of the path segment, l j is the delay of the path segment, P opt is the optimal path, m represents the number of factors representing the path segment delay, n represents the number of path segments, Denotes the selection of a path P that minimizes the total path distance and delay.

[0346] Furthermore, redundant data storage is set up at key nodes of the system (such as data acquisition terminals and synchronous control centers) to ensure that data will not be lost when equipment fails. Distributed storage systems (such as RAID technology) are used to provide redundant storage and verification of data. Among them, RAID1 (mirror) structure:

[0347] Data stored =Data original ∪Data mirrored (45)

[0348] Among them, Data stored To store data, Data original is the original data, Data mirrored The mirror data.

[0349] At the same time, set up a scheduled backup mechanism to regularly back up data to cloud storage to prevent data loss. Set the backup interval time T b , backup data volume D b for:

[0350]

[0351] Among them, D total is the total amount of data, T total For total time.

[0352] Furthermore, during the data transmission process, encryption algorithms are used to encrypt data to prevent data theft or tampering, ensuring the privacy and security of classroom data. AES (Advanced Encryption Standard) is used for data encryption with a key length of 256 bits to ensure the security of data transmission. AES encryption formula:

[0353] C k =E k (P) (47)

[0354] Among them, C k is the ciphertext, P is the plaintext, E k is the encryption function and k is the key.

[0355] At the same time, a strict identity authentication mechanism is set up to ensure that only authorized users can access the system and data. Use two-factor authentication (2FA), combining passwords and dynamic verification codes, to enhance system security. Among them, the identity authentication process:

[0356] The user enters a username and password.

[0357] The system generates a dynamic verification code and sends it to the user's device.

[0358] The user enters the dynamic verification code, and the system grants access rights after verification.

[0359] Furthermore, a system monitoring module is set up to monitor the system's operating status and data transmission in real time to ensure stable operation of equipment and networks in the classroom environment. Monitoring indicators include CPU usage, memory usage, network latency, etc. Monitoring formula:

[0360] M(t)={CPU(t),Memory(t),Latency(t)} (48)

[0361] Among them, M(t) is the monitoring indicator, CPU(t) is the CPU usage, Memory(t) is the memory usage, and Latency(t) is the network delay.

[0362] When the system is abnormal, an alarm notification is issued in time to ensure that the problem can be handled in time to avoid affecting classroom teaching. Set the alarm threshold, and trigger the alarm when the monitoring indicator exceeds the threshold. Alarm formula:

[0363]

[0364] Among them, A is the alarm signal, T alarm is the alarm threshold.

[0365] It is understandable that the present invention sets an automatic fault recovery mechanism, which can automatically perform fault switching and recovery when the system fails, ensuring the continuity of classroom teaching. Using hot backup technology, when the main system fails, it automatically switches to the backup system to ensure the continuous operation of the system. Fault switching formula:

[0366]

[0367] Among them, S active is the currently active system, S main As the main system, S backup For backup system.

[0368] When the system resumes normal operation, the data during the failure period is restored from the backup data to ensure data integrity. Data recovery formula:

[0369] D recovered =D backup +D new (51)

[0370] Among them, D recovered To recover data, D backup To back up data, D new It is the new data generated during the failure period.

[0371] S8. At the same time, design a friendly user interface and interaction mechanism to optimize the system based on user feedback.

[0372] In this embodiment, a user-friendly interface and interactive mode are designed to provide intuitive operation guidance and help. The system should be able to continuously optimize according to user feedback to improve the overall user experience.

[0373] Further explanation, user needs analysis:

[0374] Teachers' needs: Convenient classroom content management: including uploading, editing, and displaying courseware; real-time interactive functions: such as asking questions, answering questions, and real-time feedback; post-class data analysis: student performance, class participation, etc.

[0375] Student needs: Easy-to-use learning interface: such as note-taking, courseware viewing, real-time questioning, etc.; Timely feedback: see the teacher's feedback and comments in real time; Learning data tracking: be able to view your own learning progress and performance analysis.

[0376] It is understandable that the interface design principles are: the interface layout is simple, the information is clearly displayed, and redundancy and complex operations are avoided. Maintain the consistency of interface elements, such as fonts, colors, button styles, etc., to enhance the user experience. After the user operates, the system can give timely feedback, such as successful operation prompts, error prompts, etc. Provide operation instructions and help documents to help users quickly get started with the system.

[0377] Teacher interface:

[0378] Main interface: Contains modules such as course list, courseware management, real-time classroom monitoring, and student feedback.

[0379] Interactive interface: Students’ questions and answers are displayed in real time, and teachers can interact and provide feedback instantly.

[0380] Data analysis interface: displays students’ learning data and classroom performance, and provides visual analysis charts.

[0381] Student interface:

[0382] Main interface: Contains modules such as course list, note taking, courseware viewing, question interaction, etc.

[0383] Note-taking interface: Students can record notes in electronic notebooks, which will be automatically saved and synchronized by the system.

[0384] Feedback interface: displays the teacher's comments and feedback in real time, and students can view and respond instantly.

[0385] Furthermore, interaction design:

[0386] Drag-and-drop operation: supports teachers to upload and manage courseware by dragging and dropping, simplifying the operation process.

[0387] Handwritten notes: Support students to use a stylus to take notes on electronic devices, and the system can automatically recognize and save them.

[0388] Real-time synchronization: The teacher and student interfaces are synchronized in real time, and the teacher's explanations and comments can be displayed instantly on the student side.

[0389] Shortcut key operation: Provide shortcut key function to improve operation efficiency, such as starting, pausing, and ending a class.

[0390] Furthermore, it includes user guidance and help:

[0391] Newbie guide: When using the system for the first time, a newbie guide is provided to help users quickly understand system functions and operation methods.

[0392] Operation Tutorial: Provide detailed operation tutorials and video demonstrations, which users can view and learn at any time.

[0393] Online Help: The system has built-in online help documents and FAQs, so users can quickly find solutions when they encounter problems.

[0394] Further, provide user feedback and optimization:

[0395] Feedback collection: Provide feedback channels where users can submit their usage experience and suggestions.

[0396] Regular updates: Based on user feedback and usage data, we regularly update and optimize the system to enhance user experience.

[0397] A / B testing: Conduct A / B testing to verify the effectiveness of new functions and interface designs, and continuously optimize the user experience.

[0398] Furthermore, the interface design should be combined with the characteristics of the smart classroom paper-screen synchronization control system to ensure that the unique functions of the system (such as paper-screen synchronization, real-time feedback, etc.) are fully displayed and applied in the interface. Support users to personalize the interface, such as changing the theme color, adjusting the layout, etc., to improve user comfort. Ensure the compatibility of the system on different devices, such as PC, tablet, smart pen, etc., so that users can seamlessly switch between different devices.

[0399] In one or more embodiments, Figure 2 As shown, a smart classroom paper-screen synchronization control system is disclosed, and the system includes:

[0400] The sensor deployment unit 101 is used to arrange the perception units in the classroom, collect student data in real time, pre-process the student data, and perform data fusion on the pre-processed data to obtain multi-modal data; the perception units include a visual perception unit, a speech perception unit, and a tactile perception unit;

[0401] The data processing unit 102 is used to perform format standardization processing on the multimodal data, then extract key features from the standardized multimodal data, and perform feature fusion and dimensionality reduction processing on the key features;

[0402] The data integration unit 103 is used to design a paper-screen synchronization algorithm for updating the paper note content on the electronic screen; based on the paper-screen synchronization algorithm, a virtual 3D interactive space is created, and the integration and synchronization control of the holographic projection are designed;

[0403] A scenario prediction unit 104 is used to construct an intelligent scenario prediction model based on a machine learning algorithm, wherein the intelligent scenario prediction model dynamically predicts the student's learning scenario based on multimodal data and historical data, and generates personalized teaching content and suggestions;

[0404] An interactive feedback unit 105 is used to design an interactive feedback mechanism, according to which the teacher annotates and explains the student's notes through holographic projection;

[0405] The data transmission encryption unit 106 is used to encrypt the data transmission using encryption technology, and set identity authentication access and alarm devices to ensure the security of data transmission, and design a fault recovery mechanism to automatically perform fault switching and recovery when a fault occurs;

[0406] A system updating unit 107, for simultaneously designing a user-friendly interface and an interactive mechanism, for optimizing the system according to user feedback;

[0407] Wherein, the intelligent situation prediction model includes:

[0408] Establish a state space model and construct the state transfer equation, which is expressed as follows:

[0409] x t+1 =Ax t +Bu t +w t

[0410] Among them, x t+1 represents the updated state vector, x trepresents the state vector, A represents the state transfer matrix, B represents the control matrix, u t represents the control vector, w t represents process noise;

[0411] The observation equation is expressed as follows:

[0412] y t =Cx t +v t

[0413] Among them, y t represents the observation vector, C represents the observation matrix, v t represents the observation noise;

[0414] Use the Kalman filter to predict the situation and update the state estimate, which is expressed as follows:

[0415]

[0416] P t|t-1 =AP t-1|t-1 A T +Q

[0417] in, represents the predicted state, P t|t-1 represents the prediction covariance, Q represents the process noise covariance matrix, represents the predicted state at the previous moment, P t-1|t-1 represents the prediction covariance of the previous moment, and T represents transposition;

[0418] Updated as follows:

[0419] K t =P t|t-1 C T (CP t|t-1 C T +R) -1

[0420]

[0421] P t|t =(IK t C)P t|t-1

[0422] Among them, K t represents the Kalman gain, R represents the observation noise covariance matrix, I represents the identity matrix, represents the updated prediction state, P t|t represents the updated forecast covariance.

[0423] It is worth noting that the specific working process of a smart classroom paper screen synchronization control system provided by an embodiment of the present invention is the same as the process of a smart classroom paper screen synchronization control method described in the above embodiment, which will not be repeated here.

[0424] Compared with the prior art, the embodiment of the present invention provides a smart classroom paper-screen synchronization control system, which arranges a perception unit in the classroom, collects student data in real time, pre-processes the student data, and fuses the pre-processed data to obtain multimodal data; the perception unit includes a visual perception unit, a voice perception unit, and a tactile perception unit; the multimodal data is format-standardized, and then key features are extracted from the standardized multimodal data, and feature fusion and dimensionality reduction are performed on the key features; a paper-screen synchronization algorithm is designed to update the content of paper notes to the electronic screen; according to the paper-screen synchronization algorithm, a virtual 3D interactive space is created, and a holographic projection is designed integration and synchronous control; construct an intelligent situation prediction model based on machine learning algorithm, which dynamically predicts students' learning situations according to multimodal data and historical data, and generates personalized teaching content and suggestions; design an interactive feedback mechanism, according to which teachers annotate and explain students' notes through holographic projection; use encryption technology to encrypt data transmission, set up identity authentication access and alarm devices to ensure the security of data transmission, and design a fault recovery mechanism to automatically perform fault switching and recovery when a fault occurs; and design a friendly user interface and interaction mechanism to optimize the system according to user feedback.

[0425] An embodiment of the present invention further provides a smart classroom paper screen synchronization control device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the steps in the above-mentioned smart classroom paper screen synchronization control method embodiment are implemented, for example Figure 1 or, the processor implements the functions of each module in the above-mentioned system embodiments when executing the computer program.

[0426] Exemplarily, the computer program may be divided into one or more modules, and the one or more modules are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the smart classroom paper screen synchronization control device.

[0427] The intelligent classroom paper screen synchronization control device can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The intelligent classroom paper screen synchronization control device can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the intelligent classroom paper screen synchronization control device can also include input and output devices, network access devices, buses, etc.

[0428] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the intelligent classroom paper screen synchronization control device, and uses various interfaces and lines to connect the various parts of the entire intelligent classroom paper screen synchronization control device.

[0429] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the intelligent classroom paper screen synchronization control device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc.; the data storage area can store data created according to the operation of the air conditioning controller, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMedaaCard, SMC), a secure digital (SecureDagatal, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0430] Wherein, if the module integrated in the paper screen synchronization control device of the intelligent classroom is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0431] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0432] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for synchronous control of paper screen in smart classroom, characterized in that: The method comprises: Arrange a perception unit in the classroom to collect student data in real time, pre-process the student data, and perform data fusion on the pre-processed data to obtain multimodal data; the perception unit includes a visual perception unit, a speech perception unit, and a tactile perception unit; Standardize the format of multimodal data, extract key features from the standardized multimodal data, and perform feature fusion and dimensionality reduction on the key features; Design a paper-screen synchronization algorithm to update paper note content to the electronic screen; Based on the paper-screen synchronization algorithm, a virtual 3D interactive space is created, and the integration and synchronization control of holographic projection is designed; Building an intelligent context prediction model based on machine learning algorithms, which dynamically predicts students' learning contexts based on multimodal data and historical data, and generates personalized teaching content and suggestions; Design an interactive feedback mechanism, based on which the teacher annotates and explains the students' notes through holographic projection; Encryption technology is used to encrypt data transmission, and identity authentication access and alarm devices are set up to ensure the security of data transmission. Fault recovery mechanisms are designed to automatically perform fault switching and recovery when a fault occurs; At the same time, a friendly user interface and interactive mechanism are designed to optimize the system based on user feedback; Wherein, the intelligent situation prediction model includes: Establish a state space model and construct the state transfer equation, which is expressed as follows: x t+1 =Ax t +Bu t +w t Among them, x t+1 represents the updated state vector, x t represents the state vector, A represents the state transfer matrix, B represents the control matrix, u t represents the control vector, w t represents process noise; The observation equation is expressed as follows: y t =Cx t +v t Among them, y t represents the observation vector, C represents the observation matrix, v t represents the observation noise; Use the Kalman filter to predict the situation and update the state estimate, which is expressed as follows: P t|t-1 =AP t-1|t-1 From T +Q in, represents the predicted state, P t|t-1 represents the prediction covariance, Q represents the process noise covariance matrix, represents the predicted state at the previous moment, P t-1|t-1 represents the prediction covariance of the previous moment, and T represents transposition; Updated as follows: K t =P t|t-1 CT(CP t|t-1 C T +R) -1 P t|t =(I-K t C)P t|t-1 Among them, K t represents the Kalman gain, R represents the observation noise covariance matrix, I represents the identity matrix, represents the updated prediction state, P t|t represents the updated forecast covariance.

2. The method according to claim 1, characterized in that A visual perception unit is installed in the classroom to collect visual data and pre-process the visual data; a voice perception unit is installed in the classroom to collect voice data and pre-process the voice data; a tactile perception unit is installed on the student side to collect tactile data and pre-process the tactile data; the pre-processed visual data, voice data and tactile data are fused to obtain fused data; Wherein, the preprocessing of visual data includes image correction, image segmentation and image enhancement; The preprocessing of the speech data includes noise reduction and enhancement; The preprocessing of the tactile data includes pressure calibration and writing speed calculation; The fused data obtained by performing data fusion is expressed as follows: D fusion (t)=αI(t)+βS(t)+γP(t) Among them, D fusion (t) represents the fused data, I(t) represents the image data, S(t) represents the speech data, P(t) represents the pressure data, and α, β, and γ represent the fusion weight coefficients.

3. The method according to claim 2, characterized in that The format standardization process includes formatting and time aligning the visual data, the voice data, and the tactile data; The visual data is represented as a two-dimensional matrix I ij (t), where i, j represent pixel positions and t represents the time frame; Speech data is represented as a one-dimensional vector S k (t), where k represents the sampling point and t represents the time frame; Tactile data is represented as a one-dimensional vector P m (t), where m represents the pressure sensor reading and t represents the time frame; The key features extracted from the standardized multimodal data include: Extracting the outline of the handwriting from the visual data using edge detection and morphological transformation, and extracting straight line features as image features using Hough transformation; Extracting spectral features from the speech data using short-time Fourier transform, and then extracting Mel-frequency cepstral coefficients as speech features; From the tactile data, calculating the mean and standard deviation of the pressure signal, and the mean and standard deviation of the writing speed as tactile features; The feature fusion and dimensionality reduction processing of the key features includes: Let the image feature be F I , the speech feature is F S , the tactile characteristic is F P , then the fused multimodal feature vector F is expressed as: F c =[F I ,F S ,F P ] Among them, F I , F s , F P Represent the feature sub-vectors extracted from image, speech and tactile data respectively; Assume that the fused feature vector matrix is ​​X and its covariance matrix is ​​R: Among them, μ X represents the mean of the eigenvector; Calculate the eigenvalues ​​and eigenvectors of R, and select the eigenvectors corresponding to the first k largest eigenvalues ​​to form the projection matrix W: W=[w1,w2,...,w k ] The feature representation after dimensionality reduction is: F PCA =XW Among them, F PCA Represents the feature representation after dimensionality reduction.

4. The method according to claim 1, characterized in that: The paper-screen synchronization algorithm includes: Determine the data transmission path and nodes, and use the network time protocol to synchronize the time of each node to ensure clock consistency; Design data transmission protocols to ensure low-latency and real-time data transmission between nodes, and use the User Datagram Protocol for data transmission to reduce transmission delays; A ring buffer is designed for data caching, and data enters the buffer in FIFO mode; wherein the management algorithm of the ring buffer is as follows: Buffer[i]=Data if 0≤i <L Where Buffer[i] represents the data at the i-th buffer position; A synchronization control algorithm based on a physical model is designed, and a proportional-integral-differential control algorithm is used to adjust data synchronization; the proportional-integral-differential control algorithm is expressed as follows: Among them, u(t) represents the control amount, e(t) represents the error, K p , K i , K d They represent proportional, integral and differential coefficients respectively, dt represents the derivative, and de(t) represents the differential of the error e(t); Perform timestamp correction on the received data packets to ensure data synchronization, and display the corrected data in timestamp order to ensure real-time synchronization between paper notes and electronic screen content; The synchronization error is detected regularly and adjusted using a synchronization error correction algorithm; wherein the synchronization error correction algorithm is expressed as follows: E s =T c -T d Among them, T c Indicates the current timestamp, T d Indicates the expected timestamp, E s Indicates synchronization error; Monitor data transmission and synchronization status in real time and handle abnormalities promptly.

5. The method according to claim 3, characterized in that: The integration and synchronous control of the holographic projection includes Select high-resolution holographic projection equipment to be installed in the classroom, and generate holographic images based on the principles of physical optics; The fused multimodal feature vector F is converted into the input parameter of the holographic image, and the corresponding image and text content are generated according to the feature vector. Then, the image contrast is improved by using histogram equalization, and the text content is generated according to the content of the feature vector and rendered on the image. The projection content is adjusted in real time according to the teacher's explanation and the students' notes, and the proportional integral differential control algorithm is used to adjust the projection content, which is expressed as follows: e(t)=r(t)-y(t) Among them, e(t) represents the error, r(t) represents the reference input, y(t) represents the actual output, u(t) represents the control amount, K p , K i , K d denote the proportional, integral and differential coefficients respectively; Synchronize paper notes with holographic projection content, and provide real-time feedback through the holographic projection device.

6. The method according to claim 1, characterized in that The intelligent context prediction model dynamically predicts students’ learning contexts based on multimodal data and historical data, and generates personalized teaching content and suggestions, including: The multimodal data is standardized so that it can be analyzed under the same dimension. Then the multimodal data is segmented according to a fixed time window, and each segment is processed as a time series data block, which is expressed as follows: D k ={X t |kT≤t<(k+1)T} Among them, D k represents the data in the kth time window, X t represents the data at time t; Use the time series pattern recognition model to identify time series patterns and extract key features, which are expressed as follows: Where R(τ) represents the autocorrelation function of lag τ, N represents the data length, μ represents the data mean, and X t+τ represents the data mean; The time domain data is then converted to frequency domain data and the frequency components are identified, which is represented as follows: Among them, F(f) represents the Fourier transform result of frequency f, e -i2πft represents the complex exponential function, which is the basis function in Fourier transform; The features extracted by temporal pattern recognition are fused with the original multimodal features to form a comprehensive feature vector; The comprehensive feature vector is input into the intelligent situation prediction model, and a situation-based rule library is established. According to the situation prediction results, personalized teaching suggestions are generated and provided to teachers and students.

7. The method according to claim 1, characterized in that The interactive feedback includes visual feedback, voice feedback and tactile feedback.

8. The method according to claim 7, characterized in that The interactive feedback mechanism generates personalized feedback information based on the multimodal data collected in real time, including: The visual feedback provides real-time suggestions for improving students’ notes based on the content and structure of their notes, including: Use OCR technology to recognize text and formulas in student notes and generate note scores, as shown below: S note =α1P+α2L+α3F Among them, S note represents the note score, P represents completeness, L represents logic, F represents formula correctness, α1, α2, α3 represent weight coefficients; The voice feedback uses voice recognition technology to transcribe the voice information in the classroom and generates a note score based on the voice information, which is expressed as follows: S voice =β1q+β2r+β3i Among them, S voice represents speech score, q represents clarity, r represents speech rate, i represents interaction frequency, β1, β2, β3 represent weight coefficients; The tactile feedback analyzes the student's writing behavior through sensor data, identifies the student's writing posture, and generates a pressure evaluation index, which is expressed as follows: S pressure =γ1μ P +γ2σ P Among them, S pressure represents the stress score, μ P represents the mean pressure, σ P represents the pressure standard deviation, γ1 and γ2 represent the weight coefficients.

9. The method according to claim 1, characterized in that: The method of using encryption technology to encrypt data transmission includes: In the classroom network environment, an adaptive optimal path algorithm is used to dynamically select the data transmission path, as shown below: Among them, d i represents the distance of the path segment, l j represents the delay of the path segment, P opt represents the optimal path, m represents the number of factors of path segment delay, n represents the number of path segments, It indicates that the path P is selected so that the total distance and delay of the path are minimized; Use distributed storage systems at key nodes to provide redundant storage and verification of data, and set up a scheduled backup mechanism to regularly back up data to cloud storage; During data transmission, the data is encrypted using an encryption algorithm, as shown below: C k =E k (P) Among them, C k represents ciphertext, P represents plaintext, E k represents the encryption function, and k represents the key.

10. A smart classroom paper screen synchronization control system, characterized in that: The system comprises: A sensor deployment unit is used to arrange perception units in the classroom, collect student data in real time, pre-process the student data, and perform data fusion on the pre-processed data to obtain multimodal data; the perception units include a visual perception unit, a speech perception unit, and a tactile perception unit; A data processing unit is used to perform format standardization processing on the multimodal data, then extract key features from the standardized multimodal data, and perform feature fusion and dimensionality reduction processing on the key features; The data integration unit is used to design a paper-screen synchronization algorithm to update the paper note content to the electronic screen; based on the paper-screen synchronization algorithm, a virtual 3D interactive space is created, and the integration and synchronization control of the holographic projection is designed; A scenario prediction unit, which is used to build an intelligent scenario prediction model based on a machine learning algorithm. The intelligent scenario prediction model dynamically predicts students' learning scenarios based on multimodal data and historical data, and generates personalized teaching content and suggestions; An interactive feedback unit, used to design an interactive feedback mechanism, according to which the teacher annotates and explains the student's notes through holographic projection; Data transmission encryption unit, used to encrypt data transmission using encryption technology, and set identity authentication access and alarm devices to ensure the security of data transmission, and design a fault recovery mechanism to automatically perform fault switching and recovery when a fault occurs; System update unit, which is responsible for designing user-friendly interfaces and interactive mechanisms to optimize the system based on user feedback; Wherein, the intelligent situation prediction model includes: Establish a state space model and construct the state transfer equation, which is expressed as follows: x t+1 =Ax t +Bu t +w t Among them, x t+1 represents the updated state vector, x t represents the state vector, A represents the state transfer matrix, B represents the control matrix, u t represents the control vector, w t represents process noise; The observation equation is expressed as follows: y t =Cx t +v t Among them, y t represents the observation vector, C represents the observation matrix, v t represents the observation noise; Use the Kalman filter to predict the situation and update the state estimate, which is expressed as follows: P t|t-1 =AP t-1|t-1 From T +Q in, represents the predicted state, P t|t-1 represents the prediction covariance, Q represents the process noise covariance matrix, represents the predicted state at the previous moment, P t-1|t-1 represents the prediction covariance of the previous moment, and T represents transposition; Updated as follows: K t =P t|t-1 CT(CP t|t-1 C T +R) -1 P t|t =(I-K t C)P t|t-1 Among them, K t represents the Kalman gain, R represents the observation noise covariance matrix, I represents the identity matrix, represents the updated prediction state, P t|t represents the updated forecast covariance.

Images