Intelligent teaching method and system based on artificial intelligence and intelligent dot matrix pen
Through intelligent teaching methods based on artificial intelligence and intelligent dot matrix pen, the problems of low efficiency and difficulty in evaluation in existing educational technologies are solved, efficient teaching experience and personalized learning suggestions are achieved, and intelligent interaction and blockchain are used to achieve intelligent teaching and experience. Certification improves the quality and coverage of education.
Patent Information
- Application Number
- CN202411199149.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing educational technologies are difficult to achieve efficient interactive teaching, cannot meet the needs of quality education and information education, and cannot effectively evaluate students' learning effects. There are health and control problems with the use of electronic equipment, which limits teaching efficiency and user experience.
Adopt intelligent teaching methods based on artificial intelligence and intelligent dot matrix pens, collect writing behavior data, use deep learning algorithms and machine learning algorithms to decode and analyze data, generate personalized courseware, realize online immersive interaction between teachers and students, and Credit certification is realized through blockchain.
Improve teaching efficiency and experience, realize the automated generation of teaching content and interactive courseware, reduce the work burden of teachers, be able to comprehensively evaluate students' learning situation, enhance students' sense of participation and learning interest, and provide personalized learning suggestions and credits Certification.
Smart Images

Figure CN119006243A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent teaching method and system based on artificial intelligence and an intelligent dot matrix pen. Background Art
[0002] The combination of information technology and education is the development direction of modern education. At present, in classroom teaching, teaching equipment such as multimedia large screens, computers and projectors are usually connected to each other for teaching. Teachers use computers to edit and upload courseware before class, and use multimedia large screens or computers and projectors for teaching. The teaching is teacher-led, and there is a lack of interaction between teachers and students in class. The class is not interesting and the efficiency of interactive teaching is not high. This one-way teaching model is increasingly difficult to adapt to today's requirements for quality education and information-based education, and it is impossible to intuitively and quickly evaluate students' learning effects.
[0003] Currently, there are also solutions on the market that enable in-class interaction and after-class exercises by equipping students with tablets and styluses or other similar terminal devices. However, this solution has problems with eye health, control of electronic device use, and prevention of addiction. Schools also require the use of paper assignments in principle and control of the use of electronic devices in teaching, which greatly limits teaching efficiency and poor user experience. Summary of the invention
[0004] The present invention is based on the above-mentioned problems and proposes a smart teaching method and system based on artificial intelligence and a smart dot pen. The personalized smart teaching method based on data collection and deep learning algorithm of the smart dot pen provided by the solution of the present invention has better intelligence, improves teaching efficiency, and can bring a better teaching experience.
[0005] In view of this, one aspect of the present invention proposes a smart teaching method based on artificial intelligence and a smart dot pen, comprising:
[0006] When the smart dot matrix pen writes on the dot matrix code paper, the writing behavior data is collected;
[0007] Based on the deep learning algorithm, the handwriting image in the collected writing behavior data is used for target detection. After image enhancement, the deep convolutional neural network model is used to decode the dot matrix code in the image and determine the handwriting coordinate information.
[0008] Based on the handwriting coordinate information, using a machine learning algorithm to determine the student's answer data;
[0009] Determine the student's learning emotion data based on the collected student's physiological data and voice data in combination with the writing behavior data;
[0010] Automatically generate courseware by automatically identifying teaching resources, and automatically generate courseware details using video splitting technology;
[0011] Use smart terminals and smart whiteboards to enable teachers and students to interact and participate in offline immersive virtual classrooms;
[0012] Integrate data collected from various terminals for deep learning analysis, explore students' characteristics and patterns to provide personalized after-school services, and use blockchain to implement credit certification services.
[0013] Optionally, the step of performing target detection on the handwriting image in the collected writing behavior data based on a deep learning algorithm, and using a deep convolutional neural network model to decode the dot code in the image after image enhancement and determine the handwriting coordinate information includes:
[0014] Extracting a handwriting image from the writing behavior data;
[0015] Preprocessing the handwriting image using a target detection algorithm to identify the location of the handwriting area;
[0016] Perform image enhancement processing on the handwriting area to improve the quality of the dot matrix code image;
[0017] Using a pre-trained deep convolutional neural network model, the enhanced handwriting image is repeatedly recognized and the image is decomposed into multiple sub-regions;
[0018] The dot matrix code in each sub-area is digitally decoded and restored to two-dimensional coordinate information, and the handwriting trajectory is represented according to the continuous coordinate points;
[0019] Match the identified coordinate point sequence with the content of the dot matrix code paper corresponding to the lesson and restore it into text or formula information.
[0020] Optionally, the step of determining the student's answer data using a machine learning algorithm based on the handwriting coordinate information includes:
[0021] For fill-in-the-blank questions and multiple-choice questions, the classification algorithm in machine learning is used to learn the training samples and obtain the classification model;
[0022] Extracting handwriting data features from the handwriting image;
[0023] Input the handwriting data features into a trained classification model for automatic judgment to determine whether it is correct or wrong;
[0024] For subjective questions, we use OCR technology to recognize text answers and natural language processing to determine whether the answers are correct;
[0025] For subjective schema or formula answers, deep learning is used to perform image recognition to determine the accuracy;
[0026] The student answer data is obtained by comprehensive analysis and judgment results.
[0027] Optionally, the step of automatically generating courseware by automatically identifying teaching resources and automatically generating courseware details by using video splitting technology includes:
[0028] Automatically extract metadata from teaching resources uploaded by teachers to identify structured information such as titles and keywords;
[0029] Use machine learning algorithms to automatically classify and cluster teaching resources and build a knowledge point framework for each course;
[0030] Automatically generate courseware templates based on the knowledge point framework. The courseware templates include outlines, chapters, and content details that are automatically extracted and generated;
[0031] Automatically split the video using video object detection technology, and each video subsection corresponds to a courseware node, so as to insert the course video into the courseware template;
[0032] Automatically add interactive analysis questions for each knowledge point;
[0033] Automatically embed student learning data statistics in courseware nodes;
[0034] Configure courseware authorization to support granular access control for different roles.
[0035] Optionally, the step of using a smart terminal and a smart whiteboard to enable teachers and students to interact and participate in an offline immersive virtual classroom includes:
[0036] Establish real-time video connection between the smart terminals used by students and the smart whiteboards used by teachers to simulate on-site teaching scenarios;
[0037] The smart whiteboard obtains the teacher's drawing board usage data on the virtual drawing board, and generates drawing board display content according to the drawing board usage data;
[0038] Synchronize the display content of the drawing board to the smart terminal in real time for display;
[0039] The intelligent terminal obtains the writing behavior data and receives the student's terminal operation data;
[0040] The intelligent terminal generates terminal display data according to the terminal operation data and the writing behavior data, and generates a three-dimensional immersive learning environment for students based on the terminal display data in combination with virtual reality technology;
[0041] The intelligent terminal evaluates the student's learning attitude through face recognition, and provides the learning attitude evaluation result to the teacher terminal and the cloud server;
[0042] The intelligent terminal collects student voice data for online question-and-answer interactive evaluation;
[0043] The smart terminal sends the learning record process data to the blockchain platform for storage.
[0044] Optionally, the step of integrating the data collected by each terminal for deep learning analysis, mining the characteristics of students to provide personalized after-school services, and using blockchain to implement credit certification services includes:
[0045] Build a big data platform on the cloud server to integrate multi-source teaching data from various terminals;
[0046] Use deep learning algorithm models to extract student personality characteristics from integrated multi-source teaching data;
[0047] Perform collaborative filtering recommendations on student assignments to predict difficulties and trends for individual learning behaviors;
[0048] Mining teaching rules, calculating the difficulty of knowledge points, and generating refined teaching suggestion data;
[0049] Generate personalized learning paths and suggestions for each student;
[0050] Generate secure academic achievement certificates on the blockchain platform to enable mutual recognition and exchange of certificates among different schools;
[0051] Receive plan modification data submitted by students based on their own learning situation to modify their study plans, adjust students' study courses and learning progress according to the plan modification data, and realize flexible independent learning.
[0052] Optionally, the step of automatically extracting metadata from the teaching resources uploaded by the teacher and identifying structured information such as titles and keywords includes:
[0053] Identify the document data in the teaching resources and extract text content;
[0054] Extracting audio and video amplitude features from the video in the teaching resource, and identifying key frames and characters using a deep learning method;
[0055] Establishing a topic model and extracting important words for the text content, the key frames and the characters to obtain structured information such as titles and keywords;
[0056] Use pre-built knowledge graphs to represent structured information in an entity and relational manner;
[0057] Utilize multi-source information from multiple devices to perform mutual verification and clustering to improve metadata.
[0058] Optionally, the courseware template is automatically generated on the knowledge point framework, and the courseware template includes the steps of automatically extracting and generating the outline, chapters, and content details, including:
[0059] According to the metadata extracted from teaching resources, a directory tree structure framework of knowledge points is constructed;
[0060] Prepare courseware templates and define the corresponding relationship model between titles and content modules at all levels;
[0061] Automatically fill in the title level in the courseware template according to the directory tree structure framework and knowledge point framework of the knowledge point;
[0062] Extract keywords and phrases from teaching resources to automatically generate chapter content outlines;
[0063] Mining important paragraphs and teaching points in teaching resources, using automatic summarization to generate each detailed content;
[0064] Identify difficult modules based on learning data and give them annotations for enhanced processing;
[0065] Configure the courseware layout, style, and interaction details to generate personalized, publishable courseware.
[0066] Optionally, the step of automatically splitting the video by video object detection technology, wherein each video subsection corresponds to a courseware node, and inserting the course video into the courseware template includes:
[0067] An object detection neural network is used on the course video to identify the location of the teacher and the key teaching objects in the video;
[0068] Track and match the time sequence of the teacher and key teaching objects in the course video, find the frames or time points where the course video can be naturally divided, and automatically divide the course video into multiple segments or sub-videos. The length of each video segment is automatically determined according to the degree of change of the object;
[0069] Extract metadata of video object detection predictions;
[0070] According to the courseware layout and the content of each cluster, automatically select appropriate video clips and embed them into the corresponding nodes of the courseware;
[0071] Configure user cropping and editing functions to optimize the automatic insertion effect. After embedding, play the corresponding video on the courseware node as needed to collaborate on dual-screen learning.
[0072] Another aspect of the present invention provides a smart teaching system based on artificial intelligence and a smart dot matrix pen, comprising: a smart dot matrix pen, a cloud server, a smart terminal and a smart whiteboard;
[0073] The cloud server is configured as follows:
[0074] When the smart dot matrix pen writes on the dot matrix code paper, obtaining writing behavior data collected by the smart dot matrix pen;
[0075] Based on the deep learning algorithm, the handwriting image in the collected writing behavior data is used for target detection. After image enhancement, the deep convolutional neural network model is used to decode the dot matrix code in the image and determine the handwriting coordinate information.
[0076] Based on the handwriting coordinate information, using a machine learning algorithm to determine the student's answer data;
[0077] Determine the student's learning emotion data based on the collected student's physiological data and voice data in combination with the writing behavior data;
[0078] Automatically generate courseware by automatically identifying teaching resources, and automatically generate courseware details using video splitting technology;
[0079] Use smart terminals and smart whiteboards to enable teachers and students to interact and participate in offline immersive virtual classrooms;
[0080] Integrate data collected from various terminals for deep learning analysis, explore students' characteristics and patterns to provide personalized after-school services, and use blockchain to implement credit certification services.
[0081] By adopting the technical solution of the present invention, the intelligent teaching method based on artificial intelligence and smart dot matrix pen includes: when the smart dot matrix pen writes on the dot matrix code paper, writing behavior data is collected; based on the deep learning algorithm, the handwriting image in the collected writing behavior data is subjected to target detection, and after the image is enhanced, the dot matrix code in the image is decoded using a deep convolutional neural network model to determine the handwriting coordinate information; based on the handwriting coordinate information, the student's answer data is determined using a machine learning algorithm; according to the collected physiological data and voice data of the students, combined with the writing behavior data, the student's learning emotion data is determined; the courseware is automatically generated by automatically identifying teaching resources, and the courseware details are automatically generated by using video splitting technology; the offline immersive virtual classroom interaction and participation between teachers and students is realized by using smart terminals and smart whiteboards; the data collected at each end are integrated for deep learning analysis, the student's characteristic laws are mined to provide personalized after-school services, and the credit certification service is realized by using blockchain. Through the scheme of the present invention, the automatic and personalized generation of teaching content and interactive courseware can be realized, reducing the workload of teachers in preparing teaching resources; the multi-source teaching data collected by the smart dot pen can be used to comprehensively and meticulously evaluate and record students' learning situation and performance, providing teachers with personalized teaching references; the immersive classroom set up with virtual reality technology can enhance students' sense of participation and learning interest; the use of machine learning and other algorithms can automatically review and evaluate students' homework, reducing the workload of teachers in correcting homework; through emotion recognition, students' learning emotions can be monitored in real time, helping teachers to identify and deal with learning obstacles in a timely manner; big data analysis can discover teaching rules and provide customized learning suggestions for students with different learning levels and styles; blockchain is used to realize public certification of credits, making it convenient for students to switch between different schools; cloud server (or Internet of Things server) services can realize the sharing and collaboration of teaching resources, helping to expand the coverage of education; and an open and lifelong learning model is adopted to meet the personalized learning needs of society. BRIEF DESCRIPTION OF THE DRAWINGS
[0082] Figure 1 It is a flow chart of a smart teaching method based on artificial intelligence and a smart dot pen provided by an embodiment of the present invention;
[0083] Figure 2 It is a schematic block diagram of an intelligent teaching system based on artificial intelligence and a smart dot pen provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0084] In order to be able to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. In addition, although the embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and the scope of the present disclosure can be fully conveyed to those skilled in the art.
[0085] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.
[0086] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are only used for description purposes or to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features (or describing a specific order). Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.
[0087] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0088] Refer to the following Figure 1 to Figure 2 To describe an intelligent teaching method and system based on artificial intelligence and a smart dot pen provided according to some embodiments of the present invention.
[0089] like Figure 1 As shown, an embodiment of the present invention provides a smart teaching method based on artificial intelligence and a smart dot pen, comprising:
[0090] When the smart dot matrix pen writes on the dot matrix code paper, the writing behavior data (such as writing target, coordinates, pressure value, time and speed, etc.; for example, the smart dot matrix pen collects handwriting images through an infrared camera and collects handwriting pressure signals through a pressure sensor; the sensor set on the smart dot matrix pen can also collect the physiological data of the writer / student);
[0091] Based on the deep learning algorithm, the handwriting image in the collected writing behavior data is used for target detection. After image enhancement, the deep convolutional neural network model is used to decode the dot matrix code in the image and determine the handwriting coordinate information.
[0092] Based on the handwriting coordinate information, a machine learning algorithm is used to determine the student's answer data (such as determining the answers to fill-in-the-blank questions and multiple-choice questions, and evaluating subjective questions by combining image recognition technology and speech recognition technology, etc.);
[0093] Determine the student's learning emotion data based on the collected student's physiological data and voice data in combination with the writing behavior data;
[0094] In this step, students' facial expression data can be collected through the smart terminal camera, and visual emotion analysis can be used to determine their learning enthusiasm. The microphone can be used to collect speech related to the learning status and classify it to determine the learning status (enthusiasm, dissatisfaction, etc.).
[0095] Automatically generate courseware by automatically identifying teaching resources, and automatically generate courseware details using video splitting technology;
[0096] Use smart terminals and smart whiteboards to enable teachers and students to interact and participate in offline immersive virtual classrooms;
[0097] Integrate data collected from various terminals (such as smart dot pens, cloud servers, smart terminals, smart whiteboards and other sensors) for deep learning analysis, explore students' characteristics and patterns to provide personalized after-school services, and use blockchain to implement credit certification services.
[0098] In the embodiment of the present invention, the above steps realize a personalized intelligent teaching method based on data collection of a smart dot pen and a deep learning algorithm, which has better intelligence, improves teaching efficiency, and can bring a better teaching experience.
[0099] Through the scheme of this embodiment, the teaching content and interactive courseware can be automatically and personalized generated, reducing the workload of teachers in preparing teaching resources; the multi-source teaching data collected by the smart dot pen can comprehensively and meticulously evaluate and record the students' learning situation and performance, providing teachers with personalized teaching references; the immersive classroom set up with virtual reality technology can enhance students' sense of participation and learning interest; the use of machine learning and other algorithms can automatically review and evaluate students' homework, reducing the workload of teachers in correcting homework; through emotion recognition, students' learning emotions can be monitored in real time, helping teachers to identify and deal with learning obstacles in a timely manner; big data analysis can discover teaching rules and provide customized learning suggestions for students with different learning levels and styles; blockchain is used to realize public certification of credits, making it convenient for students to switch between different schools; cloud server (or Internet of Things server) services can realize the sharing and collaboration of teaching resources, helping to expand the coverage of education; an open and lifelong learning model is adopted to meet the personalized learning needs of society.
[0100] In some possible implementations of the present invention, the step of performing target detection on the handwriting image in the collected writing behavior data based on a deep learning algorithm, and using a deep convolutional neural network model to decode the dot code in the image after image enhancement and determine the handwriting coordinate information includes:
[0101] Extracting a handwriting image from the writing behavior data;
[0102] In this step, the writing behavior data includes the handwriting coordinate sequence data recorded by the intelligent dot pen input device (these coordinate sequence data record the handwriting motion trajectory), from which the morphological features need to be extracted to represent the image data of the handwriting. The specific operations include: reconstructing the shape contour of the handwriting according to the handwriting coordinate sequence; digitizing the reconstructed handwriting contour into a binary image with a fixed pixel resolution; performing some preprocessing on the image, such as normalization, removing extra handwriting, etc.; finally outputting a binary image representing a single handwriting is to extract the handwriting image from the writing behavior data. In short, it refers to the process of extracting a binary image representing the morphology of a single handwriting from the writing behavior data that stores the handwriting motion trajectory.
[0103] Preprocessing the handwriting image using a target detection algorithm to identify the location of the handwriting area;
[0104] In this step, the handwriting image refers to a single binary image representing a complete handwriting extracted from the writing behavior data through the aforementioned process; the target detection algorithm (such as YOLO) is a type of algorithm used to detect and distinguish multiple targets in an image, which can identify the category and frame position of each target in the image; applying the target detection algorithm to the handwriting image, the handwriting can be regarded as a target for detection; the algorithm will predict the position frame of the handwriting in the image, that is, identify the position of the handwriting area; this is a preprocessing step, the purpose of which is to provide an effective handwriting area for subsequent feature extraction or recognition, and filter out the background and other parts; in this way, the preprocessing work of identifying and locating the handwriting area itself from the original handwriting image through target detection is completed. In short, the target detection algorithm is used to locate and strip the effective handwriting part in the handwriting image.
[0105] Perform image enhancement processing (such as color space conversion and histogram equalization) on the handwriting area to improve the quality of the dot matrix code image;
[0106] In this step, the handwriting area refers to the part of the image area containing handwriting obtained by localization of target detection; image enhancement is a preprocessing technology, the purpose of which is to correct image quality problems and enhance the display of useful information in the image; specifically, it can include: color space conversion (such as converting RGB to grayscale, removing color signals to improve contrast), histogram equalization (adjusting the histogram distribution of each pixel in the image, stretching the depth range to improve brightness differences); these methods can remove influences, reduce noise, and enhance features after processing; thereby effectively improving the quality and recognition effect of extracted features and binary dot code images representing handwriting; in short, it refers to optimizing the extracted handwriting image through common image preprocessing methods to improve its quality as a feature input.
[0107] Using a pre-trained deep convolutional neural network model, the enhanced handwriting image is repeatedly recognized and the image is decomposed into multiple sub-regions;
[0108] In this step, the previous step enhances the handwriting image quality through preprocessing; a pre-trained (such as on an identity verification dataset) deep convolutional neural network model (DCNN model) is used to recognize such a handwriting image; but DCNN can only achieve global-level handwriting recognition and cannot extract local details; so DCNN is used here for repeated recognition: first, the entire image is recognized once; then the image is slid in a window with a certain step size, and each window is input into DCNN recognition as a sub-region image; such repeated recognition can decompose the original image into multiple overlapping sub-regions; the purpose is to achieve decomposition from the whole to the details through multiple local recognitions, and extract richer local features to support subsequent applications; in short, DCNN is used to form overlapping sliding windows on the original image to achieve fine-grained sub-region decomposition of the handwriting image.
[0109] The dot matrix code in each sub-area is digitally decoded and restored to two-dimensional coordinate information, and the handwriting trajectory is represented according to the continuous coordinate points;
[0110] In this step, the previous step uses DCNN to decompose the handwriting image into multiple sub-regions; each sub-region actually stores the morphological feature information of the handwriting, which is digitally encoded in the form of a dot matrix code; here it is necessary to digitally decode the dot matrix code in each sub-region and restore it to two-dimensional coordinate information; since the feature information in the dot matrix code can be restored to a continuous handwriting point coordinate sequence in the sub-region; the coordinate point sequences decoded from different sub-regions are continuously connected, and the complete trajectory of the handwriting in the entire image can be restored according to the order of the points; this is equivalent to completing the digital encoding and restoration process from image features to the real handwriting motion trajectory; in general, the dot matrix code digital information in each sub-region is used to restore and connect the continuous coordinate trajectory information of the handwriting in the entire image.
[0111] Match the identified coordinate point sequence with the content of the dot matrix code paper corresponding to the lesson and restore it into text or formula information.
[0112] In this step, the previous step uses image processing technology to extract the coordinate point sequence information of the handwriting from the handwriting image; however, a single coordinate point sequence cannot directly reflect the semantic content represented by the handwriting; the dot code technology is used here: a specific dot code paper is used to record handwriting during teaching; this paper contains various preset semantic points, each point corresponding to a word or formula symbol; the extracted coordinate point sequence can now be matched with the corresponding points in the dot code paper; the successfully matched points can be restored to the corresponding text or formula symbol; thereby realizing the use of handwriting image information to restore the text or formula content actually recorded in the handwriting of the class; the whole process is: extracting coordinates through image processing -> matching the coordinates with the dot code -> realizing restoration to the semantic content actually written by the teacher.
[0113] Through the scheme of this embodiment, the handwriting area can be detected and located with high precision, eliminating the influence of interference information on the recognition result; the quality of the dot code image can be improved, which is conducive to subsequent efficient deep learning recognition; the dot code is recognized by the deep learning model, which has a higher recognition rate and accuracy than the traditional graphic recognition algorithm; the accurate digitization of handwriting is achieved, and rich coordinate trajectory student operation records are extracted, which are further restored into teaching entity content, providing a reliable basis for dynamic evaluation and interactive teaching.
[0114] In some possible implementations of the present invention, the following factors need to be considered when selecting a suitable deep convolutional neural network model for training:
[0115] Dataset selection and classification tasks: According to the characteristics of dot-code handwritten images, select a suitable training dataset, such as MNIST, EMNIST and other handwritten digital datasets; clearly identify whether the dot-code recognition task is a classification task or an object detection task;
[0116] Network structure: For simple dot code classification and recognition, you can choose the classic network LeNet; for complex handwriting, you can choose ResNet, VGG, etc. as the basis for improving the network structure;
[0117] Training parameters: Choose reasonable learning rate, optimizer, loss function and other training parameters, such as using Adam optimizer and cross entropy as loss function;
[0118] Data enhancement: Perform data enhancement such as rotation, scaling, and distortion to enrich the sample size;
[0119] Training scale: Choose the mini-batch size based on the complexity of the task, and set the appropriate network depth and number of layers;
[0120] Auxiliary technology: Use data enhancement, regularization and other methods to prevent overfitting;
[0121] Evaluation indicators: Use multiple indicators such as accuracy and recall to evaluate the effect of the model on the training set and validation set;
[0122] Model selection: After multiple trials, select the model with the best performance on the validation set;
[0123] Parameter adjustment: Fine-tune the selected model and optimize the parameters to obtain the best recognition effect.
[0124] In summary, it is necessary to select matching task types and structures for the model according to the specific scenario, and use iterative optimization methods to obtain the best model.
[0125] In some possible implementations of the present invention, the step of determining the student's answer data using a machine learning algorithm based on the handwriting coordinate information includes:
[0126] For fill-in-the-blank questions and multiple-choice questions, use the classification algorithm in machine learning (such as KNN, decision tree, etc.) to learn the training samples and obtain the classification model;
[0127] Extracting handwriting data features from the handwriting image;
[0128] Input the handwriting data features into a trained classification model for automatic judgment to determine whether it is correct or wrong;
[0129] For subjective questions, we use OCR technology to recognize text answers and natural language processing to determine whether the answers are correct;
[0130] For subjective schema or formula answers, deep learning is used to perform image recognition to determine the accuracy;
[0131] The student answer data is obtained by comprehensive analysis and judgment results.
[0132] In this embodiment, the judgment and evaluation of a large number of assignments are automatically completed, freeing up teachers' workload; qualitative and quantitative judgments are made on subjective questions to improve evaluation results; and refined learning status monitoring can be implemented to assist personalized teaching. Through learning analysis, teachers can be guided to optimize teaching quality, improve learning efficiency and personalization level.
[0133] In some possible implementations of the present invention, the step of determining the student's learning emotion data based on the collected physiological data and voice data of the student in combination with the writing behavior data includes:
[0134] Facial feature point detection: Use the facial key point detection algorithm to detect the face in the image and extract the coordinates of 68 feature points;
[0135] Expression recognition: judge basic expressions such as happiness, anger, and head hanging according to the position relationship of key points. SVM classifier can be used to train sample data for expression classification;
[0136] Emotion recognition: Based on the expression recognition results, the emotion labels are used for secondary classification to identify emotions such as happiness, sadness, and boredom. CNN+LSTM network can be used for deep learning;
[0137] Expression change analysis: Collect multiple frames of images over a period of time, use the optical flow method to calculate the displacement of feature points, analyze the frequency and amplitude of expression changes, and determine the degree of emotional fluctuation;
[0138] Combined with speech analysis: Speech emotion analysis technology can identify characteristics such as tone and speaking speed, and combined with video analysis to improve recognition accuracy;
[0139] Long short-term memory network: using time series data to achieve long-term tracking of emotional states and determine factors affecting learning;
[0140] Multimodal deep fusion: Input multiple modal features such as video, voice and physiological signals into the deep network to achieve better output;
[0141] Update sample data regularly: Continue to collect labeled data sets and continuously improve deep learning models to enhance recognition capabilities.
[0142] In this embodiment, facial feature extraction, multimodal fusion, and temporal context analysis are implemented to obtain more accurate and detailed learning emotion judgment results.
[0143] In some possible implementations of the present invention, in the step of inputting multiple modal features such as video, voice and physiological signals into a deep network to achieve better output, the following points need to be considered when dealing with the synchronization problem of multimodal data:
[0144] Delay between acquisitions: There is bound to be a time delay in the acquisition of video and audio data, and timestamp synchronization is required during acquisition;
[0145] Correspondence during segmentation: When video and audio are segmented into frames or blocks at fixed time intervals (e.g., 1 second), the corresponding timestamps need to be recorded for subsequent correspondence;
[0146] Synchronization of processing flow: Videos require additional processing such as face detection and feature extraction. To maintain synchronization, multi-threaded parallel processing of different modalities can be used;
[0147] Time window for feature extraction: The extracted video and voice features need to come from data within the same time range, such as using the sliding time window function to extract;
[0148] Alignment of model fusion levels: Multimodal network structures need to fuse features of different modalities at corresponding levels, such as merging video and speech features at the back layer of CNN;
[0149] Accurate correspondence at the event level: The occurrence time of key emotional events should have accurate corresponding time tags in each modal data;
[0150] Synchronous presentation of result output: Emotion recognition output is presented together with multimodal input to facilitate subsequent analysis and understanding.
[0151] In this embodiment, correctly handling the above problems can ensure that multimodal data is highly synchronized in each stage of training and testing, thereby better utilizing complex multi-source information and improving the effect of emotion recognition.
[0152] In some possible implementations of the present invention, in feature extraction for sentiment analysis, the following points need to be considered when selecting a suitable time window:
[0153] The time scale of emotional changes: Different emotions, such as happiness, anger, anxiety, etc., have different expression durations. The length of the time window should be set considering the characteristics of the emotion itself;
[0154] Computational cost: The longer the time window, the greater the feature computation, which tends to increase model complexity. It is necessary to balance computation and representation capabilities;
[0155] Time dependency: If the time window is too short, it will not show the changing pattern of emotions in the time dimension; if the time window is too long, it will introduce redundant information.
[0156] Statistical characteristics of the data set: By counting the average duration of emotional events in the video database, an appropriate time window range is determined;
[0157] Interactive feedback cycle: For example, for video teaching feedback, a standard classroom assignment unit time such as 5 minutes can be considered as the window length.
[0158] Generally speaking, in sentiment analysis, short time windows of 1-5 seconds can be used to extract static features from videos; medium time windows of 50-300 milliseconds are suitable for speech to show dynamic changes; physiological indicators match one interaction cycle as a window; model learning uses parallel multi-scale time windows to fuse information; and flexible design of time domain feature extraction is conducive to sentiment computing.
[0159] In some possible implementations of the present invention, a method for evaluating the influence of different time windows in sentiment analysis is also included:
[0160] Select a set of sample data and extract features under different time windows, such as video static features extracted from time windows of 1s, 3s, and 5s;
[0161] Use the same classifier (such as SVM, CNN, etc.) for training and testing, and compare the classification accuracy under different time windows;
[0162] Apply more complex neural network models to compare the recognition capabilities of long and short time series features;
[0163] Analyze misclassified samples to see whether different time windows can better capture the details of emotional expression;
[0164] Densely sample the video, extract the features of the two frames with the largest and smallest differences within 1 second, and observe the two-category recognition effect;
[0165] Extract acoustic features of speech at different time scales, such as short-time energy and short-time amplitude, and determine its recognition efficiency;
[0166] Use time-frequency analysis on different frequency components of physiological data to determine the frequency band characteristics that are most likely to identify emotions;
[0167] Visual and acoustic features are fused early and late in different time windows, and the recognition results are compared;
[0168] Statistics of various evaluation indicators are given to rank the performance of emotion recognition in different time windows.
[0169] In this embodiment, evaluating the optimization space of the time window parameters in a quantitative manner is helpful to find the optimal window length setting for emotion recognition.
[0170] In some possible implementations of the present invention, a method for processing the time window overlap problem in the feature extraction process is also included:
[0171] Use sliding time window: continuously slide the time window with a certain step length (less than the time window length) to extract the features of each window segment;
[0172] Use average pooling: average the features of adjacent time windows to reduce the impact of overlapping parts;
[0173] Max pooling: Take the maximum value of features in adjacent time windows and retain newly emerged features;
[0174] Joint optimization: Use the overlapping and non-overlapping parts of the time windows as different channels to jointly train network parameters;
[0175] Add a window code to each time feature, such as adding a one-dimensional feature to represent the time window and distinguish the source of the feature;
[0176] Dynamic time window: dynamically adjust the time window length according to the video content to avoid fixed overlap;
[0177] In this step, the implementation method of the dynamic time window includes: using video analysis technology to perform scene judgment on each frame of the image, identifying whether the current video clip is a static background or a character action process, etc.; setting the corresponding time window length lower limit based on the scene type, such as 1s for static scenes and 0.5s for action processes, etc.; for consecutive clips of the same scene type, using a fixed-length time window to extract features; when the scene type changes, detecting the change frame, and increasing the length of the previous time window by a certain increment value, that is, realizing dynamic adjustment of the window length; while focusing on retaining the features in the time window before and after the change point, a symmetrical time window span centered on the change point can be used; for clips with rapid changes or repeated short-time switching scenes, the window length needs to be appropriately shortened to prevent the correlation from decreasing; adding a metadata information channel representing the dynamic attributes of the current window during feature extraction; and fusing window features of different lengths according to metadata during subsequent training to realize dynamic learning of the time window influence. By dynamically adjusting the window length, the temporal variation law of the emotional state of the video can be better captured, and the representation ability of emotion recognition can be improved.
[0178] Multi-resolution analysis: Modeling using a combination of features with different temporal resolutions;
[0179] Attention mechanism: Use the self-attention mechanism to mine information related to the time dimension;
[0180] Transformation learning: Map features to disjoint subspaces and then learn them, effectively solving the overlapping problem.
[0181] In this embodiment, the above methods can well handle the problem of extracting repetitive features by overlapping time windows, and avoid selective bias during learning.
[0182] In some possible implementations of the present invention, the step of automatically generating courseware by automatically identifying teaching resources and automatically generating courseware details by using video splitting technology includes:
[0183] Automatically extract metadata from teaching resources uploaded by teachers (such as PPT, PDF, videos, etc.) to identify structured information such as titles and keywords;
[0184] Use machine learning algorithms to automatically classify and cluster teaching resources and build a knowledge point framework for each course;
[0185] In this step, teaching resources refer to various teaching contents, such as PPT, videos, test papers, exercises, etc.; automatic classification refers to the automatic classification of teaching resources using machine learning algorithms (such as SVM, decision tree and other classification algorithms); automatic clustering refers to the automatic grouping of teaching resources using clustering algorithms (such as K-Means, etc.); the purpose is to automatically classify teaching resources according to the subject content and knowledge points of teaching resources; and then build the corresponding knowledge point framework structure under each course system based on the classification and clustering results; the knowledge point framework includes the various topics and second and third level knowledge points covered by the course; it realizes the automatic mining of the internal logical framework of subject knowledge points from a large number of unstructured teaching materials; that is, the teaching resources are automatically analyzed using machine learning technology to build a knowledge point content structure system corresponding to professional courses.
[0186] Automatically generate courseware templates (PPT or HTML5) based on the knowledge point framework. The courseware templates include automatically extracted and generated outlines, chapters, and content details.
[0187] Automatically split the video using video object detection technology, and each video subsection corresponds to a courseware node, so as to insert the course video into the courseware template;
[0188] Automatically add interactive analysis questions for each knowledge point (support online submission to improve learning effect);
[0189] Automatically embed student learning data statistics in courseware nodes (to help teachers understand the difficulties);
[0190] Configure courseware authorization to support granular access control for different roles.
[0191] Through the solution of this embodiment, standardized interactive courseware is automatically generated, saving teachers' resource preparation time; the whole set of courses is divided into various knowledge points in detail to promote refined teaching; the difficulty of the courseware is optimized in real time according to student data to improve learning effects; the courseware has embedded interactive functions to improve participation, and the integrated video makes the explanation more empathetic, which greatly improves teaching efficiency while also realizing personalized and situational teaching.
[0192] In some possible implementations of the present invention, the steps of using a smart terminal and a smart whiteboard to realize offline immersive virtual classroom interaction and participation between teachers and students include:
[0193] (Using WebRTC and other technologies) establish real-time video connection between smart terminals used by students and smart whiteboards used by teachers to simulate on-site teaching scenarios;
[0194] The smart whiteboard obtains the teacher's drawing board usage data (handwriting or operation object data) on the virtual drawing board, and generates drawing board display content according to the drawing board usage data;
[0195] Synchronize the display content of the drawing board to the smart terminal in real time for display;
[0196] The smart terminal obtains the writing behavior data and receives the student's terminal operation data (the student can provide interactive feedback such as answering, liking, asking questions, etc. to the teacher's operation through the smart terminal);
[0197] The intelligent terminal generates terminal display data according to the terminal operation data and the writing behavior data, and generates a three-dimensional immersive learning environment for students based on the terminal display data in combination with virtual reality technology;
[0198] The intelligent terminal evaluates the student's learning attitude through face recognition, and provides the learning attitude evaluation result to the teacher terminal and the cloud server (or Internet of Things server);
[0199] The intelligent terminal collects student voice data for online question-and-answer interactive evaluation;
[0200] The smart terminal sends the learning record process data to the blockchain platform for storage.
[0201] The scheme of this embodiment has a real-time interactive teaching effect that is better than online education and cultivates students' learning habits; the virtual environment realizes all-round sensory contact in learning and improves the quality of learning; multi-dimensional interactive recording and evaluation of students' learning situation; improves student participation and learning efficiency; forms personalized learning process management and learning record storage; and greatly improves students' immersive learning experience and teacher management efficiency.
[0202] In some possible implementations of the present invention, the steps of integrating the data collected by each terminal for deep learning analysis, mining the characteristics of students to provide personalized after-school services, and using blockchain to implement credit certification services include:
[0203] Build a big data platform on the cloud server (or IoT server) to integrate multi-source teaching data (such as writing behavior data, drawing board usage data, terminal operation data, face data, sound data, etc.; specifically including handwriting, video, expression, etc.) from various terminals (such as dot matrix pen, smart whiteboard, smart terminal, teacher terminal and other sensors);
[0204] Use deep learning algorithm models (which can be pre-selected or trained) to extract student personality characteristics (such as learning speed, difficulty type, etc.) from the integrated multi-source teaching data;
[0205] Perform collaborative filtering recommendations on student assignments to predict difficulties and trends for individual learning behaviors;
[0206] Mining teaching rules, calculating the difficulty of knowledge points, and generating refined teaching suggestion data;
[0207] Generate personalized learning paths and suggestions for each student (based on the data obtained above) (such as key learning modules and supporting teaching materials);
[0208] Generate secure academic achievement certificates on the blockchain platform to enable mutual recognition and exchange of certificates among different schools;
[0209] Receive plan modification data submitted by students based on their own learning situation to modify their study plans, adjust students' study courses and learning progress according to the plan modification data, and realize flexible independent learning.
[0210] Through the solution of this embodiment, the teaching quality and learning efficiency can be improved, personalized learning and customized services can be achieved, the learning process is more convenient and traceable, the development of open online education is promoted, and the system uses the advantages of big data to comprehensively improve the teaching level.
[0211] In some possible implementations of the present invention, the step of automatically extracting metadata from the teaching resources uploaded by the teacher and identifying structured information such as titles and keywords includes:
[0212] Identify the document data in the teaching resources (such as performing OCR on PPT / PDF documents) and extract text content;
[0213] Extracting audio and video amplitude features from the video in the teaching resource, and identifying key frames and characters using a deep learning method;
[0214] Establish a topic model and extract important words for the text content, key frames and characters (using algorithms such as TF-IDF and LSA) to obtain structured information such as titles and keywords (and collect a large number of annotated samples for training);
[0215] Use pre-built knowledge graphs to represent structured information in an entity and relational manner;
[0216] Utilize multi-source information from multiple devices to perform mutual verification and clustering to improve metadata.
[0217] The solution of this embodiment automatically processes teaching resources in different formats to reduce manual extraction costs; extracts structured and enriched metadata to reproduce the structural relationship of teaching content; provides basic data support for the subsequent automatic generation of courseware and knowledge graph services; effectively manages and retrieves massive teaching resource information to improve the reuse rate; can optimize metadata extraction models for different disciplines and types to improve the level of automation; and realizes intelligent management and utilization of personalized services for teaching resources through automatic metadata parsing.
[0218] In some possible implementations of the present invention, a method for evaluating the quality and accuracy of extracted metadata is also included:
[0219] Manually annotate a portion of teaching resources as a test set, compare the extracted metadata with the manual annotation results, and calculate the precision and recall rates;
[0220] The different structured fields extracted, such as titles and keywords, were evaluated separately to analyze the performance of each type of metadata;
[0221] Manually check some simple content extraction results to find the main causes of errors, which are used for model iterative optimization;
[0222] Use the extracted metadata as query conditions to retrieve relevant documents in the resource library and calculate the correct retrieval rate;
[0223] Use metadata to build a knowledge graph and test whether the facts in the graph are correct;
[0224] Filter teaching materials by complexity and other conditions, and test the effect curve of the model in different scenarios;
[0225] Compare the metadata with the instructional video and evaluate whether the text description is accurate and complete against the video content;
[0226] Extract the same teaching material multiple times to observe the stability of the results and evaluate the robustness of the extraction process;
[0227] Statistics are collected on quantitative indicators such as the system's overall AUC and accuracy, and the changing trends over time are recorded.
[0228] The above multiple rounds of testing can more realistically reflect the quality of system metadata extraction and guide iterative optimization work.
[0229] In some possible implementations of the present invention, the courseware template is automatically generated on the knowledge point framework, and the courseware template includes the steps of automatically extracting and generating the outline, chapters, and content details, including:
[0230] According to the metadata extracted from teaching resources, a directory tree structure framework of knowledge points is constructed;
[0231] Prepare courseware templates and define the corresponding relationship model between titles and content modules at all levels;
[0232] Automatically fill in the title level in the courseware template according to the directory tree structure framework and knowledge point framework of the knowledge point;
[0233] Extract keywords and phrases from teaching resources to automatically generate chapter content outlines;
[0234] Mining important paragraphs and teaching points in teaching resources, using automatic summarization to generate each detailed content;
[0235] Identify difficult modules based on learning data and give them annotations for enhanced processing;
[0236] Configure the courseware layout, style, and interaction details to generate personalized, publishable courseware.
[0237] The solution of this embodiment automatically generates standardized courseware, saving teachers' production time; realizes modular courseware design according to the detailed division of knowledge points; extracts and pays attention to difficult points to promote refined teaching; embeds interactive functions in the courseware to improve learning participation; and meets different teaching needs with personalized configuration; effectively improves teaching efficiency and promotes personalized and precise teaching.
[0238] In some possible implementations of the present invention, the step of automatically splitting the video using the video object detection technology, wherein each video subsection corresponds to a courseware node, and inserting the course video into the courseware template includes:
[0239] An object detection neural network is used on the course video to identify the location of the teacher and the key teaching objects in the video;
[0240] Track and match the time sequence of the teacher and key teaching objects in the course video, find the frames or time points where the course video can be naturally divided, and automatically divide the course video into multiple segments or sub-videos. The length of each video segment is automatically determined according to the degree of change of the object;
[0241] Extract metadata (e.g. time point, duration, etc.) of video object detection predictions;
[0242] According to the courseware layout and the content of each cluster, automatically select appropriate video clips and embed them into the corresponding nodes of the courseware;
[0243] Configure user cropping and editing functions to optimize the automatic insertion effect. After embedding, play the corresponding video on the courseware node as needed to collaborate on dual-screen learning.
[0244] The solution of this embodiment can achieve a high degree of integration between courseware and video, thereby enhancing the influence of learning; it can save teachers' working time for manually inserting videos; it ensures that the natural paragraphs of the video correspond to the courseware objectives, while taking into account the fluency of teaching; it supports personalized editing to optimize the automatic insertion effect; and it effectively improves the video teaching experience and teaching efficiency.
[0245] In some possible implementations of the present invention, the following points need to be considered when selecting a network model for object detection in video content:
[0246] Number of target object categories: For simple types such as people and PPT, you can consider single-stage detectors such as YOLO; for more object types, you can consider two-stage detectors;
[0247] Computational cost: If it is an online segmentation, give priority to models with high computational efficiency such as SSD; if it is an offline task, you can choose Faster R-CNN, which has better results;
[0248] Input resolution: Video resolution is generally high, so choose a model that can handle high-resolution input, such as ResNet or ResNeXt;
[0249] Utilization of temporal information: Considering that the video itself contains temporal information, a model based on video object tracking such as Faster R-CNN+LSTM can be selected;
[0250] Representativeness of the dataset: It is better to choose a model that has been pre-trained on a similar scene set;
[0251] Dynamic and static object recognition: If dynamic objects are involved, it is better to choose a model that combines optical flow or displacement;
[0252] Experimental evaluation: Select the model with the highest average accuracy and speed on a certain data set and hardware.
[0253] In summary, we recommend Faster R-CNN or Mask R-CNN series models to achieve object detection and segmentation in teaching videos. Combining with context tracking can improve the effect.
[0254] In some possible implementations of the present invention, there is an imbalance problem in the object detection dataset. The following methods can be adopted when selecting and training the model:
[0255] Dataset sampling: oversample minority class samples and undersample majority class samples to make the number of samples in each class relatively balanced;
[0256] Loss function design: Increase loss weight for minority class samples to force the model to focus on these samples;
[0257] Hard case mining: After multiple trainings, misclassified samples are selected as new samples to continuously enhance capabilities;
[0258] Transfer learning: pre-training with balanced datasets from the source domain and fine-tuning applied to target domain data;
[0259] Model structure design: Add a class attention mechanism in the subsequent prediction layer to improve sensitivity to minority classes;
[0260] Multi-task learning: adding auxiliary tasks to strengthen the model to learn the difference between difficult and easy samples;
[0261] Data enhancement: Rotate, scale, and other transformations are performed on minority samples to obtain new styles;
[0262] Model integration: After training multiple models, voting is used to improve the balance;
[0263] Selection of evaluation indicators: Comprehensive effect indicators such as AOC are used, not only considering the accuracy.
[0264] The above method can significantly reduce the adverse impact of data deviation on model training and select a better object detection model.
[0265] In some possible implementations of the present invention, the specific steps for automatically embedding a video clip into a PPT are as follows:
[0266] Perform object detection and segmentation on course videos, extract metadata of each video clip such as start time, duration, etc.;
[0267] Automatically build a knowledge point directory tree structure based on the courseware PPT content;
[0268] Match the metadata of the video clip with the knowledge points in the PPT nodes to determine the nodes suitable for insertion;
[0269] Insert a video control into a PPT node and automatically fill in the control properties based on the matching results: set the network address or local address of the file, set the start time and duration properties, and set the control size and position;
[0270] Optimization: If the duration is too long, it will be automatically replaced with an image axis; if there is no match, this segment will be skipped;
[0271] Provide editing function to manually correct matching results;
[0272] After completion, the video embedded courseware format is generated for different terminals, such as PPT and HTML5;
[0273] During playback, the video can be loaded and played on demand at the node.
[0274] This technology can truly achieve a high degree of integration between courseware and videos, helping learners to fully understand knowledge points from various perspectives.
[0275] In some possible implementations of the present invention, for different courseware formats, there are several ways to implement the video control:
[0276] For PPT format: Use the built-in video control in PPT to insert a video object into the slide, set the control properties to specify the video file address and playback range, etc., and record the video control to achieve automatic playback of the video in the courseware node;
[0277] For HTML5 format: Use Video HTML5 element to insert video node at the specified position of the page, set video source address and playback configuration through JavaScript, and bind control events to achieve click-to-play and auto-play effects;
[0278] For online players: embed Flash or HTML5 plug-ins in courseware to play online videos, set the video file address and play time through API, and respond to courseware events to synchronize with video playback;
[0279] For mobile courseware: encapsulate the native video playback component and insert it into the page card as needed, configure component properties to complete synchronization settings, and listen to courseware and component events to achieve seamless playback.
[0280] The above methods can all achieve automatic loading and playing of videos within the courseware node, providing a better experience for learners. The method you choose depends on the specific courseware format and mobile restrictions.
[0281] See also Figure 2 ,Another embodiment of the present invention provides a smart teaching system based on artificial intelligence and a smart dot matrix pen, comprising: a smart dot matrix pen, a cloud server, a smart terminal and a smart whiteboard;
[0282] The cloud server is configured as follows:
[0283] When the smart dot matrix pen writes on the dot matrix code paper, obtaining writing behavior data collected by the smart dot matrix pen;
[0284] Based on the deep learning algorithm, the handwriting image in the collected writing behavior data is used for target detection. After image enhancement, the deep convolutional neural network model is used to decode the dot matrix code in the image and determine the handwriting coordinate information.
[0285] Based on the handwriting coordinate information, using a machine learning algorithm to determine the student's answer data;
[0286] Determine the student's learning emotion data based on the collected student's physiological data and voice data in combination with the writing behavior data;
[0287] Automatically generate courseware by automatically identifying teaching resources, and automatically generate courseware details using video splitting technology;
[0288] Use smart terminals and smart whiteboards to enable teachers and students to interact and participate in offline immersive virtual classrooms;
[0289] Integrate data collected from various terminals for deep learning analysis, explore students' characteristics and patterns to provide personalized after-school services, and use blockchain to implement credit certification services.
[0290] It should be known that Figure 2 The block diagram of the intelligent teaching system based on artificial intelligence and smart dot pen is for illustration only, and the number of modules shown therein does not limit the protection scope of the present invention.
[0291] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0292] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0293] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.
[0294] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0295] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0296] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.
[0297] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0298] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
[0299] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions without departing from the spirit and scope of the present invention, and can make various changes and modifications, including the combination of the above-mentioned different functions and implementation steps, including software and hardware implementation methods, all of which are within the scope of protection of the present invention.
Claims
1. A smart teaching method based on artificial intelligence and smart dot pen, characterized in that: include: When the smart dot matrix pen writes on the dot matrix code paper, the writing behavior data is collected; Based on the deep learning algorithm, the handwriting image in the collected writing behavior data is used for target detection. After image enhancement, the deep convolutional neural network model is used to decode the dot matrix code in the image and determine the handwriting coordinate information. Based on the handwriting coordinate information, using a machine learning algorithm to determine the student's answer data; Determine the student's learning emotion data based on the collected student's physiological data and voice data in combination with the writing behavior data; Automatically generate courseware by automatically identifying teaching resources, and automatically generate courseware details using video splitting technology; Use smart terminals and smart whiteboards to enable teachers and students to interact and participate in offline immersive virtual classrooms; Integrate data collected from various terminals for deep learning analysis, explore students' characteristics and patterns to provide personalized after-school services, and use blockchain to implement credit certification services.
2. The intelligent teaching method based on artificial intelligence and intelligent dot matrix pen according to claim 1 is characterized in that: The step of performing target detection on the handwriting image in the collected writing behavior data based on the deep learning algorithm, and using the deep convolutional neural network model to decode the dot matrix code in the image after image enhancement and determine the handwriting coordinate information includes: Extracting a handwriting image from the writing behavior data; Preprocessing the handwriting image using a target detection algorithm to identify the location of the handwriting area; Perform image enhancement processing on the handwriting area to improve the quality of the dot matrix code image; Using a pre-trained deep convolutional neural network model, the enhanced handwriting image is repeatedly recognized and the image is decomposed into multiple sub-regions; The dot matrix code in each sub-area is digitally decoded and restored to two-dimensional coordinate information, and the handwriting trajectory is represented according to the continuous coordinate points; The identified coordinate point sequence is matched with the content of the dot matrix code paper corresponding to the lesson, and restored into text or formula information.
3. The intelligent teaching method based on artificial intelligence and intelligent dot pen according to claim 2 is characterized in that: The step of determining the student's answer data using a machine learning algorithm based on the handwriting coordinate information includes: For fill-in-the-blank questions and multiple-choice questions, the classification algorithm in machine learning is used to learn the training samples and obtain the classification model; Extracting handwriting data features from the handwriting image; Input the handwriting data features into a trained classification model for automatic judgment to determine whether it is correct or wrong; For subjective questions, we use OCR technology to recognize text answers and natural language processing to determine whether the answers are correct; For subjective schema or formula answers, deep learning is used to perform image recognition to determine the accuracy; The student answer data is obtained by comprehensive analysis and judgment results.
4. The intelligent teaching method based on artificial intelligence and intelligent dot matrix pen according to claim 3 is characterized in that: The steps of automatically generating courseware by automatically identifying teaching resources and automatically generating courseware details by using video splitting technology include: Automatically extract metadata from teaching resources uploaded by teachers to identify structured information such as titles and keywords; Use machine learning algorithms to automatically classify and cluster teaching resources and build a knowledge point framework for each course; Automatically generate courseware templates based on the knowledge point framework. The courseware templates include outlines, chapters, and content details that are automatically extracted and generated; Automatically split the video using video object detection technology, and each video subsection corresponds to a courseware node, so as to insert the course video into the courseware template; Automatically add interactive analysis questions for each knowledge point; Automatically embed student learning data statistics in courseware nodes; Configure courseware authorization to support granular access control for different roles.
5. The intelligent teaching method based on artificial intelligence and intelligent dot pen according to claim 4 is characterized in that: The steps of using the smart terminal and the smart whiteboard to realize the offline immersive virtual classroom interaction and participation between teachers and students include: Establish real-time video connection between the smart terminals used by students and the smart whiteboards used by teachers to simulate on-site teaching scenarios; The smart whiteboard obtains the teacher's drawing board usage data on the virtual drawing board, and generates drawing board display content according to the drawing board usage data; Synchronize the display content of the drawing board to the smart terminal in real time for display; The intelligent terminal obtains the writing behavior data and receives the student's terminal operation data; The intelligent terminal generates terminal display data according to the terminal operation data and the writing behavior data, and generates a three-dimensional immersive learning environment for students based on the terminal display data in combination with virtual reality technology; The intelligent terminal evaluates the student's learning attitude through face recognition, and provides the learning attitude evaluation result to the teacher terminal and the cloud server; The intelligent terminal collects student voice data for online question-and-answer interactive evaluation; The smart terminal sends the learning record process data to the blockchain platform for storage.
6. The intelligent teaching method based on artificial intelligence and intelligent dot matrix pen according to claim 4 is characterized in that: The steps of integrating the data collected from each terminal for deep learning analysis, mining the characteristics of students to provide personalized after-school services, and using blockchain to implement credit certification services include: Build a big data platform on the cloud server to integrate multi-source teaching data from various terminals; Use deep learning algorithm models to extract student personality characteristics from integrated multi-source teaching data; Perform collaborative filtering recommendations on student assignments to predict difficulties and trends for individual learning behaviors; Mining teaching rules, calculating the difficulty of knowledge points, and generating refined teaching suggestion data; Generate personalized learning paths and suggestions for each student; Generate secure academic achievement certificates on the blockchain platform to enable mutual recognition and exchange of certificates among different schools; Receive plan modification data submitted by students based on their own learning situation to modify their study plans, adjust students' study courses and learning progress according to the plan modification data, and realize flexible independent learning.
7. The intelligent teaching method based on artificial intelligence and intelligent dot pen according to claim 6 is characterized in that: The step of automatically extracting metadata from the teaching resources uploaded by the teacher and identifying structured information such as titles and keywords includes: Identify the document data in the teaching resources and extract text content; Extracting audio and video amplitude features from the video in the teaching resource, and identifying key frames and characters using a deep learning method; Establishing a topic model and extracting important words for the text content, the key frames and the characters to obtain structured information such as titles and keywords; Use pre-built knowledge graphs to represent structured information in an entity and relational manner; Utilize multi-source information from multiple devices to perform mutual verification and clustering to improve metadata.
8. The intelligent teaching method based on artificial intelligence and intelligent dot pen according to claim 7 is characterized in that: The courseware template is automatically generated on the knowledge point framework, and the courseware template includes the steps of automatically extracting and generating the outline, chapters, and content details, including: According to the metadata extracted from teaching resources, a directory tree structure framework of knowledge points is constructed; Prepare courseware templates and define the corresponding relationship model between titles and content modules at all levels; Automatically fill in the title level in the courseware template according to the directory tree structure framework and knowledge point framework of the knowledge point; Extract keywords and phrases from teaching resources to automatically generate chapter content outlines; Mining important paragraphs and teaching points in teaching resources, using automatic summarization to generate each detailed content; Identify difficult modules based on learning data and give them annotations for enhanced processing; Configure the courseware layout, style, and interaction details to generate personalized, publishable courseware.
9. The intelligent teaching method based on artificial intelligence and intelligent dot pen according to claim 8 is characterized in that: The step of automatically splitting the video by video object detection technology, wherein each video subsection corresponds to a courseware node, and inserting the course video into the courseware template includes: An object detection neural network is used on the course video to identify the location of the teacher and the key teaching objects in the video; Track and match the time sequence of the teacher and key teaching objects in the course video, find the frames or time points where the course video can be naturally divided, and automatically divide the course video into multiple segments or sub-videos. The length of each video segment is automatically determined according to the degree of change of the object; Extract metadata of video object detection predictions; According to the courseware layout and the content of each cluster, automatically select appropriate video clips and embed them into the corresponding nodes of the courseware; Configure user cropping and editing functions to optimize the automatic insertion effect. After embedding, play the corresponding video on the courseware node as needed to collaborate on dual-screen learning.
10. A smart teaching system based on artificial intelligence and smart dot pen, characterized in that: include: Smart dot pen, cloud server, smart terminal and smart whiteboard; The cloud server is configured as follows: When the smart dot matrix pen writes on the dot matrix code paper, obtaining writing behavior data collected by the smart dot matrix pen; Based on the deep learning algorithm, the handwriting image in the collected writing behavior data is used for target detection. After image enhancement, the deep convolutional neural network model is used to decode the dot matrix code in the image and determine the handwriting coordinate information. Based on the handwriting coordinate information, using a machine learning algorithm to determine the student's answer data; Determine the student's learning emotion data based on the collected student's physiological data and voice data in combination with the writing behavior data; Automatically generate courseware by automatically identifying teaching resources, and automatically generate courseware details using video splitting technology; Use smart terminals and smart whiteboards to enable teachers and students to interact and participate in offline immersive virtual classrooms; Integrate data collected from various terminals for deep learning analysis, explore students' characteristics and patterns to provide personalized after-school services, and use blockchain to implement credit certification services.
Citation Information
Patent Citations
Teaching live broadcast method, device, computer equipment and storage medium
CN112528799A
Intelligent classroom interaction system and method
CN117037552A
Method and system for intelligently identifying corrected handwriting
CN118071553A
Analysis method and device for video teaching content
CN118447435A
Cited By
Intelligent accounting teaching method combined with behavior recognition
CN120781297A
Grouping whiteboard control method for automatic split-screen display
CN120950161A
A grouping whiteboard control method for automatic split-screen display
CN120950161B
Intelligent classroom auxiliary teaching management system based on intelligent handwritten data real-time feedback
CN121809863A
Intelligent classroom auxiliary teaching management system based on real-time feedback of intelligent handwriting data
CN121809863B
