Teaching data processing method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention discloses a teaching data processing method and device, a storage medium and electronic equipment, and the method comprises the steps: detecting a hand drawing action of a speaker user in a teaching conference scene, collecting hand drawing action data corresponding to the hand drawing action, and obtaining reference teaching scene information, and performing hand-drawn content identification based on the hand-drawn action data and the reference teaching scene information to obtain target hand-drawn content, and synchronizing the target hand-drawn content to the participant user side, so that the participant user side displays the target hand-drawn content.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a teaching data processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] With the rapid development of computer technology, remote video conferencing teaching has become increasingly popular. However, in video teaching, hand-drawn content (such as formulas, graphs, and structural diagrams) by the teacher, as the main speaker, has become a crucial part of knowledge transmission. This is especially true in subjects like mathematics, physics, and chemistry, where graphics and formulas are essential for expression; hand-drawn content effectively helps students understand complex concepts.

[0003] However, traditional video teaching lacks the ability to intelligently recognize and process hand-drawn content. Teachers' hand-drawn content usually relies on camera capture, which is easily affected by factors such as video clarity, hand-drawn accuracy, and transmission delay, making it difficult for students to clearly and accurately acquire knowledge points, thus affecting the teaching effect. Summary of the Invention

[0004] This specification provides a teaching data processing method, apparatus, storage medium, and electronic device, the technical solutions of which are as follows:

[0005] Firstly, embodiments of this specification provide a method for processing teaching data, the method comprising:

[0006] In a teaching meeting scenario, the hand-drawing gestures of the main speaker are detected, and the hand-drawing gesture data corresponding to the hand-drawing gestures are collected, along with reference teaching scenario information.

[0007] The target hand-drawn content is obtained by recognizing the hand-drawn action data and reference teaching scenario information.

[0008] The target hand-drawn content is synchronized to the participating user's terminal so that the participating user's terminal displays the target hand-drawn content.

[0009] In one feasible implementation, obtaining the reference teaching scenario information includes:

[0010] Extract the hand-drawn action change features from the hand-drawn action data, and determine the target hand-drawn recognition difficulty for the hand-drawn recognition model based on the hand-drawn action change features;

[0011] Based on the difficulty of the target hand-drawn recognition, a combination of target auxiliary recognition data is determined, and reference teaching scenario information corresponding to the target auxiliary recognition data combination is obtained.

[0012] In one feasible implementation, determining the auxiliary recognition data combination based on the target hand-drawn recognition difficulty includes:

[0013] The auxiliary recognition data combination corresponding to the target hand-drawn recognition difficulty is queried in the preset difficulty combination mapping relationship. The preset difficulty combination mapping relationship includes multiple auxiliary recognition data combinations corresponding to hand-drawn recognition difficulty.

[0014] In one feasible implementation, the step of identifying the target hand-drawn content based on the hand-drawn action data and reference teaching scenario information includes:

[0015] Determine hand-drawing recognition task prompts based on the hand-drawing motion data and the reference teaching scenario information;

[0016] The hand-drawn action data, reference teaching scenario information, and hand-drawn recognition task prompts are input into the hand-drawn recognition model, and the target hand-drawn content is obtained by recognizing the hand-drawn content through the hand-drawn recognition model.

[0017] In one feasible implementation, the step of obtaining the target hand-drawn content through hand-drawn content recognition using the hand-drawn recognition large model includes:

[0018] The hand-drawn recognition model is used to perform action recognition on the hand-drawn action data to obtain initial hand-drawn content. The hand-drawn content description information of the initial hand-drawn content is determined. Based on the hand-drawn content description information, the hand-drawn association information of the reference teaching scenario information is mined to obtain teaching demonstration association information.

[0019] Based on the teaching demonstration association information, the initial hand-drawn content is adjusted to obtain the target hand-drawn content.

[0020] In one feasible implementation, the step of adjusting the initial hand-drawn content based on the teaching demonstration association information to obtain the target hand-drawn content includes:

[0021] Multiple teaching demonstration related content are identified from the teaching demonstration association information;

[0022] From the multiple teaching demonstration related content, determine the target teaching demonstration related content that matches the initial hand-drawn content;

[0023] Based on the target teaching demonstration related content, the initial hand-drawn content is modified to obtain the target hand-drawn content.

[0024] In one feasible implementation, the step of modifying the initial hand-drawn content based on the target teaching demonstration-related content to obtain the target hand-drawn content includes:

[0025] The content structure information and standard knowledge point content in the target teaching demonstration related content are determined, and the initial hand-drawn content is structurally optimized based on the content structure information to form the first hand-drawn content;

[0026] Based on the standard content of the aforementioned knowledge points, the first hand-drawn content is optimized in terms of content details to obtain the second hand-drawn content;

[0027] The target hand-drawn content is obtained by performing visual enhancement processing on the second hand-drawn content.

[0028] Secondly, embodiments of this specification provide a teaching data processing device, the device comprising:

[0029] The data acquisition module is used to detect the hand-drawing actions of the main speaker in a teaching meeting scenario, collect the hand-drawing action data corresponding to the hand-drawing actions, and obtain reference teaching scenario information.

[0030] The content recognition module is used to identify the target hand-drawn content based on the hand-drawn action data and reference teaching scenario information.

[0031] The content synchronization module is used to synchronize the target hand-drawn content with the participating user terminals so that the participating user terminals can display the target hand-drawn content.

[0032] In one feasible implementation, the data acquisition module is used for:

[0033] Extract the hand-drawn action change features from the hand-drawn action data, and determine the target hand-drawn recognition difficulty for the hand-drawn recognition model based on the hand-drawn action change features;

[0034] Based on the difficulty of the target hand-drawn recognition, a combination of target auxiliary recognition data is determined, and reference teaching scenario information corresponding to the target auxiliary recognition data combination is obtained.

[0035] In one feasible implementation, the data acquisition module is used for:

[0036] The auxiliary recognition data combination corresponding to the target hand-drawn recognition difficulty is queried in the preset difficulty combination mapping relationship. The preset difficulty combination mapping relationship includes multiple auxiliary recognition data combinations corresponding to hand-drawn recognition difficulty.

[0037] In one feasible implementation, the content recognition module is used to:

[0038] Determine hand-drawing recognition task prompts based on the hand-drawing motion data and the reference teaching scenario information;

[0039] The hand-drawn action data, reference teaching scenario information, and hand-drawn recognition task prompts are input into the hand-drawn recognition model, and the target hand-drawn content is obtained by recognizing the hand-drawn content through the hand-drawn recognition model.

[0040] In one feasible implementation, the content recognition module is used to:

[0041] The hand-drawn recognition model is used to perform action recognition on the hand-drawn action data to obtain initial hand-drawn content. The hand-drawn content description information of the initial hand-drawn content is determined. Based on the hand-drawn content description information, the hand-drawn association information of the reference teaching scenario information is mined to obtain teaching demonstration association information.

[0042] Based on the teaching demonstration association information, the initial hand-drawn content is adjusted to obtain the target hand-drawn content.

[0043] In one feasible implementation, the content recognition module is used to:

[0044] Multiple teaching demonstration related content are identified from the teaching demonstration association information;

[0045] From the multiple teaching demonstration related content, determine the target teaching demonstration related content that matches the initial hand-drawn content;

[0046] Based on the target teaching demonstration related content, the initial hand-drawn content is modified to obtain the target hand-drawn content.

[0047] In one feasible implementation, the content recognition module is used to:

[0048] The content structure information and standard knowledge point content in the target teaching demonstration related content are determined, and the initial hand-drawn content is structurally optimized based on the content structure information to form the first hand-drawn content;

[0049] Based on the standard content of the aforementioned knowledge points, the first hand-drawn content is optimized in terms of content details to obtain the second hand-drawn content;

[0050] The target hand-drawn content is obtained by performing visual enhancement processing on the second hand-drawn content.

[0051] Thirdly, embodiments of this specification provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.

[0052] Fourthly, embodiments of this specification provide an electronic device that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0053] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0054] In one or more embodiments of this specification, in a teaching conference scenario, the electronic device detects the hand-drawn gestures of the presenter, collects the corresponding hand-drawn gesture data and obtains reference teaching scenario information, identifies the target hand-drawn content based on the hand-drawn gesture data and reference teaching scenario information, and synchronizes the target hand-drawn content to the participating users' terminals so that the participating users' terminals can display the target hand-drawn content. This effectively solves the problem of intelligent collection, accurate identification, and synchronous display of teachers' hand-drawn content in remote teaching. This technical solution can not only capture and identify teachers' hand-drawn content in real time, but also synchronously display the hand-drawn content to all participants, making the knowledge transfer in remote teaching clearer and more accurate, and helping to improve teaching effectiveness and students' learning experience. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of a teaching data processing system provided in the embodiments of this specification;

[0057] Figure 2 This is a flowchart illustrating a teaching data processing method provided in an embodiment of this specification;

[0058] Figure 3 This is a flowchart illustrating a method for obtaining reference teaching scenario information provided in the embodiments of this specification;

[0059] Figure 4 This is a flowchart illustrating a hand-drawn content recognition method provided in the embodiments of this specification;

[0060] Figure 5 This is a flowchart illustrating a hand-drawn content recognition process provided in the embodiments of this specification;

[0061] Figure 6 This is a schematic diagram of the structure of a teaching data processing device provided in the embodiments of this specification;

[0062] Figure 7 This is a schematic diagram of the structure of an electronic device provided in the embodiments of this specification;

[0063] Figure 8 This is a schematic diagram of the operating system and user space structure provided in the embodiments of this specification;

[0064] Figure 9 yes Figure 8 Architecture diagram of the Android operating system in China;

[0065] Figure 10 yes Figure 8 Architecture diagram of the iOS operating system. Detailed Implementation

[0066] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0067] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this specification, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this specification based on the specific circumstances. Furthermore, in the description of this specification, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0068] Please see Figure 1 This is a schematic diagram of a teaching data processing system provided in this manual. Figure 1 As shown, the teaching data processing system may include at least a client cluster and a service platform 100.

[0069] The client cluster may include at least one client, such as Figure 1 As shown, it specifically includes client 1 corresponding to user 1, client 2 corresponding to user 2, ..., client n corresponding to user n, where n is an integer greater than 0.

[0070] Each client in a client cluster can be an electronic device with communication capabilities, including but not limited to: wearable devices, handheld devices, personal computers, tablets, in-vehicle devices, smartphones, computing devices, or other processing devices connected to a wireless modem. Electronic devices may have different names in different networks, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, electronic device, wireless communication device, user agent or user device, cellular phone, cordless phone, personal digital assistant (PDA), and electronic devices in 5G networks or future evolved networks.

[0071] The service platform 100 can be a standalone server device, such as a rack-mount, blade, tower, or cabinet-type server device, or a workstation, mainframe, or other hardware device with strong computing power; or it can be a server cluster composed of multiple servers. The servers in the service cluster can be composed in a symmetrical manner, wherein each server is functionally and hierarchically equivalent in the transaction chain, and each server can provide services independently. The independent provision of services can be understood as not requiring the assistance of other servers.

[0072] In one or more embodiments of this specification, the service platform 100 can establish a communication connection with at least one client in the client cluster, and complete the data interaction during the teaching data processing based on the communication connection. For example, the service platform 100 can synchronize the target hand-drawn content obtained by the teaching data processing method of this specification to the client (equivalent to the participating user terminal), and the participating user terminal can display the target hand-drawn content to the participating user.

[0073] It should be noted that the service platform 100 establishes a communication connection with at least one client in the client cluster via a network for interactive communication. This network can be a wireless network or a wired network. Wireless networks include, but are not limited to, cellular networks, wireless LANs, infrared networks, or Bluetooth networks. Wired networks include, but are not limited to, Ethernet, universal serial bus (USB), or controller area networks. In one or more embodiments of the specification, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network (such as target compressed packets). Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0074] The teaching data processing system embodiments provided in this specification and the teaching data processing methods described in one or more embodiments belong to the same concept. The execution entity corresponding to the teaching data processing method involved in one or more embodiments of this specification can be the aforementioned service platform 100; the execution entity corresponding to the teaching data processing method involved in one or more embodiments of this specification can also be the electronic device corresponding to the client, specifically determined based on the actual application environment. The implementation process of the teaching data processing system embodiments can be detailed in the following method embodiments, and will not be repeated here.

[0075] The present specification will now be described in detail with reference to specific embodiments.

[0076] In one embodiment, such as Figure 2 As shown, a teaching data processing method is proposed. This method can be implemented using a computer program and can run on a teaching data processing device based on the von Neumann architecture. The computer program can be integrated into an application or run as a standalone tool application. The teaching data processing device can be an electronic device.

[0077] Specifically, the teaching data processing method includes:

[0078] S102: In a teaching meeting scenario, the hand-drawing action of the main speaker is detected, and the hand-drawing action data corresponding to the hand-drawing action is collected, as well as the reference teaching scenario information is obtained.

[0079] Hand-drawing actions: In teaching meetings, this refers to the process of the presenter (such as a teacher) drawing on a blackboard, touchscreen, tablet, or in the air. Hand-drawing actions can include writing formulas, drawing diagrams, and describing structural illustrations.

[0080] Hand-drawn motion data: This data collects specific action information of the presenter during the hand-drawing process, including hand movement trajectory, angle, speed, force, and stroke order. This hand-drawn motion data is used to accurately identify the content drawn by the presenter.

[0081] Reference teaching scenario information: Relevant information extracted by the system from the current teaching scenario, including the course topic, knowledge points, context, and key terms. This reference teaching scenario information helps the electronic device understand the current teaching background and provides guidance for subsequent recognition.

[0082] As an example, the system first uses detection devices such as cameras and touchscreens to detect the speaker's hand gestures during the teaching session, and then collects the corresponding hand gesture data in real time. For example, it captures the hand gesture trajectory of the teacher drawing geometric shapes on the touchscreen.

[0083] Simultaneously, the system acquires teaching context information by analyzing the current course content and keywords used by the lecturer to establish contextual references. For example, if the system identifies the teaching context as "physics and mechanics," it will record this contextual information for reference in subsequent identification processes.

[0084] For example, in a geometry lesson, the presenter is drawing a triangle on a touchscreen. The system collects the presenter's hand-drawing motion data through the touchscreen, including the drawing trajectory and stroke order. Simultaneously, the system detects that the current teaching context is the geometry topic of "similar triangles."

[0085] S104: Based on the hand-drawn action data and reference teaching scenario information, the hand-drawn content is identified to obtain the target hand-drawn content;

[0086] Hand-drawn content recognition: Based on hand-drawn action data and teaching context information, electronic devices use hand-drawn recognition models to analyze the characteristics of hand-drawn actions, thereby recognizing specific information about the hand-drawn content (such as formulas, graphics, diagrams, etc.).

[0087] Target hand-drawn content: The hand-drawn content finally determined by the recognition model is a digital and optimized version of the specific content drawn by the speaker. It has a clear structure and accurate graphic or textual information, making it easy to show to the participants.

[0088] As an illustration, the collected hand-drawn motion data and teaching context information are input into the hand-drawn recognition model. The model makes an initial judgment based on the trajectory and shape information of the hand-drawn data, and then performs precise recognition by combining the teaching context. For example, if a triangle is recognized in a geometric scene, and the geometric theme is "similar triangles," the model will determine that it is related to similar triangles. The hand-drawn recognition model then generates the recognition result, i.e., the target hand-drawn content, which undergoes further digitization and visual optimization processing for subsequent synchronization to the participants' devices.

[0089] Example: In a geometry lesson, the teacher's hand-drawn action data (i.e., the trajectory and angle of the triangle) and contextual information are input into the hand-drawn recognition model. The model determines that the teacher has drawn a triangle. Combining the contextual information of similar triangles, the model further identifies that this may be a schematic diagram used to explain similarity relationships. After optimization, the hand-drawn recognition model generates a clear triangle image and marks the relevant vertices and edges to generate the target hand-drawn content.

[0090] Furthermore, the hand-drawn recognition model can be trained based on a machine learning model. An initial hand-drawn recognition model for the hand-drawn recognition scenario is created in advance using a machine learning model, and the model is trained using sample data (sample hand-drawn action data and sample reference teaching scenario information).

[0091] It should be noted that the machine learning models involved in one or more embodiments of this specification include, but are not limited to, fitting one or more of the following machine learning models: Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Networks (RNN), embedding models, Gradient Boosting Decision Tree (GBDT) models, and Logistic Regression (LR) models.

[0092] S106: Synchronize the target hand-drawn content with the participating user terminal so that the participating user terminal displays the target hand-drawn content.

[0093] Participant user terminals: All devices of students or attendees participating in the conference, including computers, tablets, and mobile phones, are capable of receiving and displaying the content shared by the speaker.

[0094] In a demonstrative manner, the identified target hand-drawn content is sent to all participating users' devices. Through data transmission and synchronized display technology, the content just hand-drawn by the teacher will be displayed simultaneously on the screens of participating users. The participating users' devices will display the target hand-drawn content optimized by the system, making it clear and easy for students to understand.

[0095] In one or more embodiments of this specification, in a teaching conference scenario, the electronic device detects the hand-drawn gestures of the presenter, collects the corresponding hand-drawn gesture data and obtains reference teaching scenario information, identifies the target hand-drawn content based on the hand-drawn gesture data and reference teaching scenario information, and synchronizes the target hand-drawn content to the participating users' terminals so that the participating users' terminals can display the target hand-drawn content. This effectively solves the problem of intelligent collection, accurate identification, and synchronous display of teachers' hand-drawn content in remote teaching. This technical solution can not only capture and identify teachers' hand-drawn content in real time, but also synchronously display the hand-drawn content to all participants, making the knowledge transfer in remote teaching clearer and more accurate, and helping to improve teaching effectiveness and students' learning experience.

[0096] Please see Figure 3 , Figure 3 This is a flowchart illustrating a method for obtaining reference teaching scenario information as proposed in this specification. The following implementation method can be used to perform the acquisition of reference teaching scenario information:

[0097] S3002: Extract the hand-drawn action change features from the hand-drawn action data, and determine the target hand-drawn recognition difficulty for the hand-drawn recognition model based on the hand-drawn action change features;

[0098] Hand-drawn gesture variation characteristics: These refer to the information about the changes in the presenter's gestures during the hand-drawing process, including the speed, direction, continuity, complexity of curves, and pressure applied to strokes. These characteristics help the system determine the complexity of the hand-drawn content.

[0099] Hand-drawn drawing recognition difficulty: The recognition complexity is assessed based on the characteristics of hand-drawn movements. Complex hand-drawn content (such as multi-stroke formulas or intricate graphics) is generally more difficult to recognize, while simple graphics or single formulas are easier to recognize.

[0100] To illustrate, the system first extracts the hand-drawn motion variation features from the hand-drawn motion data. For example, when a presenter draws a complex set of curves or multiple strokes (such as chemical structural formulas or geometric figures), the system detects a high degree of motion variation complexity. Based on these hand-drawn motion variation features, the system assesses the difficulty of hand-drawn recognition. For instance, the system can use machine learning algorithms to analyze the recognition difficulty corresponding to different combinations of features in the hand-drawn motion variation features. Through this process, the system assigns a "recognition difficulty level" to the hand-drawn content to provide a reference for subsequent steps.

[0101] For example, suppose the presenter is drawing a chemical molecule structure with multiple rings. The electronic device assesses the hand-drawn structure as having high recognition difficulty based on motion variation characteristics (such as the complexity of curves and the overlap of strokes). Therefore, the electronic device will mark the recognition difficulty as "high" and call on additional auxiliary data in subsequent steps to ensure recognition accuracy.

[0102] S3004: Determine the target auxiliary recognition data combination based on the target hand-drawn recognition difficulty, and obtain the reference teaching scenario information corresponding to the target auxiliary recognition data combination.

[0103] Target-assisted identification data combination: This refers to an additional combination of data sources used to improve identification accuracy when the identification difficulty is high. The auxiliary identification data combination can include data types related to the current hand-drawn content (such as teaching context data), previous course content data, and similar knowledge point data.

[0104] Reference teaching scenario information: Based on the various auxiliary recognition data types in the auxiliary recognition data combination, the auxiliary recognition data information under each auxiliary recognition data type is obtained, and all the auxiliary recognition data information is integrated to obtain the reference teaching scenario information;

[0105] Reference teaching context information includes, but is not limited to, one or more of the following: the topic of the current course, previous similar content (such as a feature library of similar formulas or graphs), key teaching terms or diagrams, etc. This information provides the hand-drawn recognition model with more background information when analyzing hand-drawn content, improving the accuracy of recognition.

[0106] For example, electronic devices can dynamically assess the recognition difficulty based on the changing characteristics of hand-drawn movements and flexibly retrieve target auxiliary recognition data combinations. This difficulty-based auxiliary recognition method not only improves the recognition accuracy of complex hand-drawn content but also provides richer background information for the drawing recognition model, enabling hand-drawn content in teaching to be displayed more accurately and clearly to participating users. This helps participating users accurately understand knowledge points in real-time teaching and improves the overall efficiency of the intelligent teaching system.

[0107] In one feasible implementation, determining the auxiliary recognition data combination based on the target hand-drawn recognition difficulty includes:

[0108] The auxiliary recognition data combination corresponding to the target hand-drawn recognition difficulty is queried in the preset difficulty combination mapping relationship. The preset difficulty combination mapping relationship includes multiple auxiliary recognition data combinations corresponding to hand-drawn recognition difficulty.

[0109] Preset Difficulty Combination Mapping: A pre-defined mapping table is used to associate different hand-drawn recognition difficulty levels with specific auxiliary recognition data combinations. The mapping table contains multiple difficulty levels and their corresponding auxiliary data combinations, facilitating quick lookup and retrieval by the system during the recognition process.

[0110] Auxiliary recognition data combination: A set of additional data sources pre-selected by electronic devices based on the recognition difficulty, including one or more fittings such as teaching context information, course materials, and key concept libraries, to help the hand-drawn recognition model analyze and understand hand-drawn content more accurately.

[0111] As an illustration, the electronic device, based on the target hand-drawn recognition difficulty determined in the previous step (S3002), finds the corresponding auxiliary recognition data combination by querying a preset difficulty combination mapping relationship (table). The mapping table associates different difficulty levels with multiple auxiliary recognition data sources to ensure that the system can dynamically select the optimal data combination according to the actual recognition difficulty.

[0112] As an illustration, the difficulty combination mapping relationship can be set so that only basic course content is called when recognizing low-difficulty data, while detailed knowledge point base and multiple reference scenario information are called when recognizing high-difficulty data. This mapping process quickly locates the required auxiliary data combination, so as to provide more sufficient data support for the hand-drawn recognition model in the next recognition process.

[0113] For example, in a teaching scenario, if the instructor is detected drawing a complex organic chemical molecule structure by hand, its recognition difficulty is assessed as "high." Based on the difficulty combination mapping table, the corresponding auxiliary recognition data combination is retrieved, which includes:

[0114] The current course's theme and key concepts (e.g., "Fundamentals of Organic Chemistry")

[0115] A library of examples of similar molecular structures (such as benzene rings and aldehydes).

[0116] Student feedback and questions regarding organic chemistry submitted in previous lessons.

[0117] Furthermore, this combined data is used as supplementary information for the hand-drawn recognition model during the analysis process. In this way, the recognition model can not only analyze the changing characteristics of hand-drawn movements, but also refer to common organic chemical structures in teaching contexts and common questions from students, thereby more accurately identifying complex molecular structures.

[0118] In the embodiments described in this specification, by pre-mapping the difficulty of hand-drawn content recognition with the combination of auxiliary recognition data into a relational table, the system can dynamically and accurately select the most suitable combination of auxiliary data for the current hand-drawn content during the recognition process. This difficulty-based mapping query mechanism makes the recognition of hand-drawn content more efficient and intelligent, especially in the recognition of complex content, providing the system with more sufficient contextual data support, thereby significantly improving recognition accuracy and teaching effectiveness.

[0119] Please see Figure 4 , Figure 4 This is a flowchart illustrating a hand-drawn content recognition method proposed in this specification. The hand-drawn recognition model can be a large-scale hand-drawn recognition model obtained by adapting a basic large-scale language model to the hand-drawn recognition scenario. The process of performing hand-drawn content recognition based on the hand-drawn action data and reference teaching scenario information to obtain the target hand-drawn content can be implemented as follows:

[0120] Hand-drawn recognition models are formed by adapting basic large-scale language models (LLMs) to specific scenarios, designed to handle hand-drawn recognition tasks. Large-scale language models possess powerful natural language understanding and generation capabilities. By training and fine-tuning them for specific scenarios, the model can be adapted to the needs of hand-drawn recognition. This adaptation process typically involves training an initial large-scale hand-drawn recognition model based on sample data, resulting in a trained large-scale hand-drawn recognition model that can better understand the features and context of hand-drawn content.

[0121] S4002: Determine hand-drawn recognition task prompts for the hand-drawn motion data and the reference teaching scenario information;

[0122] Hand-drawn drawing recognition task prompts: These are prompts or instructions used to guide the hand-drawn drawing recognition model to focus on the task of recognizing the current hand-drawn content. The prompts include the category of the hand-drawn content, the focus of the recognition task, and the teaching context, helping the model focus on the specific recognition task within a broad knowledge base.

[0123] In a schematic manner, hand-drawn action data and reference teaching context information are analyzed to extract key elements for the current recognition task. For example, based on the type of hand-drawn content (such as formulas or diagrams) and the teaching context (such as physics, chemistry, or mathematics), a specific prompt word is generated so that the hand-drawn recognition model can quickly understand the specific requirements of the current task.

[0124] The generation of prompts is typically based on templates or preset rules, such as "identify the structure of the current chemical molecule" and "determine similarity based on the content of geometric figures." The generated prompts clearly define the current recognition target, enabling the model to complete the recognition task more accurately.

[0125] Example: Suppose the current hand-drawn content is a chemical structural formula, and the teaching context is a "Fundamentals of Organic Chemistry" course. The system-generated hand-drawn recognition task prompt might be: "Identify and analyze the currently hand-drawn organic chemical molecular structure," helping the model clarify that the goal is to identify complex molecular structures and associate them with chemical background information.

[0126] S4004: Input the hand-drawn action data, reference teaching scenario information, and hand-drawn recognition task prompts into the hand-drawn recognition big model, and obtain the target hand-drawn content by recognizing the hand-drawn content through the hand-drawn recognition big model.

[0127] For example, hand-drawn action data, reference teaching scenario information, and generated task prompts are input into the hand-drawn recognition model. Based on the input data and prompts, the model first analyzes the characteristics of the hand-drawn actions, then performs a deeper understanding by combining the teaching scenario information, thereby accurately identifying the target content. The recognition process is completed through the multi-layered understanding capabilities of the hand-drawn recognition model: the model first extracts the basic features of the hand-drawn data, then combines the guidance of the task prompts and the context provided by the reference teaching scenario to generate accurate target hand-drawn content. The final output content undergoes structured optimization to ensure its clarity and completeness, making it suitable for direct display to participating users.

[0128] Optionally, the following illustrates the training process of a large-scale hand-drawn recognition model:

[0129] In one feasible implementation, the large hand-drawn recognition model can be obtained by training a basic large (language) model specifically for hand-drawn recognition scenarios. The basic large (language) model (LLM) is an artificial intelligence content generation model designed to understand and generate human language. Trained on large amounts of data, the basic large model can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more.

[0130] Optionally, the basic large model can be the Tongyi Qianwen large model, the Bailing large model, the GPT system large model, etc.

[0131] In some embodiments, a pre-trained basic model can be obtained and adapted to the hand-drawn recognition scenario to obtain a hand-drawn recognition model. However, directly applying the basic model to hand-drawn recognition processing in a typical hand-drawn recognition scenario is difficult to adapt to new hand-drawn recognition scenarios. Therefore, an initial hand-drawn recognition model is created by first obtaining the basic model and then obtaining sample data for the new hand-drawn recognition scenario (the sample data consists of sample hand-drawn action data and sample reference teaching scenario information). Since the basic model is usually a pre-trained AIGC model with content generation capabilities, this specification only needs to adapt it to the hand-drawn recognition scenario. Specifically, the initial hand-drawn recognition model can be fine-tuned using the sample data. After the model fine-tuning training is completed, a hand-drawn recognition model adapted to the hand-drawn recognition scenario is obtained.

[0132] For example, the training process of a large-scale hand-drawn character recognition model is illustrated below:

[0133] 1. Obtain sample training data from sample users in the hand-drawn recognition scenario. The sample training data includes sample hand-drawn action data and sample reference teaching scenario information; use expert-end services to label the hand-drawn content of the sample training data.

[0134] 2. Create an initial large-scale model for hand-drawn recognition scenarios using a basic large-scale model;

[0135] 3. Using sample training data, determine the hand-drawn recognition task prompts for the sample training data, and input the hand-drawn recognition task prompts and sample training data into the initial hand-drawn recognition large model for at least one round of model training;

[0136] In each round of model training, the initial hand-drawn recognition model is developed as follows: Action recognition is performed on the (sample) hand-drawn action data to obtain initial (sample) hand-drawn content; the descriptive information of the initial (sample) hand-drawn content is determined; based on the descriptive information, hand-drawn association information is mined from the (sample) reference teaching scenario information to obtain (sample) teaching demonstration association information; based on the teaching demonstration association information, the initial (sample) hand-drawn content is adjusted to obtain predicted hand-drawn content; based on the predicted hand-drawn content and hand-drawn content labels, the model loss is calculated using a model loss calculation formula; and the model parameters of the initial hand-drawn recognition model are adjusted using the model loss calculation formula until the initial hand-drawn recognition model meets the model termination conditions.

[0137] Optionally, the model loss calculation formula can be one or more fitting methods such as hinge loss function, contrast loss function, Euclidean distance loss function, and cross-entropy loss function from related techniques.

[0138] Optionally, the model's training termination conditions may include, for example, the loss function value being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold. Specific training termination conditions can be determined based on actual circumstances and are not specifically limited here.

[0139] In this specification, by identifying task prompts for hand-drawn content recognition and inputting them along with hand-drawn data into a customized hand-drawn content recognition model, this implementation method effectively improves the model's accuracy in recognizing hand-drawn content and its contextual understanding capabilities. Utilizing task prompts and teaching context information, the model can focus on recognizing key content, providing clearer and more accurate results, especially in complex hand-drawn content recognition scenarios, thereby enhancing the intelligence and experience of classroom teaching and conference presentations.

[0140] Please refer to Figure 5 , Figure 5 This is a flowchart illustrating a hand-drawn content recognition process. The process involves using the aforementioned hand-drawn content recognition model to identify the target hand-drawn content, which can be achieved in the following ways:

[0141] S5002: The hand-drawn action data is used to perform action recognition through the hand-drawn action recognition model to obtain the initial hand-drawn content;

[0142] Initial hand-drawn content: The first recognition result generated by the hand-drawn recognition model based on the features of the hand-drawn action data is a basic recognition version of the hand-drawn content, without further optimization or adjustment.

[0143] The collected hand-drawn motion data is input into a large-scale hand-drawn recognition model. The model analyzes the motion based on features such as trajectory, force, and angle, identifying the initial version of the hand-drawn content. This process is equivalent to the large-scale hand-drawn recognition model performing a basic interpretation of the hand-drawn motion, extracting obvious graphics, symbols, or formulas from the hand-drawn content.

[0144] Through feature analysis and knowledge matching of the hand-drawn recognition model, the hand-drawn recognition model generates an initial recognition result, namely "initial hand-drawn content", which provides basic data for subsequent further analysis and optimization.

[0145] Example: Suppose a teacher hand-draws a geometric figure during a lesson, such as a diagram containing parallel lines and angle labels. The system inputs this hand-drawn action data into a hand-drawn recognition model, which identifies the initial geometric structure, such as the initial hand-drawn content of "two parallel lines and an angle".

[0146] S5004: Determine the hand-drawn content description information of the initial hand-drawn content, and perform hand-drawn association information mining processing on the reference teaching scenario information based on the hand-drawn content description information to obtain teaching demonstration association information;

[0147] Hand-drawn content description information: A characteristic description of the initial hand-drawn content, used to further understand and identify the details of the hand-drawn content. The description information includes the type of content, structural features, knowledge point tags, etc.

[0148] Teaching demonstration related information: Based on the description information of hand-drawn content and the information of reference teaching scenarios, the related data is mined to provide knowledge background, detailed information and contextual associations related to the hand-drawn content, so as to optimize the subsequent recognition process.

[0149] As an illustration, after the large-scale hand-drawn recognition model generates initial hand-drawn content, it further extracts its feature descriptions to form hand-drawn content description information. For example, it determines the content type (such as geometric figures, physical diagrams), feature points (such as angles, intersection points), and knowledge point tags (such as parallel, perpendicular, similar).

[0150] Then, the hand-drawn illustration recognition model mines relevant teaching demonstration information based on the hand-drawn content description information in the reference teaching context to provide more background support. For example, in a mathematics teaching context, if the description information is displayed as "relationship between parallel lines and angles", the system will obtain formulas, definitions, and theorems related to the topic to help refine the subsequent recognition.

[0151] For example, in a geometry lesson scenario, the hand-drawn sketch recognition model identifies the initial hand-drawn content as "two parallel lines and an included angle," and describes it as "the relationship between parallel lines and angles in a geometric figure." The model further extracts relevant theorem information (such as alternate angles within parallel lines being equal) within the teaching context, forming related information and providing a basis for subsequent adjustments.

[0152] S5006: Based on the teaching demonstration association information, the initial hand-drawn content is adjusted to obtain the target hand-drawn content.

[0153] Adjusting hand-drawn content is the process of optimizing the initial hand-drawn content based on the information related to the teaching demonstration. This includes adding details, adjusting the structure, and enhancing the content to ensure that the content is accurate and clear.

[0154] The target hand-drawn content is the final hand-drawn content after adjustments. It is a clear and complete version of the content, which is easy for students to understand and learn.

[0155] The initial hand-drawn content is illustrated by comparing it with relevant information in the teaching demonstration, and the content is adjusted accordingly. For example, the proportions of geometric figures are adjusted, angle labels are added, or missing parts of chemical structures are supplemented. This process ensures that the target hand-drawn content better reflects the actual teaching context, allowing students to see a complete and optimized presentation on their devices. The adjusted content accurately reflects the teacher's hand-drawn sketches while also supplementing any potentially missed details, facilitating student comprehension.

[0156] For example, in a geometry lesson, the initial hand-drawn diagram was optimized by adding relevant information (the relationship between parallel lines and angles), supplementing it with specific markings for parallel lines and included angles, making the diagram more relevant to the lesson content. Ultimately, the system generated the "target hand-drawn content," displaying a complete diagram of the angle between parallel lines on the student's end, complete with angle annotations and relevant theorem information.

[0157] This illustrative process, involving the generation of initial hand-drawn content through a large-scale hand-drawn content recognition model, the extraction of descriptive information and the mining of teaching-related information, and content adjustment and optimization, intelligently transforms teachers' hand-drawn content into clear and structured target hand-drawn content. This process not only improves the accuracy of hand-drawn content recognition but also enhances the completeness and clarity of content presentation, thereby improving students' comprehension and experience during the teaching process.

[0158] In one feasible implementation, the step of adjusting the initial hand-drawn content based on the teaching demonstration association information to obtain the target hand-drawn content can be carried out in the following way:

[0159] A1. Identify multiple teaching demonstration related contents from the teaching demonstration related information;

[0160] Related content for teaching demonstrations: refers to specific knowledge points or content fragments extracted from related information. The system uses this content to supplement and optimize the hand-drawn recognition results.

[0161] Multiple potential related content items are extracted from the teaching demonstration's associated information to facilitate subsequent matching with the initial hand-drawn content. The teaching demonstration's associated content can include core knowledge points of the current course, examples, rules, or other information related to the hand-drawn content.

[0162] As an illustration, the content associated with the teaching demonstration may come from multiple sources, such as course materials, previous class notes, and theorem entries in a knowledge base. By aggregating this content, the large model can obtain broader contextual support during recognition, providing a foundation for subsequent matching and correction.

[0163] Example: Suppose the current teaching scenario is a geometry class, and the initial hand-drawn content involves two parallel lines and a straight line crossing the parallel lines. The system can extract multiple related contents from the associated information, such as the geometric theorems "alternate interior angles between parallel lines are equal," "corresponding angles between parallel lines are equal," and "corresponding angles are equal."

[0164] A2. Determine the target teaching demonstration content that matches the initial hand-drawn content from the multiple teaching demonstration related content;

[0165] Targeted teaching demonstration related content: This consists of content fragments or knowledge points selected from multiple related content sets that best match the initial hand-drawn content. It is used to guide the optimization of the hand-drawn content in the larger model, making it more closely aligned with the teaching objectives.

[0166] In a schematic manner, the descriptive information of the initial hand-drawn content is compared one by one with multiple extracted related contents to determine the target related content that best matches the current hand-drawn content. Through feature matching algorithms or semantic analysis, the large model filters out the related information that best matches the hand-drawn content. Once the target related content is determined, it will serve as a reference to guide the revision of the initial hand-drawn content. In this way, the large model can ensure that subsequent revisions are consistent with the teaching objectives, resulting in more accurate and standardized hand-drawn content.

[0167] A3. Based on the target teaching demonstration related content, the initial hand-drawn content is modified to obtain the target hand-drawn content.

[0168] Content revision: This is the process of supplementing, adjusting, and refining the initial hand-drawn content by linking it to the target audience, ensuring that the content is accurate, complete, and meets the teaching standards.

[0169] The target hand-drawn content is the final hand-drawn content generated after revision. It has a clear structure and a complete content description, making it easy to demonstrate in teaching.

[0170] This example illustrates how target-related content can be used to correct initial hand-drawn content, ensuring it meets teaching requirements. For instance, the larger model can adjust the proportions of the drawing based on correlation theorems, add appropriate angle markers, or optimize the line structure of the hand-drawn text for greater clarity and accuracy. After correction, the target hand-drawn content generated by the larger model can be clearly displayed on the student's end, making it easier for them to understand and master the relevant knowledge points. By incorporating teaching-related information for correction, the system can significantly improve the quality of hand-drawn content and teaching effectiveness.

[0171] In this manual, by filtering relevant content from teaching demonstration information, matching target relevant content, and making content corrections, the system can improve the teaching presentation effect of hand-drawn content while maintaining its original meaning. This adjustment method based on relevant information significantly optimizes the accuracy and visibility of hand-drawn content, ensuring that the content meets classroom teaching objectives, while facilitating students' intuitive and accurate understanding of the knowledge points being explained.

[0172] For example, the step of modifying the initial hand-drawn content based on the target teaching demonstration related content to obtain the target hand-drawn content includes:

[0173] 1. Determine the content structure information and standard knowledge point content in the target teaching demonstration related content, and perform structured optimization of the initial hand-drawn content based on the content structure information;

[0174] Content structure information: Information related to the knowledge point structure within the content associated with the teaching demonstration, including the arrangement of content, geometric shapes, formula structures, etc. This structural information ensures that the overall layout of the hand-drawn content conforms to standards and knowledge specifications.

[0175] Standard content for knowledge points: The core knowledge points and standardized expressions in the content related to the teaching objectives are used to ensure the academic accuracy of the content and its compliance with teaching requirements.

[0176] This process illustratively extracts content structure information and standard knowledge point content from the relevant content of the target teaching demonstration. Based on this content structure information, the hand-drawn recognition model performs structural optimization on the initial hand-drawn content to ensure it conforms to the structural standards of the corresponding knowledge points. For example, the arrangement of graphics and the format of formulas are adjusted to meet the needs of the teaching demonstration. The result of this structural optimization is the "first hand-drawn content," which is the initially optimized version of the content. This step ensures that the basic structure of the hand-drawn content aligns with the teaching objectives.

[0177] Example: Suppose the target teaching content involves the geometric concept of "alternate interior angles of parallel lines are equal". After the system extracts the structural information of the two parallel lines and their alternate interior angles, it adjusts the initial hand-drawn content to ensure that the two parallel lines are arranged in a standard parallel pattern and that the markings of the alternate interior angles conform to the standard structure, thus generating the "first hand-drawn content".

[0178] 2. Based on the standard content of the aforementioned knowledge points, the first hand-drawn content is optimized in terms of content details to obtain the second hand-drawn content;

[0179] Content detail optimization builds upon structured optimization by further supplementing and refining the details of the content. This includes adding annotations, symbols, and strengthening variable identification in formulas to make the content more complete and accurate. The result of this content detail optimization is "second hand-drawn content." This step ensures that the details of the hand-drawn content meet the teaching standards for the knowledge points, improving the accuracy and comprehensibility of the content.

[0180] Second hand-drawn content: A hand-drawn version with optimized details, which is more detailed, accurate, and complete than the first hand-drawn content.

[0181] The hand-drawn recognition model applies standard knowledge points to the initial hand-drawn content, supplementing and optimizing it with details. For example, it adds angle symbols to geometric figures and standard variable symbols to formulas, ensuring that the content meets the detailed standards required for teaching.

[0182] 3. Visual enhancement processing is performed on the second hand-drawn content to obtain the target hand-drawn content.

[0183] Visual enhancement processing optimizes the visual effects of secondhand hand-drawn content to ensure it is clear and easy to understand on student devices. This processing includes adjustments to lines, colors, contrast, and resolution to make the content more suitable for screen display.

[0184] Target hand-drawn content: The final hand-drawn content version after visual optimization has a clear display effect and high readability, making it easy for students to learn and understand.

[0185] This illustrative example demonstrates how a large-scale hand-drawn recognition model enhances the visual display of the second hand-drawn content. For instance, it adjusts line thickness, increases contrast, and ensures angles and markings are clearly visible. This visual optimization makes the target hand-drawn content more suitable for screen display, resulting in a clearer viewing experience for students. The final generated "target hand-drawn content" ensures clear display across different devices, meeting high standards for teaching presentations and enabling students to understand the content more intuitively.

[0186] Example: In a geometry lesson, the large-scale hand-drawn recognition model visually optimized the parallel lines and angle markings in the second hand-drawn content. This included thickening the parallel lines, enhancing the contrast of the angle symbols, and adjusting the spacing of the markings, ensuring the target hand-drawn content was clearly displayed on the screen. This allows students to easily identify details and understand the geometric relationships between alternate angles within parallel lines.

[0187] In one or more embodiments of this specification, through a triple process of structure optimization, detail optimization, and visual enhancement, the hand-drawn recognition model transforms initial hand-drawn content into target hand-drawn content that meets teaching standards. This process ensures that the content structure is reasonable, the details are complete, and the visuals are clear, greatly improving the presentation effect and students' comprehension experience. The final generated target hand-drawn content not only accurately reflects the teaching content but also possesses high readability suitable for screen display, providing students with better learning support.

[0188] The following will combine Figure 6 This document provides a detailed description of the teaching data processing device provided in the embodiments of this specification. It should be noted that... Figure 6 The teaching data processing device shown is used to execute this instruction manual. Figures 1-5 The methods shown in the embodiments are illustrated for ease of explanation, showing only the parts related to the embodiments of this specification. For specific technical details not disclosed, please refer to this specification. Figures 1-5 The example shown.

[0189] Please see Figure 6 This diagram illustrates the structure of a teaching data processing device according to an embodiment of this specification. The teaching data processing device 1 can be implemented as all or part of a device through software, hardware, or a combination of both. According to some embodiments, the teaching data processing device 1 includes a data acquisition module 11, a content recognition module 12, and a content synchronization module 13, specifically used for:

[0190] Data acquisition module 11 is used to detect the hand-drawing actions of the main speaker in a teaching meeting scenario, collect the hand-drawing action data corresponding to the hand-drawing actions, and obtain reference teaching scenario information.

[0191] Content recognition module 12 is used to perform hand-drawn content recognition based on the hand-drawn action data and reference teaching scenario information to obtain the target hand-drawn content;

[0192] The content synchronization module 13 is used to synchronize the target hand-drawn content to the participating user terminal so that the participating user terminal can display the target hand-drawn content.

[0193] In one feasible implementation, the data acquisition module 11 is used for:

[0194] Extract the hand-drawn action change features from the hand-drawn action data, and determine the target hand-drawn recognition difficulty for the hand-drawn recognition model based on the hand-drawn action change features;

[0195] Based on the difficulty of the target hand-drawn recognition, a combination of target auxiliary recognition data is determined, and reference teaching scenario information corresponding to the target auxiliary recognition data combination is obtained.

[0196] In one feasible implementation, the data acquisition module 11 is used for:

[0197] The auxiliary recognition data combination corresponding to the target hand-drawn recognition difficulty is queried in the preset difficulty combination mapping relationship. The preset difficulty combination mapping relationship includes multiple auxiliary recognition data combinations corresponding to hand-drawn recognition difficulty.

[0198] In one feasible implementation, the content recognition module 12 is used to:

[0199] Determine hand-drawing recognition task prompts based on the hand-drawing motion data and the reference teaching scenario information;

[0200] The hand-drawn action data, reference teaching scenario information, and hand-drawn recognition task prompts are input into the hand-drawn recognition model, and the target hand-drawn content is obtained by recognizing the hand-drawn content through the hand-drawn recognition model.

[0201] In one feasible implementation, the content recognition module 12 is used to:

[0202] The hand-drawn recognition model is used to perform action recognition on the hand-drawn action data to obtain initial hand-drawn content. The hand-drawn content description information of the initial hand-drawn content is determined. Based on the hand-drawn content description information, the hand-drawn association information of the reference teaching scenario information is mined to obtain teaching demonstration association information.

[0203] Based on the teaching demonstration association information, the initial hand-drawn content is adjusted to obtain the target hand-drawn content.

[0204] In one feasible implementation, the content recognition module 12 is used to:

[0205] Multiple teaching demonstration related content are identified from the teaching demonstration association information;

[0206] From the multiple teaching demonstration related content, determine the target teaching demonstration related content that matches the initial hand-drawn content;

[0207] Based on the target teaching demonstration related content, the initial hand-drawn content is modified to obtain the target hand-drawn content.

[0208] In one feasible implementation, the content recognition module 12 is used to:

[0209] The content structure information and standard knowledge point content in the target teaching demonstration related content are determined, and the initial hand-drawn content is structurally optimized based on the content structure information to form the first hand-drawn content;

[0210] Based on the standard content of the aforementioned knowledge points, the first hand-drawn content is optimized in terms of content details to obtain the second hand-drawn content;

[0211] It should be noted that the teaching data processing device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the teaching data processing method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the teaching data processing device and the teaching data processing method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0212] The example numbers in this specification are for descriptive purposes only and do not represent the superiority or inferiority of the examples.

[0213] In one or more embodiments of this specification, in a teaching conference scenario, the electronic device detects the hand-drawn gestures of the presenter, collects the corresponding hand-drawn gesture data and obtains reference teaching scenario information, identifies the target hand-drawn content based on the hand-drawn gesture data and reference teaching scenario information, and synchronizes the target hand-drawn content to the participating users' terminals so that the participating users' terminals can display the target hand-drawn content. This effectively solves the problem of intelligent collection, accurate identification, and synchronous display of teachers' hand-drawn content in remote teaching. This technical solution can not only capture and identify teachers' hand-drawn content in real time, but also synchronously display the hand-drawn content to all participants, making the knowledge transfer in remote teaching clearer and more accurate, and helping to improve teaching effectiveness and students' learning experience.

[0214] This specification also provides a computer storage medium that can store multiple instructions adapted to be loaded and executed by a processor as described above. Figures 1-5 The teaching data processing method described in the illustrated embodiment can be found in the following documentation for its specific execution process. Figures 1-5 The specific details of the illustrated embodiments will not be elaborated here.

[0215] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1-5 The teaching data processing method described in the illustrated embodiment can be found in the following documentation for its specific execution process. Figures 1-5 The specific details of the illustrated embodiments will not be elaborated here.

[0216] Please refer to Figure 7This diagram illustrates a structural block diagram of an electronic device provided in an exemplary embodiment of this specification. The electronic device in this specification may include one or more components such as a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, memory 120, input device 130, and output device 140 may be connected via the bus 150.

[0217] Processor 110 may include one or more processing cores. Processor 110 connects to various parts of the electronic device via various interfaces and lines, and performs various functions and processes data of electronic device 100 by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 110 may integrate one or more of the following: central processing unit (CPU), graphics processing unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately using a communication chip.

[0218] The memory 120 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 120 may include a non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described below, etc. The operating system may be the Android system, including systems deeply developed based on the Android system, the iOS system developed by Apple Inc., including systems deeply developed based on the iOS system, or other systems. The data storage area may also store data created by the electronic device during use, such as phonebook data, audio and video data, chat log data, etc.

[0219] See Figure 8 As shown, the memory 120 can be divided into operating system space and user space. The operating system runs in the operating system space, while native and third-party applications run in the user space. To ensure that different third-party applications can achieve good running performance, the operating system allocates corresponding system resources for each application. However, different application scenarios within the same third-party application have different requirements for system resources. For example, in local resource loading scenarios, third-party applications have high requirements for disk read speed; in animation rendering scenarios, third-party applications have high requirements for GPU performance. Since the operating system and third-party applications are independent of each other, the operating system often cannot promptly perceive the current application scenario of a third-party application, resulting in the operating system's inability to adapt system resources accordingly to the specific application scenario of the third-party application.

[0220] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to establish data communication between the third-party applications and the operating system. This would allow the operating system to obtain the current scenario information of the third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0221] Taking the Android operating system as an example, the programs and data stored in memory 120 are as follows: Figure 9As shown, the memory 120 can store the Linux kernel layer 320, the system runtime library layer 340, the application framework layer 360, and the application layer 380. The Linux kernel layer 320, system runtime library layer 340, and application framework layer 360 belong to the operating system space, while the application layer 380 belongs to the user space. The Linux kernel layer 320 provides low-level drivers for various hardware components of the electronic device, such as display drivers, audio drivers, camera drivers, Bluetooth drivers, Wi-Fi drivers, and power management. The system runtime library layer 340 provides support for key features of the Android system through several C / C++ libraries. For example, the SQLite library provides database support, the OpenGL / ES library provides 3D graphics support, and the Webkit library provides browser kernel support. The system runtime library layer 340 also provides the Android runtime library, which mainly provides core libraries that allow developers to write Android applications using the Java language. The Application Framework Layer 360 provides various APIs that may be used when building applications. Developers can also use these APIs to build their own applications, such as activity management, window management, view management, notification management, content provider, package management, call management, resource management, and location management. At least one application runs in the Application Layer 380. These applications can be native applications that come with the operating system, such as contacts, SMS, clock, and camera apps; or third-party applications developed by third-party developers, such as games, instant messaging, and photo editing apps.

[0222] Taking the operating system as an example (iOS), the programs and data stored in memory 120 are as follows: Figure 10As shown, the iOS system includes: Core OS layer 420, Core Services layer 440, Media layer 460, and Cocoa Touch layer 480. Core OS layer 420 includes the operating system kernel, drivers, and low-level program frameworks. These low-level program frameworks provide hardware-level functionality for use by the program frameworks located in Core Services layer 440. Core Services layer 440 provides system services and / or program frameworks required by applications, such as Foundation framework, account framework, advertising framework, data storage framework, network connectivity framework, geolocation framework, motion framework, etc. Media layer 460 provides applications with audiovisual interfaces, such as interfaces related to graphics and images, audio technology, video technology, and AirPlay (wireless playback of audio and video transmission technologies). Cocoa Touch layer 480 provides various commonly used interface-related frameworks for application development and is responsible for user touch interaction on electronic devices. Examples include local notification services, remote push services, advertising frameworks, game tool frameworks, message user interface (UI) frameworks, UIKit user interface frameworks, map frameworks, and so on.

[0223] exist Figure 10 The framework shown includes, but is not limited to, the base framework in the core service layer 440 and the UIKit framework in the touchable layer 480. The base framework provides many basic object classes and data types, offering the most basic system services to all applications, and is independent of the UI. The UIKit framework, on the other hand, provides a basic UI class library for creating touch-based user interfaces. iOS applications can use the UIKit framework to provide their UI, thus providing the application's infrastructure for building user interfaces, drawing, handling user interaction events, responding to gestures, and so on.

[0224] The methods and principles for implementing data communication between third-party applications and the operating system in the iOS system can be found in the Android system, and will not be repeated here.

[0225] The input device 130 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 140 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined into a touch screen, which is used to receive touch operations from the user using a finger, stylus, or any suitable object on or near it, and to display the user interface of various applications. The touch screen is usually located on the front panel of the electronic device. The touch screen can be designed as a full-screen, curved screen, or irregularly shaped screen. The touch screen can also be designed as a combination of a full-screen and a curved screen, or a combination of an irregularly shaped screen and a curved screen; this specification does not limit this aspect.

[0226] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device may also include radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (WiFi) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0227] In the embodiments of this specification, the executing entity for each step can be the electronic device described above. Optionally, the executing entity for each step can be the operating system of the electronic device. The operating system can be Android, iOS, or other operating systems; this specification does not limit this.

[0228] The electronic device described in this specification can also be equipped with a display device. This display device can be any device capable of displaying information, such as a cathode ray tube display (CR), a light-emitting diode display (LED), an e-ink screen, a liquid crystal display (LCD), or a plasma display panel (PDP). Users can use the display device on electronic device 101 to view displayed text, images, videos, and other information. The electronic device can be a smartphone, tablet computer, gaming device, AR (Augmented Reality) device, automobile, data storage device, audio playback device, video playback device, laptop, desktop computing device, or wearable device such as an electronic watch, electronic glasses, electronic helmet, electronic bracelet, electronic necklace, or electronic clothing.

[0229] exist Figure 7 In the illustrated electronic device, the processor 110 can be used to call the application program stored in the memory 120 and specifically perform the following operations:

[0230] In a teaching meeting scenario, the hand-drawing gestures of the main speaker are detected, and the hand-drawing gesture data corresponding to the hand-drawing gestures are collected, along with reference teaching scenario information.

[0231] The target hand-drawn content is obtained by recognizing the hand-drawn action data and reference teaching scenario information.

[0232] The target hand-drawn content is synchronized to the participating user's terminal so that the participating user's terminal displays the target hand-drawn content.

[0233] In one embodiment, the processor 110, when performing the acquisition of reference teaching scenario information, includes:

[0234] Extract the hand-drawn action change features from the hand-drawn action data, and determine the target hand-drawn recognition difficulty for the hand-drawn recognition model based on the hand-drawn action change features;

[0235] Based on the difficulty of the target hand-drawn recognition, a combination of target auxiliary recognition data is determined, and reference teaching scenario information corresponding to the target auxiliary recognition data combination is obtained.

[0236] In one embodiment, the processor 110 performs the following operations when executing the determination of the auxiliary recognition data combination based on the target hand-drawn recognition difficulty:

[0237] The auxiliary recognition data combination corresponding to the target hand-drawn recognition difficulty is queried in the preset difficulty combination mapping relationship. The preset difficulty combination mapping relationship includes multiple auxiliary recognition data combinations corresponding to hand-drawn recognition difficulty.

[0238] In one embodiment, the processor 110 performs the following operations when performing hand-drawn content recognition based on the hand-drawn motion data and reference teaching scenario information to obtain the target hand-drawn content:

[0239] Determine hand-drawing recognition task prompts based on the hand-drawing motion data and the reference teaching scenario information;

[0240] The hand-drawn action data, reference teaching scenario information, and hand-drawn recognition task prompts are input into the hand-drawn recognition model, and the target hand-drawn content is obtained by recognizing the hand-drawn content through the hand-drawn recognition model.

[0241] In one embodiment, when the processor 110 performs hand-drawn content recognition using the hand-drawn content recognition model to obtain the target hand-drawn content, it performs the following operations:

[0242] The hand-drawn recognition model is used to perform action recognition on the hand-drawn action data to obtain initial hand-drawn content. The hand-drawn content description information of the initial hand-drawn content is determined. Based on the hand-drawn content description information, the hand-drawn association information of the reference teaching scenario information is mined to obtain teaching demonstration association information.

[0243] Based on the teaching demonstration association information, the initial hand-drawn content is adjusted to obtain the target hand-drawn content.

[0244] In one embodiment, the processor 110 performs the following operations when executing the process of adjusting the initial hand-drawn content based on the teaching demonstration association information to obtain the target hand-drawn content:

[0245] Multiple teaching demonstration related content are identified from the teaching demonstration association information;

[0246] From the multiple teaching demonstration related content, determine the target teaching demonstration related content that matches the initial hand-drawn content;

[0247] Based on the target teaching demonstration related content, the initial hand-drawn content is modified to obtain the target hand-drawn content.

[0248] In one embodiment, the processor 110 performs the following operations when executing the process of adjusting the initial hand-drawn content based on the teaching demonstration association information to obtain the target hand-drawn content:

[0249] The content structure information and standard knowledge point content in the target teaching demonstration related content are determined, and the initial hand-drawn content is structurally optimized based on the content structure information to form the first hand-drawn content;

[0250] Based on the standard content of the aforementioned knowledge points, the first hand-drawn content is optimized in terms of content details to obtain the second hand-drawn content;

[0251] The target hand-drawn content is obtained by performing visual enhancement processing on the second hand-drawn content.

[0252] In one or more embodiments of this specification, in a teaching conference scenario, the electronic device detects the hand-drawn gestures of the presenter, collects the corresponding hand-drawn gesture data and obtains reference teaching scenario information, identifies the target hand-drawn content based on the hand-drawn gesture data and reference teaching scenario information, and synchronizes the target hand-drawn content to the participating users' terminals so that the participating users' terminals can display the target hand-drawn content. This effectively solves the problem of intelligent collection, accurate identification, and synchronous display of teachers' hand-drawn content in remote teaching. This technical solution can not only capture and identify teachers' hand-drawn content in real time, but also synchronously display the hand-drawn content to all participants, making the knowledge transfer in remote teaching clearer and more accurate, and helping to improve teaching effectiveness and students' learning experience.

[0253] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0254] The above-disclosed embodiments are merely preferred embodiments of this specification and should not be construed as limiting the scope of this specification. Therefore, any equivalent variations made in accordance with the claims of this specification shall still fall within the scope of this specification.

Claims

1. A method for processing teaching data, characterized in that, The method includes: In a teaching meeting scenario, the hand-drawing gestures of the main speaker are detected, and the hand-drawing gesture data corresponding to the hand-drawing gestures are collected, along with reference teaching scenario information. The target hand-drawn content is obtained by recognizing the hand-drawn action data and reference teaching scenario information. The target hand-drawn content is synchronized to the participating user's terminal so that the participating user's terminal displays the target hand-drawn content.

2. The method according to claim 1, characterized in that, The acquisition of reference teaching scenario information includes: Extract the hand-drawn action change features from the hand-drawn action data, and determine the target hand-drawn recognition difficulty for the hand-drawn recognition model based on the hand-drawn action change features; Based on the difficulty of the target hand-drawn recognition, a combination of target auxiliary recognition data is determined, and reference teaching scenario information corresponding to the target auxiliary recognition data combination is obtained.

3. The method according to claim 2, characterized in that, The determination of auxiliary recognition data combinations based on the target hand-drawn recognition difficulty includes: The auxiliary recognition data combination corresponding to the target hand-drawn recognition difficulty is queried in the preset difficulty combination mapping relationship. The preset difficulty combination mapping relationship includes multiple auxiliary recognition data combinations corresponding to hand-drawn recognition difficulty.

4. The method according to claim 1, characterized in that, The process of identifying the target hand-drawn content based on the hand-drawn action data and reference teaching scenario information includes: Determine hand-drawing recognition task prompts based on the hand-drawing motion data and the reference teaching scenario information; The hand-drawn action data, reference teaching scenario information, and hand-drawn recognition task prompts are input into the hand-drawn recognition model, and the target hand-drawn content is obtained by recognizing the hand-drawn content through the hand-drawn recognition model.

5. The method according to claim 4, characterized in that, The process of obtaining the target hand-drawn content through the hand-drawn content recognition model includes: The hand-drawn recognition model is used to perform action recognition on the hand-drawn action data to obtain initial hand-drawn content. The hand-drawn content description information of the initial hand-drawn content is determined. Based on the hand-drawn content description information, the hand-drawn association information of the reference teaching scenario information is mined to obtain teaching demonstration association information. Based on the teaching demonstration association information, the initial hand-drawn content is adjusted to obtain the target hand-drawn content.

6. The method according to claim 5, characterized in that, The process of adjusting the initial hand-drawn content based on the teaching demonstration association information to obtain the target hand-drawn content includes: Multiple teaching demonstration related content are identified from the teaching demonstration association information; From the multiple teaching demonstration related content, determine the target teaching demonstration related content that matches the initial hand-drawn content; Based on the target teaching demonstration related content, the initial hand-drawn content is modified to obtain the target hand-drawn content.

7. The method according to claim 6, characterized in that, The process of modifying the initial hand-drawn content based on the target teaching demonstration related content to obtain the target hand-drawn content includes: The content structure information and standard knowledge point content in the target teaching demonstration related content are determined, and the initial hand-drawn content is structurally optimized based on the content structure information to form the first hand-drawn content; Based on the standard content of the aforementioned knowledge points, the first hand-drawn content is optimized in terms of content details to obtain the second hand-drawn content; The target hand-drawn content is obtained by performing visual enhancement processing on the second hand-drawn content.

8. A teaching data processing device, characterized in that, The device includes: The data acquisition module is used to detect the hand-drawing actions of the main speaker in a teaching meeting scenario, collect the hand-drawing action data corresponding to the hand-drawing actions, and obtain reference teaching scenario information. The content recognition module is used to identify the target hand-drawn content based on the hand-drawn action data and reference teaching scenario information. The content synchronization module is used to synchronize the target hand-drawn content with the participating user terminals so that the participating user terminals can display the target hand-drawn content.

9. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions, which are adapted to be loaded by a processor and executed as method steps as claimed in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed the method steps as claimed in any one of claims 1 to 7.

Images