A PPT courseware generation and teaching method based on human-computer collaboration
By building a common resource library for teachers-students-parents, and using deep learning models to extract multi-modal resource features to generate PPT courseware, the problem of insufficient integration of traditional teaching resources is solved, intelligent resource matching and personalized teaching is achieved, and the The collaborative teaching effect of teachers, students and machines.
Patent Information
- Application Number
- CN202411269034.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-11
AI Technical Summary
The integration of traditional classroom teaching resources lacks automation and intelligence, which makes it difficult to match the courseware resources of "one-sided people" with personalized learning needs, and the transformation of preset resources and generative resources is inconvenient, and the three-dimensional structure of "teacher, student, and machine" Inadequate integration efforts.
By building a common resource library for teachers-students-parents, using deep learning models to extract multi-modal resource features, generate PPT courseware that meets the theme text, and integrate resources in real time in classroom teaching to achieve collaborative teaching of "teacher, student, and machine".
It realizes the intelligent organization, management and sharing of classroom teaching resources, improves the personalized matching and integration efficiency of teaching resources, and enhances the teaching coordination of "teachers, students, and machines".
Smart Images

Figure CN118799141B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer intelligent education technology, and in particular to a PPT courseware generation and teaching method based on human-computer collaboration. Background Art
[0002] Traditional classroom teaching resource integration is mainly completed by teachers, lacking automated and intelligent processing. Due to the shallow degree of integration of information technology and curriculum, there are two prominent contradictions in the classroom teaching process that have not been well resolved: the first is the contradiction between the "one-size-fits-all" courseware resources and personalized learning needs; the second is the contradiction between the mutual transformation of preset resources and generative resources. Generative resources come from the interaction between preset resources and student learning, which can be preset through artificial intelligence technology. Preset resources and generative resources are integrated in real time into classroom teaching resources, which requires full use of artificial intelligence technology and curriculum integration.
[0003] Related technical solution 1: Chinese patent with publication number CN117350264A discloses a PPT file generation method, device, equipment and storage medium, which uses a large model (Llama2-13B + P-Tuning v2 fine-tuning training) and AutoPPT algorithm to realize a system for automatically generating PPTs, greatly reducing the time and cost of making PPTs;
[0004] Related technical solution two, a Chinese patent with publication number CN111985199A discloses a PPT generation method and system based on mind mapping; according to the input mind map and the preset PPT master, a corresponding relationship is established based on the master filling rules; after the mind map is converted into JSON data, the multi-level content structure of the mind map is retained through the hierarchical nesting in the JSON data, and the mind map text content is filled into the PPT master according to the corresponding relationship, and then added to the PPT presentation, so as to realize the rapid and automatic generation of PPT reports based on the mind map.
[0005] However, in the above technical solutions, there is little consideration of the degree of structuring between multimodal resources such as images, texts, sounds, and videos, such as the accuracy of image and text matching, and whether they are designed according to the students' psychological work model. There is insufficient integration of the "teachers, students, and machines" tripartite structure of home-school co-education. Summary of the invention
[0006] The main purpose of the present invention is to provide a PPT courseware generation and teaching method based on human-computer collaboration, aiming to solve the problem of how to generate PPT courseware and teach based on the ternary structure of "teacher, student, and machine".
[0007] To achieve the above object, the present invention provides a PPT courseware generation and teaching method based on human-computer collaboration, the method comprising:
[0008] S1, Resource Library Preparation
[0009] The teacher terminal sends the homework task constructed according to the course knowledge graph to the student terminal, and the student terminal sends the student homework data formed after completing the homework task to the parent terminal;
[0010] The server collects the student homework data uploaded by the parent, collects the structured teaching resource data uploaded by the teacher after classification and editing based on the student homework data, and integrates the student homework data and the structured teaching resource data into a teacher-student-parent common resource library;
[0011] S2, deep learning model training
[0012] The server extracts features of each structured teaching resource data in the teacher-student-parent common resource library;
[0013] The server side fuses the extracted feature vectors and inputs the fused feature vectors into the corresponding neural network prediction to determine the relevance between the subject text and the multimodal resource;
[0014] S3, PPT courseware resource generation and improvement
[0015] When receiving the subject text input by the teacher, the server searches and generates a PPT courseware that meets the preset subject text in the teacher-student-parent common resource library, and sends the PPT courseware to the teacher;
[0016] The teacher terminal performs secondary editing on the generated PPT courseware;
[0017] S4, classroom teaching activities
[0018] The teacher terminal pushes the PPT courseware to the student terminal online;
[0019] The server side obtains the record data generated by the student side's annotations on the PPT courseware, and pushes the teaching resources generated according to the record data to the teacher side based on the model relevance algorithm;
[0020] The server side associates and stores the record data generated by each student side with the PPT courseware.
[0021] Optionally, the step of extracting features from each piece of structured teaching resource data in the teacher-student-parent common resource library specifically includes:
[0022] The Bert model and ERNIE model based on the pre-trained model are used to extract features from the subject text in the structured teaching resource data;
[0023] The trained image entity classification feature layer is used to cut frames of the video files in the structured teaching resource data;
[0024] Separate audio files from video files of structured teaching resource data, segment audio files and extract multi-dimensional audio features.
[0025] The extracted image features and audio features are sequentially learned using a long short-term memory network.
[0026] Optionally, the image entity classification feature layer is a hidden layer vector of 2048 dimensions before classification, and the image entity classification feature layer is input into RestNet for training.
[0027] Optionally, the steps of separating the audio file from the video file of the structured teaching resource data, segmenting the audio file and extracting multi-dimensional audio features specifically include:
[0028] The 128-dimensional features of the audio file are extracted every 960ms as a segment, where the window size within 960ms is 25ms and the step size is 10ms;
[0029] 96*64 audio features are extracted through the log-mel spectrum graph, which is input into the VGGish deep CNN network as a 96-by-64-dimensional image to extract a 128-dimensional audio feature vector.
[0030] Optionally, in S2, the multimodal resources include video resources, audio resources, and picture resources, and the step of inputting the fused feature vector into the corresponding neural network prediction specifically includes:
[0031] Extracting image features and audio characteristics of video resources in the structured teaching resource data for sequence learning, fusing them with text features and inputting them into a neural network model, and calculating the correlation between the video resources and the text features;
[0032] Calculating the correlation between the image resources and the audio resources in the structured teaching resource data based on a speech-text matching algorithm;
[0033] Based on the image-text matching algorithm, the relevance between the image resources in the structured teaching resource data and the subject text is calculated.
[0034] Optionally, in step S3, generating a PPT courseware that satisfies the preset theme text satisfies the following rules:
[0035] (1) When it is calculated that the correlation between the subject text and each modal resource is greater than a preset correlation threshold, a modal resource with a middle correlation among the modal resources is selected as the target modal resource, and the target modal resource and the text are placed in the same frame to form a PPT page;
[0036] (2) When it is calculated that the correlation between the subject text and each modal resource is less than or equal to a preset correlation threshold, a modal resource with the highest correlation among the modal resources is selected as the target modal resource, and the target modal resource and the text are in the same frame to form a page of PPT.
[0037] Optionally, in S1, the parent end uploads the student homework data in the format of resource type, resource title text, resource description text and resource attachment.
[0038] Optionally, in step S3, the teacher end performs secondary editing on the generated PPT courseware, specifically including:
[0039] Perform local spatial layout and global sequence adjustment of resources on the generated PPT courseware according to the principles of spatial proximity and temporal proximity; and / or,
[0040] Continue entering the topic text to find other resources as PPT courseware.
[0041] In addition, to achieve the above-mentioned purpose, the present invention also provides a PPT classroom teaching resource management system, which includes: a memory, a processor, and a PPT courseware generation and teaching program based on human-computer collaboration stored in the memory and runnable on the processor. When the PPT courseware generation and teaching program based on human-computer collaboration is executed by the processor, the steps of the PPT courseware generation and teaching method based on human-computer collaboration as described in any of the above items are implemented.
[0042] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which is stored a PPT courseware generation and teaching program based on human-computer collaboration. When the PPT courseware generation and teaching program based on human-computer collaboration is executed by a processor, the steps of the PPT courseware generation and teaching method based on human-computer collaboration as described in any of the above items are implemented.
[0043] The present invention has at least the following beneficial effects:
[0044] By integrating the resources of human-computer, home and school, focusing on the PPT courseware generation and teaching process, based on multimodal models and application management procedures, the intelligent organization, management, sharing and teaching of classroom teaching resources are realized. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1The present invention is a schematic diagram of the hardware operating environment of the PPT classroom teaching resource management system involved in the embodiment of the present invention.
[0046] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0047] In order to better understand the above technical solution, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0048] As an implementation plan, Figure 1 The present invention is a schematic diagram of the hardware operating environment of the PPT classroom teaching resource management system involved in the embodiment of the present invention.
[0049] like Figure 1 As shown, the PPT classroom teaching resource management system may include: a processor 1001, such as a CPU, a memory 1005, a user interface 1003, a network interface 1004, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0050] Those skilled in the art will understand that Figure 1 The PPT classroom teaching resource management system architecture shown in the figure does not constitute a limitation of the PPT classroom teaching resource management system, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.
[0051] like Figure 1As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a PPT courseware generation and teaching program based on human-computer collaboration. Among them, the operating system is a program for managing and controlling the hardware and software resources of the PPT classroom teaching resource management system, the PPT courseware generation and teaching program based on human-computer collaboration, and the operation of other software or programs.
[0052] exist Figure 1 In the PPT classroom teaching resource management system shown, the user interface 1003 is mainly used to connect to the terminal and communicate data with the terminal; the network interface 1004 is mainly used for the background server and communicates data with the background server; the processor 1001 can be used to call the PPT courseware generation and teaching program based on human-computer collaboration stored in the memory 1005.
[0053] In this embodiment, the PPT classroom teaching resource management system includes: a memory 1005, a processor 1001, and a PPT courseware generation and teaching program based on human-computer collaboration stored in the memory and executable on the processor, wherein:
[0054] When the processor 1001 calls the PPT courseware generation and teaching program based on human-computer collaboration stored in the memory 1005, the following operations are performed:
[0055] The teacher terminal sends the homework task constructed according to the course knowledge graph to the student terminal, and the student terminal sends the student homework data formed after completing the homework task to the parent terminal;
[0056] The server collects the student homework data uploaded by the parent, collects the structured teaching resource data uploaded by the teacher after classification and editing based on the student homework data, and integrates the student homework data and the structured teaching resource data into a teacher-student-parent common resource library;
[0057] The server extracts features of each structured teaching resource data in the teacher-student-parent common resource library;
[0058] The server side fuses the extracted feature vectors and inputs the fused feature vectors into the corresponding neural network prediction to determine the relevance between the subject text and the multimodal resource;
[0059] When receiving the subject text input by the teacher, the server searches and generates a PPT courseware that meets the preset subject text in the teacher-student-parent common resource library, and sends the PPT courseware to the teacher;
[0060] The teacher terminal performs secondary editing on the generated PPT courseware;
[0061] The teacher terminal pushes the PPT courseware to the student terminal online;
[0062] The server side obtains the record data generated by the student side's annotations on the PPT courseware, and pushes the teaching resources generated according to the record data to the teacher side based on the model relevance algorithm;
[0063] The server side associates and stores the record data generated by each student side with the PPT courseware.
[0064] When the processor 1001 calls the PPT courseware generation and teaching program based on human-computer collaboration stored in the memory 1005, the following operations are performed:
[0065] The Bert model and ERNIE model based on the pre-trained model are used to extract features from the subject text in the structured teaching resource data;
[0066] The trained image entity classification feature layer is used to cut frames of the video files in the structured teaching resource data;
[0067] Separate audio files from video files of structured teaching resource data, segment audio files and extract multi-dimensional audio features.
[0068] The extracted image features and audio features are sequentially learned using a long short-term memory network.
[0069] When the processor 1001 calls the PPT courseware generation and teaching program based on human-computer collaboration stored in the memory 1005, the following operations are performed:
[0070] The 128-dimensional features of the audio file are extracted every 960ms as a segment, where the window size within 960ms is 25ms and the step size is 10ms;
[0071] 96*64 audio features are extracted through the log-mel spectrum graph, which is input into the VGGish deep CNN network as a 96-by-64-dimensional image to extract a 128-dimensional audio feature vector.
[0072] When the processor 1001 calls the PPT courseware generation and teaching program based on human-computer collaboration stored in the memory 1005, the following operations are performed:
[0073] Extracting image features and audio characteristics of video resources in the structured teaching resource data for sequence learning, fusing them with text features and inputting them into a neural network model, and calculating the correlation between the video resources and the text features;
[0074] Calculating the correlation between the image resources and the audio resources in the structured teaching resource data based on a speech-text matching algorithm;
[0075] Based on the image-text matching algorithm, the relevance between the image resources in the structured teaching resource data and the subject text is calculated.
[0076] When the processor 1001 calls the PPT courseware generation and teaching program based on human-computer collaboration stored in the memory 1005, the following operations are performed:
[0077] In the S3 stage, the PPT courseware that meets the preset theme text is generated to meet the following rules:
[0078] (1) When it is calculated that the correlation between the subject text and each modal resource is greater than a preset correlation threshold, a modal resource with a middle correlation among the modal resources is selected as the target modal resource, and the target modal resource and the text are placed in the same frame to form a PPT page;
[0079] (2) When it is calculated that the correlation between the subject text and each modal resource is less than or equal to a preset correlation threshold, a modal resource with the highest correlation among the modal resources is selected as the target modal resource, and the target modal resource and the text are in the same frame to form a page of PPT.
[0080] Based on the hardware architecture of the PPT classroom teaching resource management system of the above-mentioned computer intelligent education technology, an embodiment of the PPT courseware generation and teaching method based on human-computer collaboration of the present invention is proposed.
[0081] First embodiment
[0082] A PPT courseware generation and teaching method based on human-computer collaboration includes the following steps:
[0083] S1, Resource Library Preparation
[0084] S11, the teacher terminal sends the homework task constructed according to the course knowledge graph to the student terminal, and the student terminal sends the student homework data formed after completing the homework task to the parent terminal;
[0085] In this embodiment, the teacher assigns homework tasks to students using the teacher terminal according to the course knowledge graph of the textbook, and then sends the homework tasks to the student terminal for the students to complete the homework tasks on the student terminal;
[0086] After students complete their homework, the student homework data generated after completing the homework task is sent to the parent side rather than the teacher side through the student side. One of the purposes of this setting is to introduce the role of parents to participate in the child’s learning process, thereby constructing a three-element teaching structure of "teacher, student, and machine".
[0087] It should be noted that the work of teachers, parents and students to jointly build resources is an accumulative process. Every assignment and resource collection makes the resource library gradually larger. Teachers will also search for a large number of resources from various resource libraries on the Internet and upload them to the resource library around specific topics. In addition, each topic will appear in the following stages of study and present a "spiral" upward advancement process, with the same course in different classes horizontally and a "spiral" knowledge system vertically. The amount of resources collected is sufficient to support the subsequent model training process and ensure the accuracy of resource prediction.
[0088] S12, the server collects the student homework data uploaded by the parent, collects the structured teaching resource data uploaded by the teacher obtained by classifying and editing the student homework data, and integrates the student homework data and the structured teaching resource data into a teacher-student-parent common resource library;
[0089] In this embodiment, the server side of the platform collects the student homework data uploaded by the parent side, wherein the student homework data uploaded by the parent side should contain certain operation records for the student homework data, thereby proving that the parents are involved in the child's learning process.
[0090] Similarly, the server side of the platform collects the structured teaching resource data uploaded by the teacher side after classification and editing based on the student homework data. The structured teaching resource data is the data generated after the teacher side corrects, classifies and stores the student homework in a structured manner.
[0091] Optionally, in order to facilitate the formation of structured teaching resource data, parents take photos to record students' homework, life practice, labor content and other resources, and upload them to the designated address system for structured storage on a regular basis as required. The parent side uploads the student homework data in the format of [resource type, resource title text, resource description text, resource attachment].
[0092] Similarly, the server side of the platform integrates the student homework data and the structured teaching resource data into a teacher-student-parent common resource library, and the teacher-student-parent common resource library is used for training the deep learning model.
[0093] S2, deep learning model training
[0094] S21, extracting features from each piece of structured teaching resource data in the teacher-student-parent common resource library;
[0095] S22, fusing the extracted feature vectors, and inputting the fused feature vectors into the corresponding neural network prediction to determine the relevance between the topic text and the multimodal resource;
[0096] In this embodiment, the server side extracts features from each structured teaching resource data in the teacher-student-parent common resource library. The extracted feature vectors are then fused, and the fused feature vectors are input into the corresponding neural network prediction to determine the correlation between the subject text and the multimodal resource. The correlation is used to sort the pre-set PPT courseware resource correlation based on the size of the correlation when searching and generating PPT courseware in the server background, and give priority to PPT courseware resources with greater correlation.
[0097] Optionally, the specific feature extraction steps can be: based on the Bert model and ERNIE model of the pre-trained model, feature extraction is performed on the subject text in the structured teaching resource data; then the trained image entity classification feature layer is used to cut frames of the video files in the structured teaching resource data; then the audio files are separated from the video files of the structured teaching resource data, the audio files are segmented and multi-dimensional audio features are extracted, and finally the long short-term memory network is used to perform sequence learning on the extracted image features and audio features.
[0098] Further and optionally, the image entity classification feature layer can be a hidden layer vector of 2048 dimensions before classification. The purpose of this step is to generate a high-dimensional classification feature layer so that the model can better capture the details in the image. Then, the image entity classification feature layer is input into RestNet for training.
[0099] It should be noted that the 2048-dimensional hidden layer vector before classification is selected as the empirical value obtained during training. After testing, the 2048-dimensional vector can achieve a good balance between feature expression ability and computational complexity, capturing more image details and feature information while taking into account the computing power overhead of the server.
[0100] It should be noted that ResNet (residual network) is chosen to use its skip connection structure to solve the gradient vanishing problem of deep networks, so as to better extract high-level image features. ResNet can effectively prevent the overfitting problem in deep network training through residual connections. In addition, ResNet performs well on large datasets such as ImageNet, with high accuracy and robustness.
[0101] Further and optionally, 128-dimensional features of the audio file are extracted every 960ms as a segment, wherein the window size within 960ms is 25ms and the step size is 10ms; 96*64 audio features are extracted through the log-mel spectrogram, and are input into the VGGish deep CNN network as a 96-by-64-length image to extract a 128-dimensional audio feature vector.
[0102] It should be noted that the selection of 960ms as the segment length can balance short-term features and long-term dependencies, provide sufficient contextual information, and avoid the problem of information redundancy and increased computational complexity caused by too long segments; the selection of a window size of 25ms and a step size of 10ms is to achieve a balance between frequency and time resolution. The 25ms window size provides sufficient frequency resolution, while the 10ms step size ensures time resolution and can capture rapidly changing audio signals; the log-mel spectrogram can effectively represent the spectral characteristics of audio signals, and the selection of a 96*64 size makes the feature map have a suitable resolution, which can capture sufficient spectral details while maintaining computational feasibility. In addition, the VGGish network is an optimized convolutional neural network specifically used for audio feature extraction. The use of the VGGish network can extract high-level 128-dimensional feature vectors with strong audio recognition and classification capabilities.
[0103] S3, PPT courseware resource generation and improvement
[0104] S31, when receiving the subject text input by the teacher end, the server end searches and generates a PPT courseware satisfying the preset subject text in the teacher-student-parent common resource library, and sends the PPT courseware to the teacher end;
[0105] S32, the teacher performs secondary editing on the generated PPT courseware;
[0106] In this embodiment, the server side searches and generates a PPT courseware that meets the preset theme text in the teacher-student-parent common resource library, and sends the PPT courseware to the teacher side.
[0107] Optionally, the retrieved subject text is matched with the resource library content through image, video, and audio training models respectively, and relevant resources are output based on relevance matching, and the data output in JSON format is transmitted to the PPT generation interface.
[0108] The teacher can make secondary edits to the generated PPT courseware.
[0109] S4, classroom teaching activities
[0110] S41, the teacher terminal pushes the PPT courseware to the student terminal online;
[0111] S42, the server side obtains the record data generated by the student side's annotations on the PPT courseware, and pushes the teaching resources generated according to the record data to the teacher side based on the model relevance algorithm;
[0112] S43, the server side associates and stores the record data generated by each student side with the PPT courseware.
[0113] In this embodiment, when the student makes annotations on the PPT courseware, the system records the annotation content, timestamp, PPT page and related location information in real time. These annotation data are uploaded to the server. After receiving the student's annotation data, the server performs preliminary processing and cleaning on the data, and stores the cleaned data in the database. The server calls the pre-trained relevance algorithm model to analyze the annotation data and extract important keywords and topics. According to the extracted keywords and topics, the relevance of the annotation content to the existing teaching resources is calculated, and highly relevant teaching resources are screened out. The generated teaching resource recommendation list is pushed to the teacher's side, and the teacher can view and use these resources in real time on his device, allowing the teacher to view the annotation details and recommended resources, and make further annotations and feedback.
[0114] Furthermore, how to push the teaching resources generated according to the recorded data to the teacher side based on the model relevance algorithm. The server side calls the built-in natural language processing algorithm to perform word segmentation and part-of-speech tagging on the annotation content, uses TF-IDF (term frequency-inverse document frequency) to extract key terms in the annotation content, and combines the contextual information of the annotation content to identify the main topics and core concepts;
[0115] Next, the extracted keywords and topics are input into a pre-trained relevance calculation model (BERT model), and the vector representation is used to calculate the similarity between the annotation content and the teaching resources in the database. Based on the similarity score, the teaching resources that are highly relevant to the annotation content are screened out.
[0116] The selected teaching resources are sorted by similarity score to generate a personalized teaching resource recommendation list. The recommendation list indicates the relevance score of each resource to the annotation content.
[0117] The generated personalized teaching resource recommendation list is sent to the teacher-side application through a push service (such as WebSocket or RESTful API), and the recommendation list is displayed in the teacher-side application.
[0118] Second embodiment
[0119] As an optional embodiment, in this embodiment, in the S2 stage, the multimodal resources include video resources, audio resources and picture resources, and the S22 specifically includes:
[0120] S221, extracting image features and audio characteristics of video resources in the structured teaching resource data for sequence learning, fusing them with text features and inputting them into a neural network model, and calculating the correlation between the video resources and the text features;
[0121] S222, calculating the correlation between the image resources and the audio resources in the structured teaching resource data based on a speech-text matching algorithm;
[0122] S223, based on a picture-text matching algorithm, calculating the relevance between the picture resources in the structured teaching resource data and the subject text.
[0123] In this embodiment, optionally, a pre-trained ResNet or Inception convolutional neural network is used to extract image features from each frame or key frame of the video.
[0124] The video audio part is divided into short time segments (such as 960ms), and then converted into a 96*64 two-dimensional feature map using the log-mel spectrogram, and the audio features are extracted using the pre-trained VGGish audio feature extraction model. After preprocessing the text data, word segmentation and stop word removal, the text is converted into a vector representation using a word embedding model.
[0125] How to perform sequence learning: Use the recursive neural network LSTM to perform sequence learning on the extracted image features and audio features to capture the temporal relationship between the features. Perform weighted averaging on the learned image and audio features and fuse them to form a comprehensive feature vector. Fuse the comprehensive feature vector with the text features. Common methods include feature concatenation, weighted averaging, or weighted fusion through the attention mechanism. Effect: Through sequence learning and feature fusion, the image, audio characteristics, and text features of the video are integrated to form a unified feature vector, which is convenient for subsequent correlation calculations.
[0126] Optionally, as to how to calculate the correlation between the picture resources and the audio resources in the structured teaching resource data, a pre-trained ResNet convolutional neural network is used to extract image features from the picture; then the audio features are extracted through the log-mel spectrogram and the VGGish model, and finally the fused features are input into a neural network model with a multi-layer perceptron to calculate the correlation score between the video resources and the text features.
[0127] Optionally, as to how to calculate the relevance between the image resources in the structured teaching resource data and the subject text, a pre-trained convolutional neural network (such as ResNet or Inception) is used to extract image features from the image; the subject text is then pre-processed and word-embedded to convert it into a vector representation; and then a picture-text matching model based on an attention mechanism is used to calculate the relevance score between the image features and the subject text features.
[0128] Through the above steps, the system can efficiently and accurately extract the image, audio and text features in the structured teaching resource data, and comprehensively process these features through sequence learning and feature fusion technology to calculate the relevance between video resources and text features, picture resources and audio resources, and picture resources and topic text. The refinement of each step ensures the accuracy of data processing and the effectiveness of relevance calculation, and improves the accuracy of teaching resource matching and recommendation.
[0129] Third embodiment
[0130] As an optional embodiment, in this embodiment, in the S3 stage, generating a PPT courseware that meets the preset theme text satisfies the following rules:
[0131] (1) When it is calculated that the correlation between the subject text and each modal resource is greater than a preset correlation threshold, a modal resource with a middle correlation among the modal resources is selected as the target modal resource, and the target modal resource and the text are placed in the same frame to form a PPT page;
[0132] (2) When it is calculated that the correlation between the subject text and each modal resource is less than or equal to a preset correlation threshold, a modal resource with the highest correlation among the modal resources is selected as the target modal resource, and the target modal resource and the text are in the same frame to form a page of PPT.
[0133] Specifically, the PPT generation process is an optimal selection process based on the relevance of the subject text and the three multimodal resources in the existing resource library.
[0134] According to cognitive load theory, relevance is the embodiment of the consistency principle. At the same time, based on the consistency principle, the three modal resources with the highest relevance to the subject text are not placed on the same PPT courseware at the same time, but the three resources are used to reduce cognitive load as much as possible and avoid redundancy:
[0135] (1) When the three multimodal resources are highly relevant to the topic, a modal resource with a medium relevance is selected to form a PPT page with the text, taking into account the relevance of the resources while ensuring the lowest redundancy;
[0136] (2) When the relevance of resources stored in the structured form of the topic is very low, select the modal resource with the highest relevance according to the multimedia cognitive principle to match the topic text and generate PPT courseware resources. Combining the above two situations and the individual difference principle and cognitive load balancing theory of multimedia learning cognitive theory, the load value of each page of PPT is not the higher the better for students, but needs to be close to the corresponding capacity tolerance range. We multiply the relevance of the multimodal resources (because it is a compromise between necessary correlation and non-transitive correlation). When the product is less than or equal to or less than and closest to the upper limit of cognitive load tolerance, the corresponding resource combination is defined as the relatively optimal resource combination that needs to be generated.
[0137] For example, see the storage table of relevance between resource types and subject text 1 input by the teacher end given in Table 1;
[0138] Table 1. Storage table of relevance between resource type and topic text 1
[0139]
[0140] Furthermore, combined with the relevance prediction values stored in Table 1, the PPT courseware obtained by taking the subject text 1 as the search target and combining it with the four modal resources 1 to 4 will have the following combination types, see Table 2-Table 9:
[0141] Table 2. Correspondence between PPT courseware and learners’ personalized resource needs (0.5)
[0142]
[0143] Table 3. Correspondence between PPT courseware and learners’ personalized resource needs (0.5)
[0144]
[0145] Table 4. Correspondence between PPT courseware and learners’ personalized resource needs (0.5)
[0146]
[0147] Table 5. Correspondence between PPT courseware and learners’ personalized resource needs (0.5)
[0148]
[0149] Table 6. Correspondence between PPT courseware and learners’ personalized resource needs (0.6)
[0150]
[0151] Table 7 Correspondence between PPT courseware and learners’ personalized resource needs (0.6)
[0152]
[0153] Table 8 Correspondence between PPT courseware and learners’ personalized resource needs (0.6)
[0154]
[0155] Table 9 Correspondence between PPT courseware and learners’ personalized resource needs (0.6)
[0156]
[0157] Furthermore, multimodal resources are selected in combination with the course knowledge graph to generate PPT courseware, see the optimal combination of the two components in Table 10, and local spatial layout and global order adjustment of resources are performed on the generated PPT courseware according to the principles of spatial proximity and temporal proximity; at the same time, more content is supplemented by continuing to input relevant text retrieval resources.
[0158] Table 10 Correspondence between PPT courseware and learners’ personalized resource needs (0.5, 0.6)
[0159]
[0160] In addition, it can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the PPT classroom teaching resource management system to implement the process steps of the embodiment of the above method.
[0161] Therefore, the present invention also provides a computer-readable storage medium, which stores a PPT courseware generation and teaching program based on human-computer collaboration. When the PPT courseware generation and teaching program based on human-computer collaboration is executed by a processor, it implements the various steps of the PPT courseware generation and teaching method based on human-computer collaboration described in the above embodiment.
[0162] The computer-readable storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.
[0163] It should be noted that since the storage medium provided in the embodiment of the present application is the storage medium used to implement the method of the embodiment of the present application, based on the method introduced in the embodiment of the present application, the person skilled in the art can understand the specific structure and deformation of the storage medium, so it is not repeated here. All storage media used in the method of the embodiment of the present application belong to the scope of protection of this application.
[0164] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0165] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0166] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0168] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.
[0169] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0170] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A PPT courseware generation and teaching method based on human-computer collaboration, It is characterized in that Applied to a PPT classroom teaching resource management system, the PPT classroom teaching resource management system includes a teacher end, a student end, a parent end and a server end, and the method includes the following steps: S1, resource library preparation phase; The teacher terminal sends the homework task constructed according to the course knowledge graph to the student terminal, and the student terminal sends the student homework data formed after completing the homework task to the parent terminal; The server collects the student homework data uploaded by the parent, collects the structured teaching resource data uploaded by the teacher after classification and editing based on the student homework data, and integrates the student homework data and the structured teaching resource data into a teacher-student-parent common resource library; S2, deep learning model training phase; The server extracts features of each structured teaching resource data in the teacher-student-parent common resource library; The server side fuses the extracted feature vectors and inputs the fused feature vectors into the corresponding neural network prediction to determine the relevance between the subject text and the multimodal resource; S3, PPT courseware resource generation and improvement; When receiving the subject text input by the teacher, the server searches and generates a PPT courseware that meets the preset subject text in the teacher-student-parent common resource library, and sends the PPT courseware to the teacher; The teacher terminal performs secondary editing on the generated PPT courseware; S4, classroom teaching activities; The teacher terminal pushes the PPT courseware to the student terminal online; The server side obtains the record data generated by the student side's annotations on the PPT courseware, and pushes the teaching resources generated according to the record data to the teacher side based on the model relevance algorithm; The server side associates and stores the recorded data generated by each student side with the PPT courseware; The step of extracting features from each piece of structured teaching resource data in the teacher-student-parent common resource library specifically includes: The Bert model and ERNIE model based on the pre-trained model are used to extract features from the subject text in the structured teaching resource data; The trained image entity classification feature layer is used to cut frames of the video files in the structured teaching resource data; Separate audio files from video files of structured teaching resource data, segment the audio files and extract multi-dimensional audio features; Use long short-term memory network to perform sequence learning on the extracted image features and audio features; In S2, the multimodal resources include video resources, audio resources, and picture resources, and the step of inputting the fused feature vector into the corresponding neural network prediction specifically includes: Extracting image features and audio characteristics of video resources in the structured teaching resource data for sequence learning, fusing them with text features and inputting them into a neural network model, and calculating the correlation between the video resources and the text features; Calculating the correlation between the image resources and the audio resources in the structured teaching resource data based on a speech-text matching algorithm; Based on the image-text matching algorithm, calculating the relevance between the image resources in the structured teaching resource data and the subject text; In the S3 stage, the PPT courseware that meets the preset theme text is generated to meet the following rules: (1) When it is calculated that the correlation between the subject text and each modal resource is greater than a preset correlation threshold, a modal resource with a middle correlation among the modal resources is selected as the target modal resource, and the target modal resource and the text are placed in the same frame to form a PPT page; (2) When it is calculated that the correlation between the subject text and each modal resource is less than or equal to a preset correlation threshold, a modal resource with the highest correlation among the modal resources is selected as the target modal resource, and the target modal resource and the text are in the same frame to form a page of PPT.
2. The method according to claim 1, It is characterized in that The image entity classification feature layer is a hidden layer vector of 2048 dimensions before classification, and the image entity classification feature layer is input into RestNet for training.
3. The method according to claim 1, It is characterized in that The steps of separating the audio file from the video file of the structured teaching resource data, segmenting the audio file and extracting multi-dimensional audio features specifically include: The 128-dimensional features of the audio file are extracted every 960ms as a segment, where the window size within 960ms is 25ms and the step size is 10ms; 96*64 audio features are extracted through the log-mel spectrum graph, which is input into the VGGish deep CNN network as a 96-by-64-dimensional image to extract a 128-dimensional audio feature vector.
4. The method according to claim 1, It is characterized in that In S1, the parent end uploads the student homework data in the format of resource type, resource title text, resource description text and resource attachment.
5. The method according to claim 1, It is characterized in that In step S3, the teacher's secondary editing of the generated PPT courseware specifically includes: Perform local spatial layout and global order adjustment of resources on the generated PPT courseware according to the principles of spatial proximity and temporal proximity; and / or, Continue entering the topic text to find other resources as PPT courseware.
