Cross-modal multi-level analysis method and system for digital teaching resources

Abstract

The application belongs to the technical field of personalized learning, and discloses a cross-modal multi-level analysis method and system for digital teaching resources. Digital teaching resources are collected to establish a teaching resource database, which is manually annotated by experts, classified, and then subjected to modal conversion and preprocessing. Through key frame extraction and speech recognition technology, the data teaching resources are converted to image and text modalities, and a digital education resource analysis dataset is established. Based on the text and image modalities, different methods are used to preprocess the input to further obtain the embedded representation of the teaching resources, and the multi-form representation of the resources is completed according to the form of the data. Based on the deep neural network, a shared layer for joint prediction is designed, and analysis layers are respectively created for the knowledge level, subject thinking and required cognitive level of the digital education resources. The gradient normalization method is introduced to define the total loss of the task, a multi-task framework model is built, and the cross-modal multi-level analysis of the digital education resources is completed.

Description

Technical Field

[0001] This invention belongs to the field of personalized learning technology, and in particular relates to a cross-modal multi-level analysis method and system for digital teaching resources. Background Technology

[0002] Against the backdrop of rapid development in internet and artificial intelligence technologies, the education industry is undergoing tremendous changes, continuously moving towards intelligentization. The "China Smart Education Blue Book (2022)," released in 2023, points out that "smart education is an important goal of the digital transformation of education and the future direction of education." Among these, the widespread adoption of technologies such as automatic homework grading and personalized resource optimization and recommendation in smart education systems has brought great convenience to our learning and lives, ensuring not only educational equity but also enabling individualized instruction. Digital teaching resources refer to teaching resources for teachers in intelligent education, including textbooks, lesson plans, course materials, test questions, teaching videos, and other presentation formats, existing as unstructured data and in multimodal forms such as text, images, video, and audio. In-depth analysis of digital teaching resources reveals rich educational information such as knowledge levels, subject-specific thinking, and required cognitive levels. Mastering this information at each level helps us better understand digital teaching resources, thereby assisting teachers in conducting more accurate educational measurement activities.

[0003] Traditionally, the multi-level analysis of digital teaching resources is conducted by teachers or domain experts. However, this typically consumes a significant amount of time for annotators, and due to the individual subjectivity of the annotators, it is difficult to guarantee the consistency and validity of the annotation results. Therefore, how to accurately and efficiently analyze multi-level information such as knowledge levels, subject-specific thinking, and cognitive level in digital teaching resources has become an important research question in the field of smart education.

[0004] Currently, the mainstream methods applicable to the analysis of digital teaching resources at various levels fall into two categories: measurement analysis methods based on questionnaires and annotation analysis methods relying on AI technology and machine learning. While the former has rich educational theoretical support, it is significantly affected by the subjectivity of test takers and the quality of questionnaire design, and its efficiency is low, making it difficult to apply to scenarios with complex digital teaching resources. The latter can be broadly divided into traditional machine learning and deep learning methods. Traditional machine learning-based digital teaching resource analysis methods use techniques such as Naive Bayes, K-nearest neighbors, support vector machines, and random forests to classify resources based on their main content. With the development of deep learning algorithms, techniques such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have been widely used, achieving new improvements in accuracy and enabling the processing of different modalities such as text, images, videos, and audio. CNNs can extract features at different levels and granularities from digital teaching resource content in the form of graphics and text through convolutional kernels of different sizes, while RNNs can learn information from the contextual content contained in text, audio, and video sequences. The multi-level analysis task of digital teaching resources in this invention is complex and needs to be processed separately. For example, the analysis of knowledge level and subject thinking is regarded as a multi-label classification task in classification problems. Due to the sparsity and specialization of labels, the required level of cognition is relatively large. The multi-level analysis method of digital teaching resources based solely on these two types of deep learning methods still has a lot of room for improvement in terms of accuracy and stability.

[0005] Since the digital teaching resource analysis method based on convolutional neural networks is generally superior to the digital teaching resource analysis method based on recurrent neural networks in terms of versatility and predictive performance, the following discussion of existing technologies will mainly focus on the digital teaching resource analysis method based on convolutional neural networks, and will be referred to as the traditional digital teaching resource analysis method.

[0006] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0007] (1) Traditional digital teaching resource analysis methods are designed for single-modality teaching resources. When resources are mixed, separate models need to be built for analysis, which results in low versatility and processing efficiency.

[0008] (2) When analyzing tasks at different levels in traditional digital teaching resource analysis methods, the training is relatively independent, which will consume a huge amount of training time.

[0009] (3) Traditional digital teaching resource analysis methods suffer from severe overfitting and low prediction performance when faced with datasets of specific subjects and small sample sizes. Summary of the Invention

[0010] To address the problems existing in the prior art, this invention provides a cross-modal multi-level analysis method and system for digital teaching resources.

[0011] This invention is implemented as follows: a cross-modal multi-level analysis method and system for digital teaching resources, wherein the cross-modal multi-level analysis method for digital teaching resources includes:

[0012] Digital teaching resources, including test questions, textbooks, lesson plans, teaching videos, and teaching recordings, in the form of text and images, were collected from online and campus channels to establish a teaching resource database. Experts manually labeled some digital educational resources with knowledge levels, subject thinking, and cognitive level requirements. After classifying the data according to their modality, modality conversion and preprocessing were performed. Video and audio modal teaching resources were converted to image and text modalities using keyframe extraction and speech recognition techniques, thus establishing a digital educational resource analysis dataset for high school mathematics. Based on the text and image modalities, different methods were used to preprocess the input and further obtain the embedded representation of the teaching resources, completing multi-form representation of resources according to their format. A shared layer for joint prediction was designed based on deep neural networks, and analysis layers were created for the knowledge levels, subject thinking, and cognitive level requirements of the digital educational resources. Gradient normalization was introduced to define the total task loss, and a multi-task framework was built to train the model, completing cross-modal multi-level analysis of the knowledge levels, subject thinking, and cognitive level requirements of the digital educational resources.

[0013] Furthermore, the knowledge level of the digital teaching resources represents the multi-level structured knowledge concepts corresponding to the digital teaching resources; the subject-specific thinking of the digital teaching resources represents the analytical thinking path used by the corresponding subject to analyze the resources, and in a concrete sense, it refers to which theorems, properties, tools, or clustering thinking are used to solve a certain type of problem for a certain subject; the cognitive level required by the digital teaching resources, that is, the cognitive level characteristics required by the teaching resources for learners, is defined in this invention as three levels: 1, 2, and 3, representing easy, medium, and difficult, respectively.

[0014] Furthermore, the modality conversion and preprocessing after classifying the data according to its modality involves dividing digital teaching resources into video modality, image modality, text modality, and audio modality. First, for test questions and textbooks stored in image modality, part of them can be converted into text through OCR, while the other part is still saved in image form. For teaching audio data, it is processed into text materials using speech recognition tools. For teaching video data, part of it is processed into text materials through speech recognition tools, while the other part extracts key frames from the video sequence to identify the resource content as images.

[0015] Furthermore, the preprocessing of input based on text and image modalities using different methods involves further preprocessing the pre-converted and processed text and image data respectively. Text data processing includes cleaning and tag escaping operations, such as using regular expressions to filter out tags used for formatting adjustments and invalid characters like consecutive spaces and symbols, targeting the large number of LaTeX characters and HTML tags contained in the text representation of digital teaching resources. It also involves using search and replace methods to escape Chinese characters for formula tags with specific mathematical meanings.

[0016] We introduce subject-specific vocabulary from digital teaching resources and construct a domain-specific Chinese word segmentation tool to segment the text content of digital teaching resources according to words. We then read a pre-set stop word list to find and remove stop words from the digital teaching resource text.

[0017] For image modality data processing, the main tasks are to crop out the identifiable content in the image, filter out irrelevant information such as text and people, and standardize the image size.

[0018] Furthermore, the process of obtaining the embedded representation of the teaching resources includes:

[0019] For digital teaching resource data in text form, each word w has a word frequency-based vector representation in the resource set E. It can be obtained by multiplying the term frequency and the inverse document frequency:

[0020]

[0021]

[0022]

[0023] In the formula, TF w This represents word frequency, specifically the frequency of word w throughout the entire resource document E. count(w) is the total number of times word w appears, and N is the total number of occurrences. E It is the total number of words that appear in the resource document, IDF w Indicates inverse document frequency, where I is the total number of teaching resources in the resource document, IF(w,E) i ) used to statistically analyze each teaching resource E i Does the word 'w' exist? If it does, then IF(w,E) i ) = 1;

[0024] Learn the vector representation of each word w in digital teaching resources using the skip-gram model in unsupervised word embedding methods. skip-gram can infer the words in the context of a window based on the head word;

[0025] During fusion, inverse text frequency values ​​of terms are used as weights to enhance the word vectors obtained from skip-gram. Before fusion, it is necessary to separately... and By using linear layers and activation layers to unify them into the same vector space:

[0026]

[0027]

[0028]

[0029] In the formula, φ is the activation function, set to ReLU, and W t ,b t For the purpose of The hidden layer weight matrix and bias parameters, W w ,b w For the purpose of The hidden layer weight matrix and bias parameters, and This refers to word vector representations mapped to the same vector space, based on the word frequency-inverse text frequency algorithm and the unsupervised learning semantic word embedding method. w It is a word vector representation obtained through an unsupervised learning semantic word embedding method weighted by a word frequency algorithm. Therefore, we can obtain the word-level embedding representation [v1, v2, ..., v] for each digital teaching resource. W ], where W is the number of words contained in the teaching resources;

[0030] By using average pooling, we obtain the sentence-level embedding representation x for each text-based digital teaching resource. i :

[0031]

[0032] For digital teaching resource data in image format, a VGG-16 pre-trained model was used for fine-tuning on the data. During the process, the parameters of the first 13 convolutional layers and 5 pooling layers were frozen, and the last fully connected layer was replaced with a softmax activation layer. The input image was used to obtain the embedded representation of each digital teaching resource image.

[0033] Furthermore, the process of representing data through different resource formats based on its form includes:

[0034] For teaching resources that originally exist in text and audio formats, since the representation comes only from a single modality, their multi-form characteristics are...

[0035] For teaching resources that originally exist in image and video formats, their representations simultaneously derive from transformed text and image modalities. Therefore, it is necessary to fuse the embedding representations of the two parts to obtain the multi-form features x of the teaching resources. i :

[0036]

[0037] Furthermore, the process of designing a shared layer for joint prediction based on a deep neural network includes:

[0038] A deep neural network is constructed as a shared layer for joint prediction of digital teaching resources across multiple tasks. Each layer uses an activation function φ for transformation. Therefore, for each digital teaching resource E... i The shared intermediate vector after the shared layer is x' i :

[0039]

[0040]

[0041] In the formula, For the intermediate vector of the digital teaching resource layer through a linear layer, x i For digital teaching resources E i The embedding representation, W s1 W s2 Let b be the weight matrices of the two hidden layers in the shared layer. s1 ,b s2 Let φ be the bias parameters of the two hidden layers in the shared layer, and let φ be the activation function ReLU.

[0042] Furthermore, the process of constructing analysis layers for the three tasks of knowledge level analysis, subject-specific thinking analysis, and cognitive level analysis of requirements for digital teaching resources includes:

[0043] Three linear layers are constructed as classification layers for the knowledge level, subject-specific thinking, and cognitive level analysis tasks of digital teaching resources, respectively. These layers are connected in parallel to the shared layer. The shared intermediate vector x' is then used to... i When used as input, the three different classification layers can output results of knowledge hierarchy analysis, subject-specific thinking analysis, and cognitive level analysis of teaching resources:

[0044]

[0045]

[0046]

[0047] In the formula, Wc W k W s b is the weight matrix for each of the three task analysis layers: cognitive level, knowledge level, and subject-specific thinking. c ,b k ,b s These are the deviation parameters for each of the three task analysis layers. For digital teaching resources E i The predicted probabilities of cognitive level, knowledge level, and subject-specific thinking in each category.

[0048] Furthermore, the process of introducing gradient normalization to define the total loss, building a multi-task framework training model for digital teaching resources, and completing the analysis of the knowledge level, subject-specific thinking, and cognitive level of requirements of digital teaching resources includes:

[0049] Define the total loss in multi-task training for digital instructional resource analysis:

[0050]

[0051] In the formula, N is the total number of tasks to be trained, and w n For the weights of each task, l n Let l be the loss function for each task. For different tasks: cognitive level analysis is a multi-class classification problem, therefore l c Using cross-entropy loss, the analysis of knowledge hierarchy and subject-specific thinking both fall under the category of multi-label problems, therefore for l k and l s By setting a classifier for each sub-task of knowledge level analysis or subject thinking analysis, and using binary cross-entropy loss as the loss function;

[0052] To ensure that the gradient magnitude and convergence speed of each task in the loss function are continuously adjusted during training, the gradient normalization method GradNorm is introduced to dynamically adjust the weights w of each task. n :

[0053]

[0054] in, In the formula, W represents the network parameters of the last layer of the shared layer. s2 , The weighted loss w for task n at time t n (t)l n (t) is the L2 norm of the gradient of W. For calculation of all tasks The average value, r n(t) is the relative reverse training speed of task n, and α is a hyperparameter used to control the balanced training speed. The larger the value, the greater the restriction.

[0055] After calculating the grad loss, it is necessary to calculate its value for w. n The derivative of w is used at each time step after backpropagating the total loss to update the parameters of the last layer of the shared layer. n It is continuously updated to achieve dynamic adjustment of the weight of each task;

[0056] After training, for the required cognitive level, the class with the highest probability in the classifier is taken as the predicted value. For knowledge level and subject thinking, since the multi-label problem has been transformed into a binary classification problem for all knowledge levels or subject thinking examined by digital teaching resources, the union of the classes with a predicted probability greater than 0.5 in the binary classification is taken as the prediction result. If the predicted probability of all classifiers is less than 0.5, the class with the highest probability is selected as the prediction result to ensure the non-emptiness and effectiveness of the prediction.

[0057] Another objective of this invention is to provide a cross-modal multi-level analysis method and system for digital teaching resources, wherein the cross-modal multi-level analysis method and system for digital teaching resources includes:

[0058] The Digital Teaching Resource Modal Conversion and Preprocessing Module 1 is used to clean and preprocess the collected digital teaching resource topic texts.

[0059] The digital teaching resource multi-form representation module 2 is used to calculate word-level vector representations of preprocessed digital teaching resource texts based on the word frequency inverse text frequency algorithm and unsupervised learning semantic word embedding method, and obtain sentence-level digital teaching resource multi-form representations through average pooling.

[0060] Module 3, a multi-task sharing network for digital teaching resources, is used to construct a multi-task sharing network that connects the formal representation of digital teaching resources with the analysis layer of each subsequent task.

[0061] Module 4, which analyzes the cognitive level of digital teaching resources, is used to construct a deep neural network after the multi-task sharing network to analyze the cognitive level of digital teaching resources.

[0062] The Digital Teaching Resource Knowledge Hierarchy Analysis Module 5 is used to construct a deep neural network after the multi-task sharing network to analyze the knowledge hierarchy examined by the digital teaching resources.

[0063] The Digital Teaching Resources Subject Thinking Analysis Module 6 is used to construct a deep neural network after the multi-task sharing network to analyze the subject thinking of the digital teaching resources.

[0064] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0065] First, regarding the technical problems existing in the above-mentioned prior art, the creative technical effects resulting from solving these problems are described in detail below:

[0066] (1) Traditional digital teaching resource analysis methods are designed for single-modality teaching resources. When resources are mixed, separate models need to be built for analysis, resulting in low versatility and processing efficiency. This invention first performs modality conversion and preprocessing for teaching resources of different modalities, and then further processes them through feature extraction and fusion to obtain a representation of the teaching resource form. It establishes a general model that can handle multimodal teaching resources. It not only performs well in the face of complex resource scenarios, but also effectively improves processing efficiency.

[0067] (2) In traditional digital teaching resource analysis methods, the training of the three tasks—knowledge level, subject thinking, and cognitive level analysis of requirements—is relatively independent, which consumes a huge amount of training time. This invention uses a multi-task end-to-end framework to perform joint prediction around these three tasks. The total training time is approximately equal to the training time of each task individually, which can save the cost of training each task.

[0068] (3) Traditional digital teaching resource analysis methods suffer from severe overfitting and low prediction performance when faced with datasets of specific subjects and small sample sizes. This invention constructs a multi-task sharing layer to exchange effective information learned from different tasks, and uses GradNorm adaptive learning weights to effectively utilize the knowledge learned from each task to handle situations with limited sample sizes, thus ensuring the model's prediction performance.

[0069] The cross-modal, multi-level analysis method and system for digital teaching resources proposed in this invention are significantly superior to traditional digital teaching resource analysis methods in terms of cognitive level, knowledge level, and subject-specific thinking analysis of the requirements of digital teaching resources. Experiments on indicators such as accuracy and recall show that the method proposed in this invention is more effective than traditional digital teaching resource analysis methods.

[0070] Secondly, as supplementary evidence of the inventive step of the claims of this invention, it is also reflected in the following important aspects:

[0071] (1) The expected benefits and commercial value of the technical solution of this invention after transformation are as follows:

[0072] The cross-modal multi-level analysis method and system for digital teaching resources proposed in this invention can more accurately and efficiently predict the cognitive level parameters, knowledge level, and subject thinking required by digital teaching resources. It is applicable to many scenarios under smart education and online education platforms, such as personalized learning resource recommendation, and can effectively save the cost of manual annotation, thus having great commercial value.

[0073] (2) The technical solution of this invention fills a technical gap in the industry both domestically and internationally:

[0074] Due to the abstract and complex nature of digital teaching resources, traditional methods for analyzing these resources often fail to adequately represent the core content of mathematics-related digital teaching resources, resulting in generally poor performance. Furthermore, they rely heavily on large amounts of training data, making it difficult to guarantee good predictive performance in multi-label classification tasks with small to medium-sized datasets. This invention addresses this issue by preprocessing and training data from different modalities and employing a multi-task sharing strategy to exchange learning information, thereby ensuring the accuracy of the model's classification even with limited training data.

[0075] (3) Whether the technical solution of the present invention solves the technical problem that people have long wanted to solve but have never been able to solve successfully:

[0076] Traditional digital teaching resource analysis methods process each sub-task independently, which not only incurs significant training costs but also fails to effectively utilize the knowledge learned from each sub-task. This invention employs a multi-task framework, constructing a shared layer to exchange effective information learned from different tasks, and utilizes GradNorm adaptive learning weights to aid training, thereby improving the model's prediction time and accuracy.

[0077] (4) Does the technical solution of the present invention overcome technical bias?

[0078] Traditional digital teaching resource analysis methods generally represent and classify single modalities such as text and video. However, this invention can not only take multimodal resources as input, but also transform them into text modalities and image modalities through modality transformation and other processing operations, and represent them based on different strategies, making full use of more content information. Furthermore, it can build a multi-task learning framework in the future, saving learning costs while improving prediction performance.

[0079] Third, the cross-modal multi-level analysis method and system for digital teaching resources provided by this invention brings the following significant technological advancements:

[0080] 1) Multimodal learning resource format analysis: The system can process teaching resources that originally exist in the form of text, audio, images and videos. After converting them to text and image modalities, it uses unsupervised learning semantic word embedding with word frequency information weighted by the VGG-16 pre-trained model to represent different forms of digital teaching resources. It fully extracts the effective information in the text of digital teaching resources, thereby improving the accuracy of the model in the joint analysis of knowledge level, subject thinking and cognitive level of requirements.

[0081] 2) Multi-task learning framework: This system is based on deep neural networks, constructs a shared layer to exchange effective information learned in different tasks, and uses gradient normalization to adaptively learn weights, which effectively saves the cost of model training and subsequent personnel use, and improves the accuracy of model prediction on various sub-tasks.

[0082] 3) Digital Teaching Resource Preprocessing: The system performs different preprocessing operations on the main content of different forms of digital teaching resources. For text modalities, text cleaning and tag escaping are performed to analyze digital teaching resource texts containing a large number of characters, tags, and formulas. Then, a dedicated vocabulary is introduced to construct a word segmentation tool and remove stop words. For image modalities, identifiable content in the image is cropped, irrelevant information such as text and people is filtered out, and the image size is standardized. These preprocessing steps ensure the refinement and completeness of resource information, effectively improving the quality of subsequent representations and thus ensuring the performance of model predictions.

[0083] The system provided by this invention brings several technological advancements to the multi-level analysis of digital teaching resources. These advancements not only improve the accuracy and efficiency of annotation but also provide valuable tools for educational research and application.

[0084] Fourth, the mathematical model of this invention provides a systematic method through a deep learning framework for the comprehensive analysis of digital teaching resources, solving the following problems of the prior art and bringing about significant technological progress:

[0085] Technical problems with existing technologies:

[0086] 1) Limited analytical dimensions: Existing technologies often focus only on one aspect of digital teaching resources (such as content quality or accessibility), lacking comprehensive analysis of multiple dimensions such as cognitive level, knowledge level, and subject-specific thinking required for teaching resources.

[0087] 2) Lack of cross-modal analysis: Traditional analysis methods are difficult to process digital teaching resources in multiple forms such as text and images at the same time, and cannot comprehensively evaluate teaching materials containing multiple media.

[0088] 3) Coarse analysis granularity: Previous technologies often analyzed teaching resources at a high level, lacking in-depth research on details and failing to provide specific and detailed analysis results.

[0089] Significant technological advancements achieved by this invention:

[0090] 1) Multi-dimensional analysis: By combining the cognitive level, knowledge level and subject thinking of digital teaching resources for analysis, a comprehensive evaluation of teaching resources is achieved, providing more in-depth and comprehensive analysis results.

[0091] 2) Cross-modal analysis capability: This invention can process and analyze different modal data of digital teaching resources, such as text and images. By utilizing word embeddings and deep neural networks, it achieves effective analysis of different types of teaching resources.

[0092] 3) Refined analysis: By using unsupervised learning semantic word embedding method with word frequency information weighting and deep neural network, it provides multi-level analysis from word level to sentence level and then to resource level, which significantly improves the accuracy and detail of the analysis.

[0093] 4) Dynamic weight adjustment: The multi-task training framework defined by gradient normalization is introduced to realize the dynamic adjustment of the weights of different analysis tasks, ensuring the balance of the learning process of each task and the optimization of the overall analysis results.

[0094] 5) Enhanced practicality: The cross-modal multi-level analysis method provided by this invention can be widely applied to the evaluation of various digital teaching resources, providing educators and researchers with a powerful analytical tool and promoting the efficient use and continuous improvement of digital teaching resources.

[0095] Through the aforementioned advancements, this invention significantly enhances the comprehensiveness, depth, and practicality of digital teaching resource analysis, which is of great significance for promoting the optimization of educational resources and improving teaching quality.

[0096] Sixth, this invention relates to a cross-modal, multi-level analysis method for digital teaching resources, with a particular focus on using unsupervised learning semantic word embedding methods weighted by word frequency information to perform word-level embedding representations of digital teaching resource texts, and obtaining sentence-level embedding representations through average pooling. It also introduces the process of fine-tuning image-based digital teaching resource data using a VGG-16 pre-trained model, and a method for designing a shared layer for joint prediction based on deep neural networks.

[0097] The main parameters, algorithms, and mathematical models address the following problems, along with the technological advancements:

[0098] 1) Combination of inverse text frequency algorithm with unsupervised word embedding:

[0099] Traditional text embedding methods fail to fully capture the importance of words within a specific set of digital teaching resources.

[0100] This invention utilizes the Term Frequency-Inverse Text Frequency (TF-IDF) algorithm to assign weights to each word, combined with unsupervised word embedding methods (such as the skip-gram model). This ensures that the embedding representation considers both the universality of words across the global document set and their uniqueness within specific teaching resources. Technological advancement: By integrating TF-IDF information and the skip-gram model, this method can more accurately represent words in digital teaching resources, providing high-quality input data for subsequent multi-level analysis.

[0101] 2) Fine-tuning of the VGG-16 pre-trained model on image data:

[0102] Directly using pre-trained models is not suitable for image data in specific domains, such as images in digital teaching resources. This invention freezes the parameters of the first part of the convolutional and pooling layers of the VGG-16 model, and only fine-tunes the fully connected layers to adapt to the characteristics of digital teaching resource images. By fine-tuning the pre-trained model, this method can reduce overfitting to new datasets while maintaining model performance, thus improving the processing capability of digital teaching resource images.

[0103] 3) Design of a joint prediction sharing layer based on deep neural networks:

[0104] Traditional multi-task learning methods cannot effectively share information between different tasks. This invention designs a sharing layer based on a deep neural network, which can receive embedded representations of digital teaching resources from different modalities and perform joint prediction by sharing intermediate vectors.

[0105] By designing a shared layer, this method can more effectively utilize the potential correlations between different modal data, thereby improving the accuracy and efficiency of cross-modal prediction.

[0106] This invention provides a complete and efficient solution for cross-modal, multi-level analysis of digital teaching resources by combining methods such as word frequency information weighting, unsupervised word embedding, pre-trained model fine-tuning, and deep neural network design. These technological advancements not only improve the accuracy of teaching resource processing but also provide strong support for subsequent applications such as educational data analysis and personalized teaching. Attached Figure Description

[0107] Figure 1 This is a flowchart of a cross-modal multi-level analysis method for digital teaching resources provided in an embodiment of the present invention;

[0108] Figure 2This is a schematic diagram of the cross-modal multi-level analysis method for digital teaching resources provided in this embodiment of the invention;

[0109] Figure 3 This is a schematic diagram of the cross-modal multi-level analysis system for digital teaching resources provided in an embodiment of the present invention:

[0110] In the diagram, 1. Digital teaching resource modality conversion and preprocessing module; 2. Digital teaching resource multi-form representation module; 3. Digital teaching resource multi-task sharing network module; 4. Digital teaching resource cognitive level analysis module; 5. Digital teaching resource knowledge hierarchy analysis module; 6. Digital teaching resource subject thinking analysis module. Detailed Implementation

[0111] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0112] Example 1: Cross-modal multilevel analysis of digital teaching resources for mathematics

[0113] On an educational platform, digital teaching resources for mathematics include a wealth of textual descriptions and graphic images, with the goal of improving the searchability and accessibility of these resources, as well as enhancing the learning experience.

[0114] 1) Text processing:

[0115] The TF-IDF (Term Frequency-Inverse Document Frequency) algorithm is applied to calculate the weight of each word in the mathematical resource document.

[0116] The skip-gram model is used to learn the vector representation of each word, thereby obtaining the word vectors of the text resource.

[0117] We apply weights based on word frequency and inverse document frequency to word vectors, map them to the same vector space through linear and ReLU activation layers, and then obtain sentence-level embedding representations of the text through average pooling.

[0118] 2) Image processing:

[0119] Feature extraction of images from mathematical resources is performed using the VGG-16 network.

[0120] Based on the VGG-16 pre-trained model, the last layer is adjusted to adapt to specific mathematical resource images, and the accuracy of feature extraction is improved by fine-tuning the trained model.

[0121] Obtain the embedded representation of each image resource.

[0122] 3) Multilevel analysis:

[0123] A multi-task sharing network is constructed, and separate analysis layers are set up for analysis tasks based on knowledge level, subject thinking, and cognitive level. The embedded representations of the above text resources and image resources are transmitted through the network to output multi-level digital teaching resource analysis results for mathematics.

[0124] Example 2: Cross-modal multilevel analysis of historical digital teaching resources

[0125] On an online teaching platform, digital teaching resources for the history subject include text descriptions, images, and video materials, with the goal of enhancing the interactivity and teaching effectiveness of these resources through cross-modal analysis.

[0126] 1) Text processing:

[0127] The TF-IDF algorithm is used to process the text content of historical resources to obtain the importance of each word in the text.

[0128] We use the skip-gram model, an unsupervised learning method, to obtain the vector representation of each word in the text.

[0129] The word vectors obtained by combining word frequency and inverse document frequency are then integrated into a text embedding representation of the resource using average pooling.

[0130] 2) Image processing:

[0131] The pre-trained VGG-16 model is used to analyze images from historical resources and extract image features.

[0132] The last layer of the network was adjusted to suit the characteristics of the historical discipline, and the network was fine-tuned to adapt to the feature extraction of historical images.

[0133] 3) Video processing:

[0134] The video material is converted into text modality and image modality through speech recognition and keyframe extraction, respectively. Then, feature extraction is performed based on the text and image using the above processing methods.

[0135] 4) Multilevel analysis:

[0136] A multi-task sharing network is constructed, and separate analysis layers are set up for analysis tasks based on knowledge level, subject thinking, and cognitive level, so as to obtain multi-level digital teaching resource analysis results.

[0137] Through these two examples, digital teaching resources for different disciplines can achieve effective cross-modal analysis and feature extraction, thereby improving the efficiency of teaching resource utilization and teaching quality.

[0138] like Figure 1 As shown, the cross-modal multi-level analysis method and system for digital teaching resources provided in this embodiment of the invention includes the following steps:

[0139] S101 collects digital teaching resources in the form of text, images, etc., from the Internet, campuses, etc., to establish a teaching resource database. Experts manually annotate the knowledge level, subject thinking, and cognitive level of some digital educational resources. After classifying the data according to their modality, modality conversion and preprocessing are performed. The digital teaching resources in the video modality are converted into image modality through keyframe extraction, and the digital teaching resources in the audio modality are converted into text modality through speech recognition. In this way, a digital educational resource analysis dataset for mathematics in high school is established.

[0140] S102, based on text and image modalities, uses different methods to preprocess the input and further obtains the embedded representation of teaching resources, and completes the multi-form representation of resources according to the form to which the data belongs;

[0141] S103 is a shared layer for joint prediction designed based on deep neural networks. It also creates analysis layers for different levels of digital educational resources, such as knowledge level, subject thinking, and cognitive level of requirements. It introduces gradient normalization to define the total task loss, builds a multi-task framework to train the model, and completes cross-modal multi-level analysis of knowledge level, subject thinking, and cognitive level of requirements of digital educational resources.

[0142] The working principle of the cross-modal multi-level analysis method and system for digital teaching resources is as follows:

[0143] S101: Establishment of Digital Teaching Resource Database and Multi-level Analysis and Annotation of Digital Teaching Resources

[0144] Establishment of a digital teaching resource database: Collect teaching resources such as textbooks and real test questions from various schools, input the text using OCR, and establish a digital teaching resource database composed of digital teaching resources.

[0145] Multi-level analysis and annotation of digital teaching resources: Experts manually annotate the knowledge levels, subject-specific thinking, and required cognitive level information of digital teaching resources. Taking test resources as an example, the knowledge levels include "systems of linear inequalities in one variable" and "addition and subtraction of rational numbers"; the subject-specific thinking includes "using the solution set of a system of linear inequalities in one variable to determine the range of undetermined letters" and "the application of addition and subtraction of rational numbers in practical problems"; the required cognitive level includes three levels: easy, medium, and difficult, identified by the numbers 1, 2, and 3.

[0146] Modal conversion and preprocessing: Digital teaching resources are divided into video modal, image modal, text modal and audio modal. For test questions and textbooks stored in image modal data, part of them can be converted into text through OCR, and the other part can still be saved in image form. For teaching audio data, it is processed into text materials through speech recognition tools. For teaching video data, part of it is processed into text materials through speech recognition tools, and the other part is extracted from the video sequence and the resource content is identified as images.

[0147] S102: Preprocessing and Formal Embedding Representation of Digital Teaching Resources

[0148] Text modal preprocessing: Addressing the large number of LaTeX formatting characters, HTML tags, and CSS tags contained in the text representation, regular expressions are used to filter out invalid characters such as tags used for formatting adjustments and consecutive spaces. Find and replace methods are used to perform Chinese escaping on formula character tags with specific mathematical meanings. Word segmentation and stop word removal: A subject-specific Chinese word segmentation tool is built to segment digital teaching resource text according to words and filter out pre-defined stop words.

[0149] Image modality preprocessing mainly involves cropping the identifiable content from the image, filtering out irrelevant information such as text and people, and standardizing the image size.

[0150] Multi-form representation of digital teaching resources: For the text representation, unsupervised learning semantic word embeddings weighted by word frequency information are first used to perform word-level embedding representation of the digital teaching resource text, and then average pooling is used to obtain the embedding representation. In word-level representation, algorithms based on inverse text frequency (IF) can reflect the importance of each word from a frequency perspective, while unsupervised learning semantic word embedding methods can learn the connections between words in the corpus from the distance between words. Strengthening the unsupervised learning semantic word embeddings with word frequency information yields richer text embedding features. For the image representation, a VGG-16 pre-trained model is fine-tuned based on the frozen convolutional and pooling layers to obtain image embedding features. Depending on the original form of the data, a fusion strategy is selected to ultimately obtain multi-form features of the teaching resources.

[0151] S103: Definition of Multi-Task Loss Function, Model Building and Prediction

[0152] Multi-task loss function definition: The total loss is the weighted sum of the losses of each task. GradNorm is used to adaptively adjust the weights of each sub-task based on the gradient magnitude and convergence speed. For the specific loss function of each sub-task, the cognitive level analysis is a multi-classification task, so cross-entropy is used as the loss. The knowledge level and subject thinking analysis are both multi-label tasks, so a classifier needs to be set for each knowledge level or subject thinking analysis sub-task, and binary cross-entropy is used as the loss function.

[0153] Multi-task model construction: The model consists of two parts: a shared layer and an analysis layer. The shared layer is a deep neural network composed of two hidden layers, while the analysis layer is composed of parallel linear layers for each of the three tasks to output the predicted probability.

[0154] The prediction results include three parts: analysis of the required cognitive level, analysis of knowledge level, and analysis of subject-specific thinking. For the required cognitive level, the class with the highest probability in the classifier is taken as the predicted value. For knowledge level and subject-specific thinking, since the multi-label problem has been transformed into a binary classification problem of all knowledge levels or subject-specific thinking examined by digital teaching resources, the union of the classes with a predicted probability greater than 0.5 in the binary classification is taken as the analysis result. If the predicted probabilities of all classifiers are less than 0.5, the class with the highest probability is selected as the analysis result.

[0155] Through the above steps, this method can automatically analyze the knowledge level, subject thinking, and cognitive level of digital teaching resources at multiple levels, thereby providing learners and educators with more accurate analysis and recommendations of digital teaching resources.

[0156] The cross-modal multi-level analysis method and system principle diagram for digital teaching resources provided in this invention are as follows: Figure 2 As shown.

[0157] like Figure 3 As shown, the cross-modal multi-level analysis method and system for digital teaching resources provided in this embodiment of the invention includes:

[0158] The Digital Teaching Resource Modal Conversion and Preprocessing Module 1 is used to perform modal conversion and preprocessing operations on the collected digital teaching resources based on different forms and modalities.

[0159] The digital teaching resource multi-form representation module 2 extracts the embedded representations of teaching resources of different modalities and then performs multi-form representation based on the original resource format.

[0160] Module 3, a multi-task sharing network for digital teaching resources, is used to construct a multi-task sharing network that connects the formal representation of digital teaching resources with the analysis layer of each subsequent task.

[0161] Module 4, which analyzes the cognitive level of digital teaching resources, is used to construct a deep neural network after the multi-task sharing network to analyze the cognitive level of digital teaching resources.

[0162] The Digital Teaching Resource Knowledge Hierarchy Analysis Module 5 is used to construct a deep neural network after the multi-task sharing network to analyze the knowledge hierarchy examined by the digital teaching resources.

[0163] The Digital Teaching Resources Subject Thinking Analysis Module 6 is used to construct a deep neural network after the multi-task sharing network to analyze the subject thinking of the digital teaching resources.

[0164] The detailed working principle of the cross-modal multi-level analysis method and system for digital teaching resources is as follows:

[0165] 1. Digital Teaching Resource Modality Conversion and Preprocessing Module 1:

[0166] Modal conversion: For image data, part is converted into text through OCR, while the other part remains in image form; for audio data, it is processed into text material using speech-to-text tools; for video data, part is processed into text material through speech-to-text, while the other part extracts images from the sequence that can identify the resource content.

[0167] Preprocessing: Preprocessing includes two parts: text modality preprocessing and image modality preprocessing. The former includes filtering or escaping LaTeX and front-end tags; the latter includes cropping and unifying the size of the image.

[0168] 2. Digital Teaching Resource Multi-Form Representation Module 2:

[0169] Multi-form representation of digital teaching resources: For text modality and image modality, unsupervised learning semantic word embedding with word frequency information weighting and VGG-16 fine-tuning are used to obtain the embedding representation of each modality. Then, multi-form features of teaching resources are obtained based on the original form.

[0170] 3. Digital Teaching Resource Multi-Task Sharing Network Module 3:

[0171] This module constructs a multi-task sharing network consisting of two hidden layers and an activation layer for three tasks. It takes sentence-level representations of digital teaching resources as input and then connects them to the analysis layers of each task for prediction.

[0172] 4. Analysis of Cognitive Levels Required for Digital Teaching Resources (Module 4):

[0173] This module constructs a deep neural network after the multi-task sharing network to analyze the cognitive level required by digital teaching resources. The analysis of the required cognitive level is a multi-classification task, and the required cognitive level will be divided into three categories: 1, 2, and 3, representing easy, medium, and difficult, respectively.

[0174] 5. Digital Teaching Resource Knowledge Hierarchy Analysis Module 5:

[0175] This module constructs a deep neural network after a multi-task shared network to analyze the knowledge levels of digital teaching resources. Since a single digital teaching resource examines multiple knowledge levels, knowledge level analysis falls under the category of multi-label tasks.

[0176] 6. Digital Teaching Resources Subject Thinking Analysis Module 6:

[0177] This module constructs a deep neural network after a multi-task shared network to analyze the subject-specific thinking behind digital teaching resources. Subject-specific thinking analysis, like knowledge hierarchy analysis, belongs to a multi-classification task.

[0178] In summary, this system utilizes deep learning methods to automatically label the cognitive level, knowledge level, and subject-specific thinking required for digital teaching resources. This not only helps students and teachers to more comprehensively grasp and select digital teaching resources and achieve personalized teaching, but also contributes to the practice and implementation of smart education.

[0179] The symbols involved in the embodiments of the present invention are shown in Table 2.

[0180] Table 2. Symbols involved in the embodiments of the present invention.

[0181]

[0182]

[0183]

[0184] The cross-modal multi-level analysis method and system for digital teaching resources provided in this invention specifically include:

[0185] (1) Collect digital teaching resources, such as test questions, textbooks, lesson plans, and teaching recordings, mainly in the form of text and audio modalities from the Internet, campus and other channels, and establish a teaching resource database. Experts manually annotate the knowledge level, subject thinking and cognitive level of some digital education resources. After classifying the data according to the modal, modal conversion and preprocessing are carried out. The audio modal data teaching resources are converted to the text modal through speech recognition technology, thereby establishing a digital education resource analysis dataset for mathematics in high school.

[0186] (2) Based on the text modality, different methods are used to preprocess the input and further obtain the embedded representation of the teaching resources. The multi-form representation of the resources is completed according to the form to which the data belongs.

[0187] (3) Based on the design of a shared layer for joint prediction based on deep neural networks, and create analysis layers for the knowledge level, subject thinking and cognitive level of requirements of digital education resources respectively. Introduce gradient normalization method to define the total task loss, build a multi-task framework to train the model, and complete the cross-modal multi-level analysis of the knowledge level, subject thinking and cognitive level of requirements of digital education resources.

[0188] Step one provided in this embodiment of the invention specifically includes:

[0189] (1.1) Collect test materials from online and campus channels to establish a teaching resource database. Experts will manually label the knowledge level, subject thinking, and cognitive level of the digital teaching resources.

[0190] This invention targets high school mathematics. First, it collects and inputs real digital teaching resources from textbooks published by People's Education Press, Beijing Normal University Press, and East China Normal University Press, as well as tests from various schools. Then, it breaks down the sub-questions in several major questions according to the format of main topic stem plus sub-question stem, and establishes a digital teaching resource library consisting of 57,555 high school mathematics test questions.

[0191] Secondly, experts manually annotated the knowledge levels, subject-specific thinking, and required cognitive levels of the digital teaching resources. Knowledge levels represent the knowledge concepts tested by the digital teaching resources; due to the multi-level nature of knowledge, only the lowest level of knowledge is considered here. Subject-specific thinking refers to the methods, approaches, or tools used to solve a particular type of problem. A single digital teaching resource may test multiple subject-specific thinking or knowledge levels, separated by spaces during annotation. The required cognitive level of the digital teaching resources is divided into three levels: 1, 2, and 3, representing easy, medium, and difficult, respectively.

[0192] (1.2) After classifying the data according to their mode, perform mode transformation and preprocessing:

[0193] Digital teaching resources are categorized into video, image, text, and audio modal data. For image modal data such as test questions and textbooks, some can be converted into text using OCR, while others remain as images. For audio teaching data, speech recognition tools are used to process it into text. For video teaching data, some audio is processed into text using speech recognition tools, while keyframes from the video sequence are extracted and used to identify the resource content as images.

[0194] (1.3) After text cleaning and tag escaping, the content of the questions in the digital teaching resources is used as input text to establish a multi-level analysis dataset of high school digital teaching resources:

[0195] After collecting the question stem text after breaking down the sub-questions, further preprocessing is required. Considering the characteristic of digital teaching resources containing a large number of mathematical formulas, the text generally contains a large number of LaTeX formatting characters, HTML tags, and CSS tags. Therefore, the text needs to be cleaned. Regular expressions are used to filter out invalid characters such as tags used for formatting and consecutive spaces. Methods such as find and replace are used to perform Chinese escaping on formula character tags with specific mathematical meanings. After processing, the text is saved as a txt file, with each line representing one digital teaching resource.

[0196] Next, the training set and the test set are divided in a 4:1 ratio, and the digital teaching resources in the training set are randomly shuffled. The preprocessed digital teaching resource text is used as input, and the knowledge level and subject thinking of the digital teaching resources are used as labels to form the high school digital teaching resource multi-level analysis dataset used by this invention to train the model.

[0197] Step two provided in this embodiment of the invention specifically includes:

[0198] (2.1) Perform preprocessing operations on the text, such as word segmentation and stop word removal:

[0199] By introducing specialized vocabulary from high school mathematics, and building a domain-specific Chinese word segmentation tool based on jieba, the digital teaching resource text is segmented according to words. A pre-set list of stop words is read to find and remove stop words in the text.

[0200] (2.2) The word2vec method, an unsupervised learning semantic word embedding method using TF-IDF weighted by word frequency information, is used to perform word-level embedding representation of digital teaching resource texts:

[0201] For each digital teaching resource, each word w has a TF-IDF representation in the digital teaching resource set E. It can be obtained by multiplying the term frequency and the inverse document frequency:

[0202]

[0203]

[0204]

[0205] In the formula, TF w This represents word frequency, specifically the frequency of word w in the entire digital teaching resource document E. count(w) is the total number of times word w appears, and N is the total number of occurrences.E It is the total number of words that appear in digital teaching resource documents, IDF w Represents the inverse document frequency, where I is the total number of digital teaching resources in the digital teaching resource document, IF(w,E) i ) Used to calculate E for each question i Does it contain the word w? If it does, then IF(w,E) i ) = 1, in practice we screened out The words, and only the TF-IDF of the 50,000 most frequently occurring words;

[0206] Next, the skip-gram model in the unsupervised learning semantic word embedding method is used to learn the vector representation of each word w in the digital teaching resources. skip-gram can infer the words in the context of a window based on the central word. The vector dimension is set to 50.

[0207] In terms of fusion strategy, TF-IDF values ​​are chosen as weights to enhance the word vectors obtained from skip-gram. Before fusion, it is necessary to separately... and By using linear layers and activation layers to unify them into the same vector space:

[0208]

[0209]

[0210]

[0211] In the formula, φ is the activation function, set to ReLU, and W t ,b t For the purpose of The hidden layer weight matrix and bias parameters, W w ,b w For the purpose of The hidden layer weight matrix and bias parameters, and This refers to the word vector representations of the TF-IDF algorithm (based on word frequency and inverse text frequency) and the Word2vec unsupervised semantic word embedding method, both mapped to the same vector space. w It is a Word2vec word vector representation weighted by word frequency information. From this, we can obtain the word-level embedding representation [v1, v2, ..., v] of each digital teaching resource. W ], where W is the number of words contained in the digital teaching resources;

[0212] (2.3) By using average pooling, the sentence-level formal representation of digital teaching resources is calculated:

[0213] The formal representation x of each digital teaching resource is obtained by using average pooling. i :

[0214] x i =avgpool([v1,v2,…,v W ]).

[0215] Step three provided in this embodiment of the invention specifically includes:

[0216] (3.1) Based on deep neural networks, design a shared layer for joint prediction:

[0217] By constructing a shared layer using a deep neural network for multi-task joint prediction, and transforming each layer using an activation function φ, the prediction of digital teaching resource E can be achieved. i The shared intermediate vector after the shared layer is x' i :

[0218]

[0219]

[0220] In the formula, For the intermediate vector of the digital teaching resource layer through a linear layer, x i For digital teaching resources E i The embedding representation, W s1 W s2 Let b be the weight matrices of the two hidden layers in the shared layer. s1 ,b s2 Let φ be the bias parameters of the two hidden layers in the shared layer, and let φ be the activation function ReLU.

[0221] (3.2) For the three tasks of cognitive level, knowledge level, and subject-specific thinking required for digital teaching resources, their respective analytical layers are constructed:

[0222] Three linear layers are constructed as the analysis layers for the required cognitive level, knowledge level, and subject-specific thinking analysis tasks, respectively. These layers are connected in parallel to the shared layer. The shared intermediate vector x' is then used. i When used as input, the three different analysis layers can output the cognitive level, knowledge level, and subject-specific thinking analysis results required by digital teaching resources:

[0223]

[0224]

[0225]

[0226] In the formula, W c W k W s b is the weight matrix for each of the three task analysis layers: cognitive level, knowledge level, and subject-specific thinking. c ,b k ,b s These are the deviation parameters for each of the three task analysis layers. For digital teaching resources E i The predicted probabilities of the required cognitive level, knowledge level, and subject-specific thinking in each category;

[0227] (3.2) Gradient normalization is introduced to define the total loss, a multi-task framework is built to train the model, and the analysis of the cognitive level, knowledge level and subject thinking required by digital teaching resources is completed:

[0228] First, define the total loss for multi-task training:

[0229]

[0230] In the formula, N is the total number of tasks to be trained, and w n For the weights of each task, l n Let l be the loss function for each task. For different tasks: the required cognitive level analysis is a multi-class classification problem, therefore l c Using cross-entropy loss, both knowledge hierarchy and subject-specific thinking analysis fall under the category of multi-label problems, therefore for l k and l s By setting a classifier for each knowledge level or subject-specific thinking analysis task, and using binary cross-entropy loss as the loss function;

[0231] To ensure that the gradient magnitude and convergence speed of each task in the loss function are continuously adjusted during training, this invention introduces the gradient normalization method GradNorm, thereby dynamically adjusting the weights w of each task. n :

[0232]

[0233] in, In the formula, W represents the network parameters of the last layer of the shared layer. s2 , The weighted loss w for task n at time t n (t)l n (t) is the L2 norm of the gradient of W. For calculation of all tasks The average value, r n(t) is the relative reverse training speed of task n, and α is a hyperparameter used to control the balanced training speed. The larger the value, the greater the restriction.

[0234] After calculating the grad loss, it is necessary to calculate its approximation for w. n The derivative of w is used at each time step after backpropagating the total loss to update the parameters of the last layer of the shared layer. n It is continuously updated to achieve dynamic adjustment of the weight of each task;

[0235] In implementation, the multi-task framework mentioned in this step can adopt two strategies:

[0236] ① Jointly train classifiers for three tasks (knowledge hierarchy analysis, subject-specific thinking analysis, and required cognitive level analysis). The total loss from GradNorm is composed of the sub-losses from the above three tasks. In this case, N=3, Loss=w k ·l k +w s ·l s +w c ·l c ;

[0237] ② Jointly train the classifiers for two tasks (knowledge hierarchy analysis and subject-specific thinking analysis). The total loss from GradNorm is composed only of the sub-losses from the two tasks mentioned above. The required cognitive level analysis is not involved in the multi-task learning process but is performed separately. In this case, N=2, Loss1=w k ·l k +w s ·l s Loss2 = l c ;

[0238] Furthermore, this invention uses Adam as the optimizer to change the learning rate, and sets the penalty factor gamma for updating the learning rate to 0.5, which is applied after every 5 epochs of training. After training, for the required cognitive level, the class with the highest probability in the classifier is taken as the predicted value. For knowledge level and subject-specific thinking, since the multi-label problem has been transformed into a binary classification problem for all knowledge levels or subject-specific thinking examined by digital teaching resources, the union of the classes with predicted probabilities greater than 0.5 in the binary classification is taken as the prediction result. If the predicted probabilities of all classifiers are less than 0.5, the class with the highest probability is selected as the prediction result to ensure the non-emptiness and effectiveness of the prediction.

[0239] In this embodiment of the invention, the main development environment includes: Windows 10, GTX 3060, PyTorch 2.0.0, Python 3.9.16, and the specific hyperparameter settings of the model are shown in Table 3 below.

[0240] Table 3. Hyperparameter settings of the experimental model

[0241] batch_size 32 epoch 50 learning_rate 0.01 hidden_layer_size 200

[0242] This invention compares a cross-modal, multi-level analysis method and system for digital teaching resources with traditional digital teaching resource analysis methods, specifically comparing accuracy and recall scores. Accuracy (ACC) is the frequency of correct predictions out of a total number of predictions; it represents the proportion of true and true negative predictions to all samples. Higher accuracy indicates better model classification performance. Recall measures the ratio of true to true negative predictions and is an effective indicator of prediction performance when classes are imbalanced. Higher recall indicates a better ability of the model to distinguish between positive and negative samples.

[0243] This invention compares a cross-modal multilevel analysis method and system for digital teaching resources with traditional digital teaching resource analysis methods. To ensure fairness in the comparison, the corresponding hyperparameters of the same modules in both models are set to be the same. The comparison results of the accuracy score and recall score of a cross-modal multilevel analysis method and system for digital teaching resources and traditional digital teaching resource analysis methods on this dataset are shown in Table 1.

[0244] Table 1 Comparison of experimental results from different methods

[0245]

[0246] Experimental results show that the cross-modal multi-level analysis method and system for digital teaching resources proposed in this invention improves accuracy and recall across all tasks, including knowledge level, subject-specific thinking, and cognitive level of requirements. This indicates that the present invention, after manually labeling and preprocessing the collected digital teaching resources, constructs a model to obtain a formal representation of the teaching resources; designs a shared layer for joint prediction; and creates separate analysis layers for knowledge level, subject-specific thinking, and cognitive level of requirements of digital teaching resources. Gradient normalization is introduced to define the total task loss, and a multi-task framework is built to train the model, thereby completing the labeling of knowledge level, subject-specific thinking, and cognitive level of requirements of digital teaching resources. This improves the model's accuracy in the labeling tasks of knowledge level, subject-specific thinking, and cognitive level of requirements of digital teaching resources, and the results are superior to traditional digital teaching resource analysis methods. Experiments show that the cross-modal multi-level analysis method and system for digital teaching resources proposed in this invention is more effective than traditional digital teaching resource analysis methods in terms of accuracy and recall. In conclusion, this invention has the best experimental results.

[0247] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A cross-modal, multi-level analysis method for digital teaching resources, characterized in that, include: Digital teaching resources, including test questions, textbooks, lesson plans, teaching videos, and teaching recordings, in text and image formats, were collected from online and campus channels to establish a teaching resource database. Experts manually labeled some of the digital educational resources with their knowledge levels, subject-specific thinking, and cognitive level requirements. The data underwent modal transformation and preprocessing. Video and audio modal teaching resources were converted to image and text modalities using keyframe extraction and speech recognition techniques, thus establishing a digital educational resource analysis dataset for high school mathematics. Based on the text and image modalities, different methods were used to preprocess the input and further obtain embedded representations of the teaching resources. Multi-form representation of resources was completed according to the data's format. A shared layer for joint prediction was designed based on deep neural networks, and analysis layers were created for the knowledge levels, subject-specific thinking, and cognitive level requirements of the digital educational resources. Gradient normalization was introduced to define the total task loss, and a multi-task framework was built to train the model, completing a cross-modal multi-level analysis of the knowledge levels, subject-specific thinking, and cognitive level requirements of the digital educational resources. The process of designing a shared layer for joint prediction based on a deep neural network includes: By constructing a shared layer using a deep neural network for joint prediction across multiple tasks, and transforming each layer using an activation function ϕ, the prediction of digital teaching resources can be optimized. The shared intermediate vector after the shared layer is : ； ； In the formula, This is the intermediate vector of the digital teaching resource layer that passes through a linear layer. For digital teaching resources Embedded representation, , Here are the weight matrices for the two hidden layers in the shared layer. , The bias parameters of the two hidden layers in the shared layer, and the activation function. For ReLU; The process of constructing the respective analytical layers for the three tasks of cognitive level, knowledge level, and subject-specific thinking required for digital teaching resources includes: Three linear layers are constructed as the analysis layers for the required cognitive level, knowledge level, and subject-specific thinking analysis tasks, respectively. These layers are connected in parallel after the shared layer. The shared intermediate vector is then used. When used as input, the three different classification layers can output the cognitive level, knowledge level, and subject-specific thinking analysis results required by digital teaching resources: ； ； ； In the formula, , , The weight matrices are for the three task analysis layers: cognitive level, knowledge level, and subject-specific thinking. , , These are the deviation parameters for each of the three task analysis layers. , , For digital teaching resources The predicted probabilities of the required cognitive level, knowledge level, and subject-specific thinking in each category; The process of introducing gradient normalization to define the total loss, building a multi-task framework to train the model, and completing the analysis of the cognitive level, knowledge level, and subject-specific thinking required by digital teaching resources includes: Define the total loss of multi-task training in multi-level analysis of digital teaching resources: ； In the formula, The total number of tasks to be trained. Assign weights to each task. Let be the loss function for each task; for different tasks: the required cognitive level analysis belongs to a multi-class classification problem, therefore... Using cross-entropy loss, both knowledge hierarchy and subject-specific thinking analysis fall under the category of multi-label problems, therefore for and By setting a classifier for each knowledge level or subject-specific thinking analysis task, and using binary cross-entropy loss as the loss function; To ensure Each task can continuously adjust its gradient magnitude and convergence speed during training. This is achieved by introducing the GradNorm gradient normalization method, which dynamically adjusts the weights of each task. : ； in, In the formula That is, the network parameters of the last layer of the shared layer. , for Constant Task Loss with rights for The L2 norm of the gradient, For calculation of all tasks The average value, It is a task The relative reverse training speed, It is a hyperparameter used to control the speed of balance training; the larger the value, the greater the restriction. Calculate Then, it is necessary to calculate its value for... The derivative of the total loss at each moment After backpropagation updates the parameters of the last layer of the shared layer, the parameters will be adjusted according to the aforementioned derivatives. It is continuously updated to achieve dynamic adjustment of the weight of each task; After training, for the required cognitive level, the class with the highest probability in the classifier is taken as the predicted value. For knowledge level and subject thinking, since the multi-label problem has been transformed into a binary classification problem for all knowledge levels or subject thinking examined by digital teaching resources, the union of the classes with a predicted probability greater than 0.5 in the binary classification is taken as the prediction result. If the predicted probability of all classifiers is less than 0.5, the class with the highest probability is selected as the prediction result to ensure the non-emptiness and effectiveness of the prediction.

2. The cross-modal multi-level analysis method for digital teaching resources according to claim 1, characterized in that, The knowledge hierarchy of the digital teaching resources refers to the multi-level structured knowledge concepts corresponding to the digital teaching resources; the subject-specific thinking of the digital teaching resources refers to the analytical thinking path used by the corresponding subject to analyze the resources, and in a concrete sense, it refers to which theorems, properties, tools, or clustering thinking are used to solve a certain type of problem for a certain subject; the cognitive level required by the digital teaching resources, that is, the cognitive level characteristics required by the teaching resources for learners, is defined in this invention as three levels: 1, 2, and 3, representing easy, medium, and difficult, respectively.

3. The cross-modal multi-level analysis method for digital teaching resources according to claim 1, characterized in that, The process of classifying data according to its modality and then performing modal conversion and preprocessing includes: First, for test questions and textbooks stored as image modal data, part of them can be converted into text through OCR, while the other part is still saved as images; for teaching audio data, it is processed into text materials using speech recognition tools; for teaching video data, part of it is processed into text materials by using speech recognition tools, while the other part extracts key frames from the video sequence and identifies the resource content as images; the classification means dividing digital teaching resources into video modal, image modal, text modal and speech modal categories.

4. The cross-modal multi-level analysis method for digital teaching resources according to claim 1, characterized in that, Text cleaning and tag escaping operations for digital teaching resources refer to using regular expressions to filter out tags used for formatting adjustments and invalid characters such as consecutive spaces and consecutive symbols from the large number of LaTeX characters, HTML tags, and front-end layout tags contained in the text representation of digital teaching resources, and using search and replace methods to escape Chinese characters of formula characters with specific mathematical meanings.

5. The cross-modal multi-level analysis method for digital teaching resources according to claim 1, characterized in that, The text is preprocessed by word segmentation and stop word removal. The specific process includes: introducing subject-specific vocabulary, constructing a domain-specific Chinese word segmentation tool, segmenting the digital teaching resource text according to words; reading a pre-set stop word list, and finding and removing stop words in the text.

6. The cross-modal multi-level analysis method for digital teaching resources according to claim 1, characterized in that, The unsupervised learning semantic word embedding method, which uses word frequency information as a weight, performs word-level embedding representations of digital teaching resource texts. Then, through average pooling, it calculates sentence-level embedding representations of the digital teaching resources. The specific process includes: For each digital teaching resource, each word In digital teaching resource collection Representation based on term frequency and inverse text frequency algorithm It can be obtained by multiplying the term frequency and the inverse document frequency: ； ； ； In the formula, Indicate word frequency, i.e., word frequency Throughout the digital teaching resource document Frequency of occurrence in It is a word Total number of occurrences It is the total number of words that appear in the digital teaching resource documents. Indicates inverse document frequency. It represents the total number of digital teaching resources in the digital teaching resource document. Used to calculate the results for each question. Does it contain words? If included ; Learn each word in digital teaching resources using the skip-gram model in the unsupervised word embedding method. vector representation Skip-gram can infer the words in the context of a window based on the headword; During fusion, information based on the inverse text frequency algorithm is used as weights to enhance the word vectors obtained from skip-gram. Before fusion, it is necessary to separately analyze... and By using linear layers and activation layers to unify them into the same vector space: ； ； ； In the formula, The activation function is set to ReLU. , For the purpose of The hidden layer weight matrix and bias parameters, , For the purpose of The hidden layer weight matrix and bias parameters, and It refers to the word vector representation based on the word frequency inverse text frequency algorithm and the unsupervised learning semantic word embedding method, mapped to the same vector space. It is an unsupervised learning semantic word vector representation weighted by word frequency information; thus, we can obtain the word-level embedding representation of each digital teaching resource. ,in The number of words contained in digital teaching resources; Average pooling is used to obtain the embedded representation of each digital teaching resource. : For digital teaching resource data in image format, a VGG-16 pre-trained model was used for fine-tuning on the data. During the process, the parameters of the first 13 convolutional layers and 5 pooling layers were frozen, and the last fully connected layer was replaced with a softmax activation layer. The input image was used to obtain the embedded representation of each digital teaching resource image. .

7. A cross-modal multi-level analysis system for digital teaching resources implementing the analysis method as described in any one of claims 1-6, characterized in that, include: The digital teaching resource modality conversion and preprocessing module is used to clean and preprocess the collected digital teaching resource question stem text. The digital teaching resource multi-form representation module is used to calculate word-level vector representations of preprocessed digital teaching resource texts based on word frequency inverse text frequency algorithm and unsupervised learning semantic word embedding method, and obtain sentence-level multi-form representations of digital teaching resources through average pooling. The digital teaching resource multi-task sharing network module is used to build a multi-task sharing network, connecting the formal representation of digital teaching resources with the analysis layer of each subsequent task; The cognitive level analysis module for digital teaching resources is used to construct a deep neural network after a multi-task sharing network to analyze the cognitive level of the requirements of digital teaching resources. The digital teaching resource knowledge hierarchy analysis module is used to construct a deep neural network after a multi-task sharing network to analyze the knowledge hierarchy examined by digital teaching resources. The Digital Teaching Resources Subject Thinking Analysis Module is used to construct a deep neural network after a multi-task sharing network to analyze the subject thinking of digital teaching resources.

Images