Method for constructing classical music cross-modal high-reliability experimental dataset for all-scene teaching applications

Abstract

The application provides a classical music cross-modal high-reliability experimental data set construction method for full-scene teaching application, comprising the following steps: collecting multi-modal data such as audio, video and music score, constructing an original data set, standardizing each modal data by using a preprocessing technology, and obtaining a multi-modal data set in a unified format; constructing a knowledge graph according to a semantic feature vector, wherein a node represents a music element, and an edge represents a semantic relationship between elements; encoding the knowledge graph by using a graph convolution network to obtain a semantic representation of teaching content; integrating multi-modal teaching resources into an immersive teaching scene by using virtual reality rendering technology, presenting a content sequence in real time, and obtaining a dynamic teaching experience; updating the knowledge graph and the semantic feature vector by continuously collecting learner interaction data, optimizing the cross-modal fusion model by using an incremental learning algorithm, and obtaining an adaptive teaching data set.

Description

Technical Field

[0001] This invention relates to the field of information technology, and in particular to a method for constructing a highly reliable cross-modal experimental dataset for classical music for all-scenario teaching applications. Background Technology

[0002] Background of the problem:

[0003] Classical music, as an important part of cultural heritage, holds irreplaceable value in music education and cultural transmission. With the development of digital technology, the application of cross-modal data in classical music teaching has gradually become a research hotspot. By integrating multimodal resources such as audio, video, and sheet music, the immersive experience and knowledge transfer efficiency of teaching can be significantly improved. However, current research and applications still face many challenges, and a systematic approach to data set construction is urgently needed to support the needs of teaching across all scenarios.

[0004] Existing methods for constructing classical music teaching datasets often limit themselves to single-modal data, such as relying solely on audio or sheet music, lacking deep integration of multimodal resources. Furthermore, the dataset construction process typically lacks optimization for teaching scenarios, and the data organization methods are difficult to adapt to learners' individual needs, resulting in fragmented teaching content and a lack of coherence in learning paths. These limitations make it difficult for existing datasets to achieve high reliability and universality when supporting complex teaching scenarios.

[0005] The core challenge lies in how to organically integrate cross-modal data and dynamically adapt it to teaching scenarios. First, the heterogeneity of cross-modal data makes it difficult to accurately establish semantic relationships between data points, affecting the logic and coherence of teaching content. Second, the level of intelligence in instructional arrangement is insufficient, making it difficult to dynamically adjust the order of content presentation based on learners' cognitive levels and interests. These unresolved technical factors limit the effectiveness of datasets in all-scenario teaching, making it difficult to meet diverse and personalized teaching needs.

[0006] Therefore, designing a highly reliable dataset construction method that can integrate multimodal data and dynamically generate personalized teaching sequences has become a key issue for teaching applications across all scenarios. Summary of the Invention

[0007] This invention provides a method for constructing a highly reliable cross-modal experimental dataset for classical music for all-scenario teaching applications, mainly including:

[0008] By collecting multimodal data such as audio, video, and musical scores, a raw dataset is constructed. Preprocessing techniques are used to standardize the data of each modality, resulting in a multimodal data set with a unified format.

[0009] Features are extracted from multimodal datasets. Fourier transform is used to obtain spectral features for audio data, convolutional neural networks are used to extract visual features for video data, and symbol analysis is used to obtain note sequences for musical scores, thus obtaining feature representations for each modality.

[0010] A cross-modal attention mechanism is adopted to perform weighted fusion of feature representations of each modality, calculate the semantic similarity between features, and if the similarity is greater than the preset threshold S (S is 0.8), a semantic association is established to obtain the fused semantic feature vector.

[0011] Based on semantic feature vectors, a knowledge graph is constructed, where nodes represent musical elements and edges represent semantic relationships between elements. A graph convolutional network is used to encode the knowledge graph to obtain the semantic representation of the teaching content.

[0012] Subgraphs are extracted from the semantic representation of the teaching content. Based on the cognitive level parameter C input by the learner (C is an integer from 1 to 5), a reinforcement learning algorithm is used to adjust the weights of the subgraph nodes to obtain a personalized teaching content sequence.

[0013] By using a sequence generation model, personalized teaching content sequences are mapped into multimodal outputs, generating accompaniment for audio modalities, teaching animations for video modalities, and annotations for sheet music modalities, thus obtaining multimodal teaching resources.

[0014] By using virtual reality rendering technology, multimodal teaching resources are integrated into an immersive teaching scenario, presenting the content sequence in real time and obtaining a dynamic teaching experience;

[0015] Based on real-time feedback data from learners, the parameters of the reinforcement learning model are updated using online learning algorithms. If the feedback score is lower than the preset threshold T (T is 0.7), the content sequence is readjusted to obtain an optimized teaching sequence.

[0016] By continuously collecting learner interaction data, updating the knowledge graph and semantic feature vectors, and using incremental learning algorithms to optimize the cross-modal fusion model, an adaptive teaching dataset is obtained.

[0017] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0018] This invention discloses an intelligent music teaching method based on multimodal data. The method collects multimodal data such as audio, video, and sheet music to construct a raw dataset and performs preprocessing. A cross-modal attention mechanism is used to fuse features from various modalities, constructing a music knowledge graph. Based on learners' cognitive levels, a reinforcement learning algorithm is used to generate personalized teaching content sequences, which are then mapped to multimodal teaching resources. Immersive teaching scenarios are presented using virtual reality technology, and the teaching sequences are optimized in real time based on learner feedback. This invention achieves intelligent generation and adaptive adjustment of teaching content, improving the personalization and interactivity of music teaching, and effectively enhancing teaching effectiveness and learning experience. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a method for constructing a highly reliable cross-modal experimental dataset for classical music, oriented towards all-scenario teaching applications, according to the present invention.

[0020] Figure 2 This is a schematic diagram of a method for constructing a highly reliable cross-modal experimental dataset for classical music for all-scenario teaching applications, according to the present invention.

[0021] Figure 3 This is another schematic diagram of a method for constructing a highly reliable cross-modal experimental dataset for classical music for all-scenario teaching applications according to the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] like Figure 1-3 This embodiment of a method for constructing a highly reliable cross-modal experimental dataset for classical music for all-scenario teaching applications may specifically include:

[0024] Step S101: By collecting multimodal data such as audio, video, and musical scores, an original dataset is constructed. Preprocessing techniques are used to standardize each modal data to obtain a multimodal data set with a unified format.

[0025] Audio, video, and musical score data are acquired from the environment or musical instruments using sensors and recording devices to construct a raw dataset containing multimodal data. A format conversion tool is used to unify the sampling rate of the audio data, resulting in standardized audio data; the frame rate and resolution of the video data are adjusted to obtain standardized video data; and the musical score data is serialized to obtain standardized musical score data. If the sampling rate of the standardized audio data matches a preset threshold, it is stored in the unified format dataset; if the frame rate and resolution of the standardized video data match a preset threshold, it is stored in the unified format dataset; and if the symbol sequence of the standardized musical score data is complete, it is stored in the unified format dataset, resulting in a preliminary integrated multimodal dataset. Feature extraction algorithms are used to extract spectral features of the audio, inter-frame features of the video, and note sequence features of the musical score from the preliminary integrated multimodal dataset, resulting in a multimodal feature set. Principal component analysis is used to reduce the dimensionality of the multimodal feature set, resulting in a compressed feature dataset. If the dimensionality of the compressed feature dataset is lower than a preset threshold, data alignment techniques are used to synchronize the audio, video, and musical score features along the time axis, resulting in a time-aligned fused dataset. By performing intermodal correlation analysis on time-aligned fused datasets using clustering algorithms, a unified representation of multimodal data is obtained.

[0026] For example, when constructing the multimodal dataset, a musical performance is first recorded at a sampling rate of 44.1kHz using audio acquisition devices such as a microphone array, while simultaneously capturing the performance footage at 60fps using a 4K camera, and scanning standard MIDI format sheet music files. Audio data undergoes spectral analysis using the FFT algorithm to extract 128-dimensional MFCC features. Video data is processed using the OpenCV library for keyframe extraction and optical flow calculation, and sheet music data is parsed into MusicXML format using OMR technology. In the preprocessing stage, the audio is standardized, with amplitude normalized to the range of -1dB to 1dB. Video data is uniformly adjusted to 1920x1080 resolution and converted to H.264 encoding. Sheet music data is transformed using XSLT to ensure all symbols conform to the MusicXML 3.0 standard. Feature fusion employs a cross-modal attention mechanism, aligning the Mel spectrogram of the audio with the I 3D features of the video in the temporal dimension. A Transformer architecture is used to calculate intermodal correlation weights, with 8 attention heads and 512 hidden layer dimensions. During data enhancement, the audio is subjected to a ±10% speed change and a gain perturbation of -6dB to 6dB, while the video data is subjected to random horizontal flipping and...

[0027] The musical score data was expanded by randomly transposing ±2 semitones with a rotation of ±15 degrees. The final dataset contained 1000 hours of synchronous multimodal data. Five-fold cross-validation was used to evaluate the model performance, and the bidirectional LSTM network achieved a classification accuracy of 87.3% on the fused features.

[0028] Step S102: Extract features from the multimodal dataset. For audio data, use Fourier transform to obtain spectral features. For video data, use convolutional neural network to extract visual features. For musical score data, use symbol analysis to obtain note sequences, thus obtaining feature representations for each modality.

[0029] Audio, video, and sheet music data are acquired from a multimodal dataset and stored as first audio data, first video data, and first sheet music data, respectively, to obtain an initial dataset. Fourier transform is applied to the first audio data to calculate the frequency components of each time window, generating the first spectral feature. A convolutional neural network is used to extract spatial features from the frame sequence of the first video data, generating the first visual feature. Symbol analysis is used to identify note and rhythm information from the first sheet music data, generating the first note sequence. If the dimensions of the first spectral feature, first visual feature, and first note sequence are inconsistent, linear interpolation is used to downsample the high-dimensional features, resulting in second spectral features, second visual features, and a second note sequence. Based on the second spectral feature, second visual feature, and second note sequence, principal component analysis is used to extract the common patterns of each modality feature, obtaining a fused feature representation. Using the fused feature representation, a clustering algorithm is used to group the multimodal data, obtaining the classification results.

[0030] For example, in audio data processing, the audio signal is first sampled at a sampling rate of 44,100 Hz, and then the sampled signal is subjected to Fourier transform to convert the time-domain signal into a frequency-domain signal to obtain the spectral characteristics.

[0031] For example, a 2-second audio signal can be decomposed into 1024 frequency components using the Fast Fourier Transform (FFT) algorithm, each corresponding to a specific frequency range, thus obtaining the audio's spectral characteristics. In video data processing, Convolutional Neural Networks (CNNs) are used to extract visual features.

[0032] For example, for a video frame with a resolution of 1920x1080, it is first resized to 224x224, then input into a pre-trained ResNet-50 model. Through convolutional and pooling layers, a 2048-dimensional feature vector is finally obtained, representing the visual features of the frame. In musical score data processing, a symbolic resolution method is used to obtain the note sequence.

[0033] For example, for a musical score containing 16 measures, it is first converted to MIDI format. Then, by parsing the MIDI file, the pitch, duration, and start time of each note are extracted, ultimately resulting in a note sequence representing the symbolic features of the score. Using this method, modal features can be extracted from audio, video, and musical score data respectively, providing a foundation for subsequent multimodal fusion analysis.

[0034] Step S103: A cross-modal attention mechanism is used to perform weighted fusion of the feature representations of each modality, calculate the semantic similarity between features, and if the similarity is greater than the preset threshold S (S is 0.8), then a semantic association is established to obtain the fused semantic feature vector.

[0035] Initial feature representations are obtained from audio, video, and musical notation modalities and stored as first audio features, first video features, and first musical notation features, respectively, resulting in an initial feature set. A cross-modal attention mechanism is employed to weight these features, calculating the attention weights between each modality to obtain second audio features, second video features, and second musical notation features. For each of these features, a semantic similarity matrix is ​​calculated. If any element in the matrix is ​​greater than a preset threshold S, a semantic association is determined, resulting in a semantic association set. Based on this set, a linear transformation is used to fuse the second audio features, second video features, and second musical notation features, generating a first fused feature vector. Principal component analysis (PCA) is used to reduce the dimensionality of the first fused feature vector, extracting the main semantic components to obtain a second fused feature vector. If the dimension of the second fused feature vector is greater than a preset dimension D, linear interpolation is used to downsample it, resulting in a third fused feature vector. Based on the third fusion feature vector, cosine similarity is used to calculate its matching degree with the preset semantic template, and the semantic category with the highest matching degree is determined to obtain the final semantic classification result.

[0036] For example, in the cross-modal attention mechanism, feature representations for both text and image modalities are first extracted. Text features are generated into 768-dimensional vectors using the BERT model, and image features are generated into 2048-dimensional vectors using the ResNet-50 model. Next, the semantic similarity between text and image features is calculated using the attention mechanism, specifically employing the cosine similarity algorithm, with the formula sim(T,I)=(T·I) /

[0037] The similarity vector is defined as (||T||*||I||), where T is the text feature vector and I is the image feature vector. Assuming a calculated similarity of 0.85, which is greater than the preset threshold of 0.8, a semantic association is established. Then, a weighted fusion method is used to fuse the text and image features, with the weights dynamically adjusted based on the similarity. The fused semantic feature vector is F = 0.6*T + 0.4*I. Finally, the fused feature vector is input into a classifier for classification. The classifier uses the Softmax function and outputs the probability distribution for each category, thus completing the cross-modal semantic feature fusion and classification task.

[0038] Step S104: Construct a knowledge graph based on semantic feature vectors. Nodes represent music elements, and edges represent semantic relationships between elements. Use a graph convolutional network to encode the knowledge graph to obtain the semantic representation of the teaching content.

[0039] The process involves obtaining semantic feature vectors of musical elements and generating initial vector representations using a pre-defined semantic analysis model, resulting in a set of semantic features for each musical element. Based on this set, a knowledge graph is constructed, with nodes composed of musical elements and edges represented by semantic association weights between elements, generating an initial knowledge graph structure. A graph convolutional network is used to encode the initial knowledge graph, iteratively updating node features and edge weights to obtain the encoded graph feature representation. If the encoded graph feature representation meets a pre-defined convergence condition, the semantic representation of each node in the graph is extracted to generate a preliminary semantic representation of the teaching content; otherwise, the graph convolutional network parameters are adjusted, and the graph is re-encoded. A pre-defined semantic mapping model is used to optimize the preliminary semantic representation, generating a final semantic representation that matches the teaching content. Based on the final semantic representation, a clustering algorithm is used to classify the teaching content, resulting in a set of classified teaching content. For the classified teaching content set, a structured teaching content output is generated, determining the final teaching content representation.

[0040] For example, when constructing a knowledge graph of music elements, the semantic feature vectors of the music elements are first extracted. For instance, "melody" is represented as [0.85, 0.12, 0.34], and "rhythm" is represented as [0.45, 0.78, 0.23]. These vectors are calculated from the music text data using the TF-IDF algorithm. Next, a knowledge graph is constructed based on these feature vectors. Nodes represent music elements, and edges represent semantic relationships between elements. For example, the edge weight between "melody" and "rhythm" is 0.67, indicating a strong semantic association between them. To encode the knowledge graph, a Graph Convolutional Network (GCN) is used. The input layer receives the feature vectors of the nodes, the hidden layers use the ReLU activation function, and the output layer generates the semantic representation of each node.

[0041] For example, after two layers of GCN processing, the semantic representation of the "melody" node is updated to [0.92, 0.15, 0.38], while the semantic representation of the "rhythm" node is updated to [0.48, 0.81, 0.25]. Finally, these semantic representations are used for semantic analysis of teaching content. For example, by calculating cosine similarity, the similarity between "melody" and teaching content A is determined to be 0.89, and the similarity between "melody" and teaching content B is determined to be 0.76, thus providing a basis for recommending teaching content.

[0042] Step S105: Extract subgraphs from the semantic representation of the teaching content. Based on the cognitive level parameter C input by the learner (C is an integer from 1 to 5), use a reinforcement learning algorithm to adjust the weights of the subgraph nodes to obtain a personalized teaching content sequence.

[0043] Semantic representations are obtained from the teaching content database, and subgraphs are extracted using a pre-defined semantic analysis model to obtain an initial subgraph structure. Based on the learner's input cognitive level parameter C, a reinforcement learning algorithm is used to calculate the initial weights of each node in the subgraph, determining the weight distribution. If the weight distribution deviates from the pre-defined personalized target by more than a threshold, the node weights are iteratively adjusted using a reinforcement learning algorithm to obtain an optimized subgraph. Using the optimized subgraph, a sequence generation model is used to generate candidate teaching content sequences, obtaining a sequence set. Based on the learner's input cognitive level parameter C, teaching content sequences matching parameter C are selected from the sequence set to determine the optimal sequence. If the matching degree of the optimal sequence is lower than a pre-defined threshold, the sequence generation model is returned to regenerate the sequence, resulting in the final personalized teaching content sequence. Using the final personalized teaching content sequence, structured teaching content data is output, and the delivery format is determined. These seven steps form a logical chain through semantic representation, subgraph extraction, weight optimization, sequence generation, and selection. The output of the previous step serves as the input for the next step, ensuring the achievement of business objectives.

[0044] For example, when extracting a subgraph from the semantic representation of teaching content, it is first necessary to construct a knowledge graph, with knowledge points in the teaching content as nodes and the relationships between knowledge points as edges.

[0045] For example, in an article about "machine learning," the knowledge points "supervised learning" and "unsupervised learning" can be considered nodes, with the relationship between them, "classification method," as an edge. Using natural language processing techniques, keywords and semantic relationships are extracted from the article to construct a knowledge graph. Next, based on the learner's cognitive level parameter C (assuming C is 3), a reinforcement learning algorithm is used to adjust the weights of the subgraph nodes. The reinforcement learning algorithm can use Q-learning, optimizing node weights by defining the state (current subgraph node), action (selecting the next node), and reward (learner's level of understanding).

[0046] For example, when a learner has a high level of understanding of "supervised learning," the algorithm increases the weight of the "classification method" node connected to that node, thus prioritizing the recommendation of related content in subsequent learning. Through multiple iterations, the algorithm gradually adjusts the weights of subgraph nodes, generating a personalized sequence of teaching content.

[0047] For example, the final generated sequence might be "supervised learning" → "classification method" → "logistic regression," ensuring that learners can gradually master knowledge according to their own cognitive level. The entire process is automatically processed by information technology, requiring no human intervention, thus achieving intelligent and personalized recommendations of teaching content.

[0048] Step S106: Through the sequence generation model, the personalized teaching content sequence is mapped into multimodal output, accompaniment is generated for the audio modality, teaching animation is generated for the video modality, and annotations are generated for the sheet music modality, thus obtaining multimodal teaching resources.

[0049] Personalized teaching content sequences are obtained through a sequence generation model. An encoder is used to extract features from the sequences, resulting in content feature vectors. A variational autoencoder is then used to perform modal decomposition on the content feature vectors, generating audio modal features, video modal features, and sheet music modal features, resulting in a multimodal feature set. For audio modal features, if the feature vector matches a preset accompaniment style template, a recurrent neural network is used to generate an accompaniment sequence, resulting in audio accompaniment data. For video modal features, a generative adversarial network is used to map the feature vectors to an animation frame sequence, synthesizing teaching animations, resulting in video animation data. For sheet music modal features, if the feature vector contains a note sequence, a sequence labeling algorithm is used to generate sheet music annotations, resulting in sheet music annotation data. The audio accompaniment data, video animation data, and sheet music annotation data are acquired and time-synchronized using a multimodal fusion algorithm, resulting in multimodal teaching resources. A content sequence verification module performs consistency checks on the multimodal teaching resources. If the check results meet a preset consistency threshold, the final multimodal teaching resources are output.

[0050] For example, in the sequence generation model, semantic features of personalized teaching text are first extracted using BERT. For instance, when the input is "explanation of the periodicity of trigonometric functions," a 768-dimensional vector representation is output, which is then encoded into 128-dimensional temporal features using LSTM. For the audio modality, a DiffWave model is used to generate accompaniment with a frame length of 50ms. Mel-spectrum constraints are used to ensure synchronization with the explanation rhythm. For example, when the keyword "y = sin(x)" is detected, a 440Hz fundamental frequency string timbre is generated. For the video modality, StyleGAN-V is used to generate 1280×720 resolution animation. Content consistency is controlled based on CLIP text alignment loss. When explaining phase changes, a dynamic sine wave visualization is automatically inserted. The inter-frame optical flow loss is kept below 0.03 to ensure smooth transitions. Musical notation is generated using MusicTransformer, aligning explanation nodes with a minimum unit of 0.25 beats. When analyzing the concept of "amplitude change," it is automatically inserted. <forte>Velocity marking is performed using the Hidden Markov Model (HMM) algorithm to ensure note timing errors are less than 5%. In the multimodal fusion stage, a cross-modal attention mechanism is employed, controlling audio-video synchronization errors within ±80ms. A three-modal contrastive loss function is used to achieve a cosine similarity of at least 0.85 in the feature space. The final output is encapsulated in an MP4 container, with a video bitrate of 5000kbps, AAC-LC audio encoding, and the score embedded in the metadata area as an SVG vector format.

[0051] Step S107: Using virtual reality rendering technology, multimodal teaching resources are integrated into an immersive teaching scene, and the content sequence is presented in real time to obtain a dynamic teaching experience.

[0052] Multimodal teaching resource data is acquired using virtual reality technology, and integrated processing is used to generate initial immersive scene data, resulting in a basic scene model. If the resolution of the initial immersive scene data is lower than a preset threshold, the basic scene model is optimized using rendering technology to obtain a high-resolution scene model. Based on the high-resolution scene model, a real-time rendering algorithm is used to generate a dynamic content sequence, obtaining a content sequence dataset. If the interaction response time of the content sequence dataset exceeds a preset threshold, an interactive experience optimization algorithm is used to adjust the dynamic content sequence, resulting in an optimized content sequence. User interaction data is obtained from the optimized content sequence, and scene generation technology is used to update the immersive scene, obtaining a dynamic interactive scene. Experience optimization technology is used to analyze the dynamic interactive scene in real time to determine whether the user experience data meets a preset threshold, resulting in the final interactive teaching experience.

[0053] For example, in virtual reality rendering technology, the geometric model of the teaching scene is first constructed using 3D modeling tools such as Blender or Unity. The model accuracy is controlled within 1 mm to ensure realism, and PBR material mapping is used to enhance surface details. Next, ray tracing algorithms (such as path tracing) are used for real-time rendering, with a sampling rate of 64 per pixel to reduce noise. DLSS technology is used to oversample the rendering resolution from 1080p to 4K to improve performance. In the multimodal resource integration stage, text, audio, and video resources are encoded into a unified format, such as using H.265 to compress video streams at a bitrate of 8 Mbps, and AAC encoding for audio at a sampling rate of 48 kHz. Spatiotemporal alignment algorithms (such as Dynamic Time Warping (DTW)) ensure multimodal data synchronization, with errors controlled within ±50 milliseconds. When presenting the content sequence in real time, an LSTM-based prediction model is used to preload the next 3 seconds of teaching content, with a buffer size of 500 MB to avoid stuttering. The dynamic teaching experience relies on eye-tracking technology, which collects user gaze data at a frequency of 120Hz. Combined with Foveated Rendering technology, the rendering quality is dynamically adjusted, maintaining a 4K resolution in the gaze area while reducing the resolution of the surrounding area to 720p, thus reducing GPU load by 40%. Finally, user behavior data analysis (such as the K-means clustering algorithm, k=5) is used to continuously optimize the scene layout, ensuring that more than 90% of users can locate the core teaching elements within 2 seconds.

[0054] Step S108: Based on the learner's real-time feedback data, the parameters of the reinforcement learning model are updated using an online learning algorithm. If the feedback score is lower than the preset threshold T (T is 0.7), the content sequence is readjusted to obtain an optimized teaching sequence.

[0055] The system acquires real-time feedback data from learners, cleanses the data, and extracts features to generate real-time feedback scores. If the real-time feedback score is lower than a preset threshold, the reinforcement learning model is updated online. The online learning algorithm is used to adjust the model parameters, resulting in updated model parameters. Based on the updated model parameters, the reinforcement learning model predicts scores for content sequences, generating a set of candidate content sequences. For each candidate sequence, a sequence scoring function is used to calculate the expected score, resulting in the optimized teaching sequence with the highest score. By optimizing the teaching sequence, the presentation order of the learning content is adjusted, generating an adjusted content sequence. The real-time feedback score of the adjusted content sequence is acquired and compared with a preset threshold to determine whether further sequence adjustments are needed, resulting in the final teaching sequence. If the feedback score of the final teaching sequence is still lower than the preset threshold, new features are extracted through feedback data processing, and the online learning algorithm is updated to obtain new model parameters.

[0056] For example, in a real-time learning scenario, the system updates the weight parameters of the reinforcement learning model using an online gradient descent algorithm. For instance, using an Adam optimizer with a learning rate η = 0.01, upon receiving each learner interaction (e.g., a correct answer rate of 0.65), the system immediately calculates the TD error δ = 0.12 between the action value Q(s,a) output by the current policy network and the target value. When the average of five consecutive feedback scores (0.68) is below the threshold T = 0.7, a content sequence reorganization mechanism is triggered. First, a K-means clustering algorithm (k = 3) is used to group the knowledge point mastery feature vectors [0.4, 0.7, 0.3] from the historical interaction data. Then, combined with item response theory, the difficulty parameter b = 1.2 and the discrimination index a = 0 are calculated for each knowledge point.

[0057] 8. Then, a state transition matrix P is constructed based on the Markov decision process, where the transition probability from state s_t to s_{t+1} is calculated using the Bayesian update formula P(s'|s,a)=N(μ,σ2), and the mean μ is dynamically adjusted based on the most recent 20 feedback data. The reorganized teaching sequence was tested to verify its effectiveness. The NDCG@3 index of the experimental group using the new sequence increased to 0.82, significantly higher than 0 in the control group with the original sequence.

[0058] 71. During the process, a sliding window mechanism is used to maintain a priority experience replay buffer of length 100. Samples with a TD error δ > 0.1 are sampled first for model training to ensure a 35% improvement in the utilization rate of key data. For special scenarios, such as isolated nodes with a cosine similarity lower than 0.3 in the knowledge point association matrix, a content importance reordering module based on the PageRank algorithm will be activated to advance the teaching order of core knowledge points by 2-3 positions.

[0059] Step S109 involves continuously collecting learner interaction data, updating the knowledge graph and semantic feature vectors, and using an incremental learning algorithm to optimize the cross-modal fusion model to obtain an adaptive teaching dataset.

[0060] An initial interaction dataset is generated by collecting learner interaction data. Structured information is extracted from this dataset to update the knowledge graph, resulting in an updated knowledge graph. Semantic features are generated based on the updated knowledge graph and converted into feature vectors, resulting in semantic feature vectors. If the dimension of the semantic feature vectors meets a preset threshold, a cross-modal fusion algorithm is used to process the semantic feature vectors, generating a fusion model; otherwise, the feature vector dimensions are adjusted and reprocessed to obtain the fusion model. The fusion model is optimized using an incremental learning algorithm, updating the model parameters to obtain an optimized fusion model. An adaptive teaching dataset is generated based on the optimized fusion model, resulting in an adaptive teaching dataset. Feedback information is extracted from the adaptive teaching dataset to update the learner interaction data, resulting in an updated interaction dataset.

[0061] For example, during the continuous collection of learner interaction data, tracking techniques are used to record learners' clickstream, dwell time (e.g., triggering knowledge point marking when video viewing time exceeds 90 seconds), and answer accuracy (e.g., marking a weak point when the accuracy rate of math questions is below 60%) on the teaching platform. This data is collected in real-time using the Flume framework at a throughput of 200 records per second and stored in HDFS. In the knowledge graph update phase, the Neo4j graph database is used. When it is detected that more than 30% of learners associate queries for "quadratic function" with "extreme value problem," edge relationships between these two types of nodes are automatically added with a weight of 0.85. Simultaneously, the TransR algorithm is used to expand the node vector dimension from 128 to 256 dimensions to improve representation capabilities. Semantic feature vector optimization employs incremental training of the BERT model. For every 5000 new learning note texts, the model is fine-tuned for 2 epochs with a learning rate of 0.0001, increasing the cosine similarity of "Newton's Laws" in the vector space from 0.72 to 0.89. The cross-modal fusion model adopts the MMoE architecture. When the ratio of video viewing data to exercise data reaches 1:3, a dynamic weight adjustment algorithm reduces the weight of the visual modality gateway from 0.6 to 0.45, while the weight of the text modality gateway is correspondingly increased by 0.15. Incremental learning uses the EWC algorithm, calculating the diagonal value of the Fisher information matrix when updating model parameters (e.g., locking the importance score of fully connected layer parameters when it is greater than 0.7) to ensure that the historical accuracy drop does not exceed 2%. The final generated adaptive teaching dataset is divided into knowledge unit difficulty levels using K-means clustering (k=8, silhouette coefficient 0.65), and learning paths are dynamically recommended based on Markov chain prediction (state transition probability matrix dimension 20×20), achieving a recommendation hit rate of 83% for highly relevant knowledge points. The entire system performs closed-loop updates on a daily basis, automatically triggering the knowledge graph reconstruction process when the seven-day retention rate of a knowledge point is lower than 40%.

[0062] The above are only some preferred embodiments of the present invention, but the present invention is not limited thereto, and many improvements and modifications can be made. Any improvements and modifications made based on the basic principles of the present invention should be considered to fall within the protection scope of the present invention.< / forte>

Claims

1. A method for constructing a highly reliable cross-modal experimental dataset for classical music for all-scenario teaching applications, characterized in that, The method includes: By collecting multimodal data of audio, video and musical scores, a raw dataset is constructed. Preprocessing techniques are used to standardize the data of each modality, resulting in a multimodal data set with a unified format. Features are extracted from multimodal datasets. Fourier transform is used to obtain spectral features for audio data, convolutional neural networks are used to extract visual features for video data, and symbol analysis is used to obtain note sequences for musical scores, thus obtaining feature representations for each modality. A cross-modal attention mechanism is adopted to weight the feature representations of each modality, and then the semantic similarity between features is calculated. If the similarity is greater than a preset threshold S, a semantic association is established. Then, a linear transformation is used to fuse the features to obtain the fused semantic feature vector; where S is 0.

8. Based on semantic feature vectors, a knowledge graph is constructed, where nodes represent musical elements and edges represent semantic relationships between elements. A graph convolutional network is used to encode the knowledge graph to obtain the semantic representation of the teaching content. Subgraphs are extracted from the semantic representation of the teaching content. Based on the cognitive level parameter C input by the learner, a reinforcement learning algorithm is used to adjust the weights of the subgraph nodes to obtain a personalized teaching content sequence; where C is an integer from 1 to 5. By using a sequence generation model, personalized teaching content sequences are mapped into multimodal outputs, generating accompaniment for audio modalities, teaching animations for video modalities, and annotations for sheet music modalities, thus obtaining multimodal teaching resources. By using virtual reality rendering technology, multimodal teaching resources are integrated into an immersive teaching scenario, presenting the content sequence in real time and obtaining a dynamic teaching experience; Based on real-time feedback data from learners, the parameters of the reinforcement learning model are updated using an online learning algorithm. If the feedback score is lower than a preset threshold T, the content sequence is readjusted to obtain an optimized teaching sequence; where T is 0.

7. By continuously collecting learner interaction data, updating the knowledge graph and semantic feature vectors, and using incremental learning algorithms to optimize the cross-modal fusion model, an adaptive teaching dataset is obtained. Specifically, the method employs a cross-modal attention mechanism to weight the feature representations of each modality, then calculates the semantic similarity between features. If the similarity is greater than a preset threshold S, a semantic association is established. Subsequently, a linear transformation is used to fuse the features to obtain the fused semantic feature vector. Initial feature representations are obtained from the audio modality, video modality, and musical score modality, and stored as the first audio feature, the first video feature, and the first musical score feature, respectively, to obtain the initial feature set; A cross-modal attention mechanism is adopted to weight the first audio feature, the first video feature, and the first musical score feature, and calculate the attention weight between each modal feature to obtain the second audio feature, the second video feature, and the second musical score feature. For the second audio feature, the second video feature, and the second musical score feature, the semantic similarity matrix between the features is calculated. If an element in the matrix is ​​greater than a preset threshold S, it is determined that the corresponding feature pair has a semantic association, and a semantic association set is obtained. Based on the semantic association set, a linear transformation is used to fuse the second audio feature, the second video feature, and the second musical score feature to generate the first fused feature vector.

2. The method according to claim 1, characterized in that, The process involves collecting multimodal data such as audio, video, and sheet music to construct an original dataset. Preprocessing techniques are then used to standardize the data across different modalities, resulting in a unified format of multimodal data, including: By acquiring audio data, video data, and musical score data from the environment or musical instruments using sensors and recording devices, a raw dataset containing multimodal data is constructed. The audio data is standardized by using a format conversion tool to unify the sampling rate; The frame rate and resolution of the video data are adjusted to obtain standardized video data; The musical score data is serialized using symbols to obtain standardized musical score data; If the sampling rate of the standardized audio data is consistent with the preset threshold, it will be stored in a unified format dataset. If the frame rate and resolution of the standardized video data match the preset threshold, then it is stored in a uniform format dataset; If the symbol sequence of the standardized musical score data is complete, it is stored in a unified format dataset to obtain a preliminary integrated multimodal dataset; The multimodal feature set is obtained by extracting spectral features of audio, inter-frame features of video, and note sequence features of musical score from the initially integrated multimodal dataset using feature extraction algorithms. Principal component analysis (PCA) algorithm is used to reduce the dimensionality of the multimodal feature set, resulting in a compressed feature dataset. If the dimension of the compressed feature dataset is lower than a preset threshold, the audio features, video features and musical score features are synchronized on the time axis using data alignment technology to obtain a time-aligned fused dataset. By performing intermodal correlation analysis on time-aligned fused datasets using clustering algorithms, a unified representation of multimodal data is obtained.

3. The method according to claim 1, characterized in that, The process of extracting features from a multimodal dataset involves using Fourier transform to obtain spectral features for audio data, convolutional neural networks to extract visual features for video data, and symbol analysis to obtain note sequences for musical scores, resulting in feature representations for each modality, including: Audio data, video data, and sheet music data are obtained from the multimodal dataset and stored as the first audio data, the first video data, and the first sheet music data, respectively, to obtain the initial data set; Fourier transform is applied to the first audio data to calculate the frequency components of each time window and generate the first spectral features; A convolutional neural network is used to extract spatial features from the frame sequence of the first video data to generate the first visual features. Symbol analysis is used to analyze the first musical score data, identify note and rhythm information, and generate the first note sequence. If the dimensions of the first spectral feature, the first visual feature, and the first note sequence are inconsistent, then the high-dimensional feature is downsampled by linear interpolation to obtain the second spectral feature, the second visual feature, and the second note sequence. Based on the second spectral features, the second visual features, and the second note sequence, principal component analysis algorithm is used to extract the common patterns of each modality feature to obtain the fused feature representation; By fusing feature representations, a clustering algorithm is used to group multimodal data to obtain classification results.

4. The method according to claim 1, characterized in that, The method employs a cross-modal attention mechanism to weight the feature representations of each modality, then calculates the semantic similarity between features. If the similarity is greater than a preset threshold S, a semantic association is established. The method then uses linear transformation to fuse features to obtain the fused semantic feature vector, which further includes: The first fusion feature vector is reduced in dimensionality by principal component analysis algorithm to extract the main semantic components and obtain the second fusion feature vector. If the dimension of the second fused feature vector is greater than the preset dimension D, then it is downsampled by linear interpolation to obtain the third fused feature vector; Based on the third fusion feature vector, cosine similarity is used to calculate its matching degree with the preset semantic template, and the semantic category with the highest matching degree is determined to obtain the final semantic classification result.

5. The method according to claim 1, characterized in that, The process involves constructing a knowledge graph based on semantic feature vectors, where nodes represent musical elements and edges represent semantic relationships between elements. A graph convolutional network is then used to encode the knowledge graph, resulting in a semantic representation of the teaching content. This includes: Obtain the semantic feature vectors of music elements, generate initial vector representations through a preset semantic analysis model, and obtain the semantic feature set of music elements; Based on the set of semantic features of music elements, a knowledge graph is constructed, where nodes are composed of music elements and edges are represented by the semantic association weights between elements, thus generating an initial knowledge graph structure. A graph convolutional network is used to encode the initial knowledge graph, and the node features and edge weights are iteratively updated to obtain the encoded graph feature representation. If the encoded graph feature representation satisfies the preset convergence condition, then the semantic representation of each node in the graph is extracted to generate the preliminary semantic representation of the teaching content. If the conditions are not met, adjust the graph convolutional network parameters and re-encode. The preliminary semantic representation is optimized using a pre-defined semantic mapping model to generate a final semantic representation that matches the teaching content. Based on the final semantic representation, a clustering algorithm is used to classify the teaching content, resulting in a set of classified teaching content. For the categorized set of teaching content, a structured teaching content output is generated to determine the final teaching content representation.

6. The method according to claim 1, characterized in that, The process involves extracting subgraphs from the semantic representation of the teaching content, adjusting the weights of subgraph nodes using a reinforcement learning algorithm based on the learner's input cognitive level parameter C, and obtaining a personalized teaching content sequence, including: Semantic representations are obtained from the teaching content database, and subgraphs are extracted using a pre-set semantic analysis model to obtain the initial subgraph structure; Based on the cognitive level parameter C input by the learner, the initial weights of each node in the subgraph are calculated using a reinforcement learning algorithm to determine the weight distribution; If the weight distribution deviates from the preset personalized target by more than a threshold, the node weights are iteratively adjusted through a reinforcement learning algorithm to obtain an optimized subgraph. Using the optimized subgraph, a sequence generation model is employed to generate candidate teaching content sequences, thereby obtaining a sequence set. Based on the learner's input cognitive level parameter C, the teaching content sequence that matches parameter C is selected from the sequence set, and the optimal sequence is determined. If the matching degree of the optimal sequence is lower than the preset threshold, the sequence generation model is returned to regenerate the sequence to obtain the final personalized teaching content sequence. By finalizing the personalized teaching content sequence, structured teaching content data is output, and the delivery format is determined. The above seven steps form a logical chain through semantic representation, subgraph extraction, weight optimization, and sequence generation and filtering. The output of the previous step serves as the input of the next step, ensuring the achievement of business objectives.

7. The method according to claim 1, characterized in that, The process involves using a sequence generation model to map personalized teaching content sequences into multimodal outputs. This includes generating accompaniment for audio modalities, teaching animations for video modalities, and annotations for sheet music modalities, resulting in multimodal teaching resources, including: Personalized teaching content sequences are obtained through a sequence generation model, and features are extracted from the sequences using an encoder to obtain content feature vectors. A variational autoencoder is used to perform modal decomposition on the content feature vector to generate audio modal features, video modal features, and musical score modal features, resulting in a multimodal feature set; For audio modal features, if the feature vector matches the preset accompaniment style template, then a recurrent neural network is used to generate an accompaniment sequence to obtain audio accompaniment data. For video modal features, a generative adversarial network is used to map feature vectors into animation frame sequences, synthesize teaching animations, and obtain video animation data; For the music score modal features, if the feature vector contains a note sequence, then the music score annotation is generated by the sequence annotation algorithm to obtain the music score annotation data; Audio accompaniment data, video animation data, and sheet music annotation data are acquired, and a multimodal fusion algorithm is used to synchronize the timeline to obtain multimodal teaching resources; The content sequence verification module performs consistency checks on the multimodal teaching resources. If the check results meet the preset consistency threshold, the final multimodal teaching resources are output.

8. The method according to claim 1, characterized in that, The aforementioned use of virtual reality rendering technology integrates multimodal teaching resources into an immersive teaching scene, presenting content sequences in real time to achieve a dynamic teaching experience, including: Multimodal teaching resource data is acquired through virtual reality technology, and initial immersive scene data is generated through integrated processing to obtain a basic scene model; If the resolution of the initial immersive scene data is lower than a preset threshold, the scene base model is optimized using rendering technology to obtain a high-resolution scene model. Based on the high-resolution scene model, a real-time rendering algorithm is used to generate dynamic content sequences, thereby obtaining a content sequence dataset. If the interactive response time of the content sequence dataset exceeds a preset threshold, the dynamic content sequence is adjusted through an interactive experience optimization algorithm to obtain an optimized content sequence. User interaction data is obtained from optimized content sequences, and immersive scenes are updated using scene generation technology to obtain dynamic interactive scenes; By using experience optimization technology to analyze dynamic interactive scenarios in real time, we can determine whether the user experience data meets the preset threshold and obtain the final interactive teaching experience.

9. The method according to claim 1, characterized in that, The method involves updating the reinforcement learning model parameters using online learning algorithms based on real-time feedback data from learners. If the feedback score is lower than a preset threshold T, the content sequence is readjusted to obtain an optimized teaching sequence, including: Obtain learners' real-time feedback data, and generate real-time feedback scores through data cleaning and feature extraction; If the real-time feedback score is lower than the preset threshold, the online update of the reinforcement learning model is triggered, and the model parameters are adjusted using an online learning algorithm to obtain the updated model parameters. Based on the updated model parameters, the scores of the content sequences are predicted using a reinforcement learning model to generate a set of candidate content sequences. For the set of candidate content sequences, the expected score of each sequence is calculated using a sequence scoring function, and the optimized teaching sequence with the highest score is obtained. By optimizing the teaching sequence and adjusting the presentation order of learning content, an adjusted content sequence is generated; Obtain real-time feedback scores for the adjusted content sequence, compare them with preset thresholds to determine whether to continue adjusting the sequence, and obtain the final teaching sequence; If the feedback score of the final teaching sequence is still lower than the preset threshold, new features are extracted through feedback data processing, the online learning algorithm is updated, and new model parameters are obtained.

10. The method according to claim 1, characterized in that, The process involves continuously collecting learner interaction data, updating the knowledge graph and semantic feature vectors, and using an incremental learning algorithm to optimize the cross-modal fusion model, resulting in an adaptive teaching dataset, including: An initial interaction dataset is generated by collecting learner interaction data; Structured information is extracted from the initial interactive dataset to update the knowledge graph, resulting in the updated knowledge graph. Based on the updated knowledge graph, semantic features are generated, transformed into feature vectors, and thus semantic feature vectors are obtained. If the dimension of the semantic feature vector meets the preset threshold, the semantic feature vector is processed by a cross-modal fusion algorithm to generate a fusion model. If the conditions are not met, the feature vector dimensions are adjusted and the model is reprocessed to obtain the fusion model. The fusion model is optimized by using an incremental learning algorithm, and the model parameters are updated to obtain the optimized fusion model. Based on the optimized fusion model, an adaptive teaching dataset is generated, resulting in the adaptive teaching dataset. Feedback information is extracted from the adaptive teaching dataset to update learner interaction data, resulting in an updated interaction dataset.

Images