Virtual reality music teaching system based on scene teaching method

Abstract

The invention relates to the technical field of virtual reality teaching. The virtual reality music teaching system based on the scene teaching method comprises the steps that multi-modal data collection processing is conducted on a non-western music symbol system, symbol feature deconstruction processing is conducted on collected data, and a symbol metadata set is generated; performing cultural feature matching processing to generate a compound symbol description framework; inputting the composite symbol description framework into a virtual reality engine, performing cross-culture mapping processing through a melody conversion layer and a culture element adaptation algorithm, and generating a mapping parameter set of the target teaching scene; calling a three-dimensional scene rendering module to carry out immersive environment construction processing, and generating a culture theme teaching scene; according to the method, playing behavior data in a culture teaching scene is acquired, symbol-behavior matching analysis processing is performed on the playing behavior data, and a behavior matching analysis result and a dynamically corrected mapping parameter set are generated, so that the problems of insufficient symbol digitalization compatibility and culture context splitting are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual reality teaching, and particularly to a virtual reality music teaching system based on the scenario teaching method. Background Art

[0002] With the rapid development of virtual reality technology, its application in the field of education has been continuously deepened, especially showing the potential to break through the traditional classroom in music teaching. By constructing an immersive cultural scene, the virtual reality system can achieve multi-sensory linked music cognition and skill training, becoming an important direction for the innovation of modern art education.

[0003] However, the related technologies have the following problems: due to over-reliance on the Western music coding system, the parsing error rate of microtones and special symbols of non-Western music is high, resulting in the distortion of the original characteristics of ethnic music; there is a cultural context gap in the existing virtual teaching scenarios, and the relevance between music symbols and cultural elements such as historical buildings and clothing patterns cannot be restored, resulting in cognitive gaps for learners. Summary of the Invention

[0004] Based on this, it is necessary to provide a virtual reality music teaching system based on the scenario teaching method for the above technical problems, so as to solve the problems of insufficient symbol digitization compatibility and cultural context gap, thereby improving the parsing accuracy of microtones and special symbols of non-Western music, enhancing the relevance between music symbols and cultural elements in the virtual teaching scenario, and reducing the distortion of the original characteristics of ethnic music and the cognitive gap of learners.

[0005] The present application provides a virtual reality music teaching system based on the scenario teaching method, and the system includes:

[0006] A multi-modal data acquisition and feature deconstruction module, which is used to perform multi-modal data acquisition processing on the non-Western music symbol system, and perform symbol feature deconstruction processing on the acquired data to generate a symbol metadata set, and the symbol metadata set includes a pitch dimension, a duration dimension, and a performance technique dimension;

[0007] A cultural context association module, which is used to perform cultural feature matching processing through a context association model based on the symbol metadata set to generate a composite symbol description framework;

[0008] A cross-cultural dynamic mapping module, which is used to input the composite symbol description framework into a virtual reality engine, and perform cross-cultural mapping processing through a pitch conversion layer and a cultural element adaptation algorithm to generate a mapping parameter set for the target teaching scenario;

[0009] An immersive scene construction module, which is used to call a three-dimensional scene rendering module to perform immersive environment construction processing based on the mapping parameter set to generate a cultural theme teaching scene, and the cultural teaching scene includes a three-dimensional visualization model of music symbols;

[0010] A behavior feedback and parameter correction module is used to obtain performance behavior data in a cultural teaching scenario through a motion capture device, perform symbol-behavior matching analysis and processing on the performance behavior data, generate a behavior matching analysis result, trigger a virtual tutor prompt system based on the behavior matching analysis result, and generate a set of dynamically corrected mapping parameters.

[0011] Furthermore, obtain performance behavior data in a cultural teaching scenario through a motion capture device, perform symbol-behavior matching analysis and processing on the performance behavior data, and generate a behavior matching analysis result, including:

[0012] Perform hand movement trajectory extraction processing on the performance behavior data to generate a sequence of hand joint angle changes; at the same time, perform spatial discretization processing on the pressure distribution data of the instrument contact surface to generate a contact point pressure distribution matrix;

[0013] Perform spatio-temporal alignment processing on the sequence of hand joint angle changes and the contact point pressure distribution matrix to generate a multi-modal behavior feature vector;

[0014] Perform similarity calculation processing on the multi-modal behavior feature vector and the standard action parameters in the composite symbol description framework to generate a set of action deviation scores;

[0015] Perform weighted summation processing on the set of action deviation scores to generate a behavior matching analysis result.

[0016] Furthermore, perform similarity calculation processing on the multi-modal behavior feature vector and the standard action parameters in the composite symbol description framework to generate a set of action deviation scores, including:

[0017] Perform spatio-temporal feature decoupling processing on the standard action parameters to generate a hand movement reference vector and a contact point pressure reference matrix;

[0018] Perform dynamic time warping processing on the sequence of hand joint angle changes in the multi-modal behavior feature vector and the hand movement reference vector to generate a movement trajectory deviation score;

[0019] Perform spatial convolution processing on the contact point pressure distribution matrix in the multi-modal behavior feature vector and the contact point pressure reference matrix to generate a pressure distribution deviation score;

[0020] Perform weighted fusion processing on the movement trajectory deviation score and the pressure distribution deviation score to generate a set of action deviation scores.

[0021] Furthermore, trigger a virtual tutor prompt system based on the behavior matching analysis result to generate a set of dynamically corrected mapping parameters, including:

[0022] Perform a threshold comparison process on the action deviation score in the behavior matching analysis result to generate a prompt trigger identifier;

[0023] According to the type code of the prompt trigger identifier, match the corresponding correction strategy template from the composite symbol description framework to generate a prompt type identifier;

[0024] Based on the prompt type identifier, call the three-dimensional animation database of the virtual tutor prompt system to generate multimodal prompt information including limb movement correction guidelines;

[0025] According to the key node data of the multimodal prompt information, perform an incremental adjustment process on the sound field reflection coefficient and temperament compatibility parameter in the mapping parameter set to generate a dynamically corrected mapping parameter set.

[0026] Furthermore, according to the key node data of the multimodal prompt information, perform an incremental adjustment process on the sound field reflection coefficient and temperament compatibility parameter in the mapping parameter set to generate a dynamically corrected mapping parameter set, including:

[0027] Perform a bone key point analysis process on the limb movement correction guidelines in the multimodal prompt information to generate a sound field reflection influence area identifier and a temperament adjustment priority identifier;

[0028] Based on the sound field reflection influence area identifier, perform a spatial sound field simulation process on the sound field reflection coefficient in the mapping parameter set to generate an adjusted sound field reflection coefficient;

[0029] According to the temperament adjustment priority identifier, perform a scale interpolation compensation process on the temperament compatibility parameter to generate an adjusted temperament compatibility parameter;

[0030] Integrate the adjusted sound field reflection coefficient and the adjusted temperament compatibility parameter into the mapping parameter set to generate a dynamically corrected mapping parameter set.

[0031] Furthermore, based on the symbol metadata set, perform a cultural feature matching process through a context association model to generate a composite symbol description framework, including:

[0032] Perform a similarity calculation process on the decorative symbol features in the symbol metadata set and the architectural pattern features in the national cultural feature library to generate a cultural association weight value;

[0033] Extract the architectural structure features associated with the decorative symbol features from the national cultural feature library to generate scene element parameters;

[0034] Perform a timbre spectrum analysis process on the pitch dimension features in the symbol metadata set to generate an acoustic parameter set;

[0035] Based on the performance technique dimension features in the symbol metadata set, perform spatio-temporal alignment processing on the motion trajectory and musical symbols to generate a set of limb motion parameters;

[0036] According to the cultural association weight value, perform multi-modal fusion processing on the acoustic parameter set, the limb motion parameter set, and the scene element parameters to generate a composite symbol description framework.

[0037] Furthermore, input the composite symbol description framework into the virtual reality engine, and perform cross-cultural mapping processing through the pitch conversion layer and the cultural element adaptation algorithm to generate a set of mapping parameters for the target teaching scene, including:

[0038] Through the pitch conversion layer, perform microtone compensation processing and temperament compatibility processing on the acoustic parameters in the composite symbol description framework to generate an adjusted acoustic parameter set;

[0039] Through the cultural element adaptation algorithm, perform architectural sound field adaptation processing and visual element matching processing on the scene element parameters in the composite symbol description framework to generate a cultural adaptation parameter set;

[0040] Input the adjusted acoustic parameter set and the cultural adaptation parameter set into the spatial mapping module of the virtual reality engine for parameter fusion processing to generate a set of mapping parameters for the target teaching scene.

[0041] Furthermore, based on the set of mapping parameters, call the three-dimensional scene rendering module to perform immersive environment construction processing to generate a cultural theme teaching scene. The cultural teaching scene includes a three-dimensional visualization model of musical symbols, including:

[0042] Perform acoustic field parameter analysis processing on the sound field reflection coefficient and temperament compatibility parameters in the set of mapping parameters to generate an acoustic environment model; at the same time, perform visual parameter extraction processing on the architectural structure features to generate a three-dimensional model of the historical building;

[0043] Convert the pitch dimension and duration dimension features in the composite symbol description framework into an interactive geometric topology structure to generate a three-dimensional visualization model of musical symbols;

[0044] Input the acoustic environment model, the three-dimensional model of the historical building, and the three-dimensional visualization model of musical symbols into the three-dimensional scene rendering module for spatial fusion processing to generate a cultural theme teaching scene including multi-modal teaching elements.

[0045] Furthermore, perform multi-modal data acquisition processing on non-Western musical symbol systems, and perform symbol feature deconstruction processing on the collected data to generate a symbol metadata set, including:

[0046] Perform high-resolution image scanning processing on Gongchepu characters to generate a standardized symbol image data set;

[0047] Perform multi-channel recording of the raga scale performance process to generate a raw audio dataset including microtonal features;

[0048] The wearable sensor is used to obtain the body movement trajectory data when playing the guqin Yinhua fingering method, and a three-dimensional motion capture data set is generated;

[0049] The symbol image dataset is processed by stroke topology analysis to generate pitch coding features and decorative symbol features; the original audio dataset is processed by differential audio spectrum analysis to generate musical pitch shift features; the three-dimensional motion capture dataset is processed by motion trajectory modeling to generate performance technique related parameters;

[0050] The pitch encoding features, decorative symbol features, musical scale shift features and performance technique-related parameters are subjected to multimodal alignment to generate a symbol metadata set.

[0051] Furthermore, according to the type code of the prompt trigger identifier, a corresponding correction strategy template is matched from the composite symbol description framework to generate a prompt type identifier, including:

[0052] Perform semantic segmentation on the type code of the prompt trigger identifier to generate an error type identifier and a severity identifier;

[0053] Based on the error type identifier, a candidate correction strategy template set is retrieved from a performance rule library of a composite symbol description framework;

[0054] According to the severity identifier, the candidate corrective strategy template set is sorted by applicability weight to generate the optimal corrective strategy template;

[0055] Convert metadata encodings of optimal correction strategy templates into standardized prompt type identifiers.

[0056] The technical solutions provided by this application include the following technical effects: By providing a virtual reality music teaching system based on the scenario teaching method, the system includes: a multimodal data acquisition and feature deconstruction module, which is used to perform multimodal data acquisition and processing on non-Western music symbol systems, and perform symbol feature deconstruction processing on the acquired data to generate a set of symbol metadata. The set of symbol metadata includes pitch dimension, duration dimension, and performance technique dimension; a cultural context association module, which is used to perform cultural feature matching processing through a context association model based on the set of symbol metadata to generate a composite symbol description framework; a cross-cultural dynamic mapping module, which is used to input the composite symbol description framework into a virtual reality engine and perform cross-cultural mapping processing through a pitch conversion layer and a cultural element adaptation algorithm to generate a set of mapping parameters for the target teaching scenario; an immersive scenario construction module, which is used to call a three-dimensional scene rendering module to perform immersive environment construction processing based on the set of mapping parameters to generate a cultural theme teaching scenario. The cultural teaching scenario includes a three-dimensional visualization model of music symbols; a behavior feedback and parameter correction module, which is used to obtain performance behavior data in the cultural teaching scenario through a motion capture device, perform symbol-behavior matching analysis processing on the performance behavior data to generate a behavior matching analysis result, and trigger a virtual tutor prompt system based on the behavior matching analysis result to generate a dynamically corrected set of mapping parameters, so as to solve the problems of insufficient symbol digitization compatibility and cultural context fragmentation, thereby improving the parsing accuracy of non-Western music microtones and special symbols, enhancing the relevance between music symbols and cultural elements in the virtual teaching scenario, and reducing the distortion of the original characteristics of ethnic music and the cognitive gap of learners. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for the description of the embodiments or related technologies. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0058] Figure 1 It is a structural diagram of a virtual reality music teaching system based on the scenario teaching method in an embodiment of the present invention;

[0059] Figure 2 It is a flowchart of obtaining performance behavior data in the cultural teaching scenario through a motion capture device, performing symbol-behavior matching analysis processing on the performance behavior data, and generating a behavior matching analysis result in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] In order to make the above-mentioned objects, features, and advantages of the present application more obvious and understandable, the following will describe in detail the specific implementation manners of the present application with reference to the accompanying drawings. Many specific details are set forth in the following description in order to fully understand the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the connotation of the application. Therefore, the present application is not limited by the specific embodiments disclosed below.

[0061] As Figure 1 shown, the present application provides a virtual reality music teaching system 100 based on the scenario teaching method. The system includes:

[0062] A multi-modal data acquisition and feature deconstruction module 101, which is used to perform multi-modal data acquisition and processing on non-Western music symbol systems, perform symbol feature deconstruction processing on the acquired data, and generate a set of symbol metadata. The set of symbol metadata includes a pitch dimension, a duration dimension, and a performance technique dimension.

[0063] Specifically, high-resolution image scanning is performed on non-Western music symbols (such as Gongche notation and Guqin reduced character notation) to generate a standardized symbol image data set. Multi-channel recording is performed on the performance process of non-Western music (such as Rag scales and Guqin vibrato fingering techniques) to generate an original audio data set containing microtone features. The limb movement trajectories during performance (such as Guqin fingering movements) are recorded through wearable sensors to generate a three-dimensional motion capture data set. Stroke topology analysis is performed on the symbol image data set to extract pitch encoding features (such as pitch symbols in Gongche notation) and decorative symbol features (such as fingering symbols in Guqin reduced character notation).

[0064] Differential audio spectrum analysis is performed on the original audio data set to extract pitch deviation features (such as frequency deviations of Arabic quarter tones). Motion trajectory modeling is performed on the three-dimensional motion capture data set to extract performance technique correlation parameters (such as the strength and speed features of the Guqin "vibrato" fingering technique). The pitch encoding features, decorative symbol features, pitch deviation features, and performance technique correlation parameters are subjected to multi-modal alignment processing to ensure the synchronization of different modal data. The aligned features are integrated into a set of symbol metadata, including a pitch dimension (such as pitch encoding), a duration dimension (such as rhythm features), and a performance technique dimension (such as fingering movement features). Through the above steps, it is possible to more accurately analyze non-Western music symbol systems and generate a structured set of symbol metadata, providing a basis for subsequent cultural context association and virtual scene construction.

[0065] A cultural context association module 102, which is used to perform cultural feature matching processing through a context association model based on the set of symbol metadata, and generate a composite symbol description framework.

[0066] Specifically, a set of symbolic metadata including pitch dimension, duration dimension, and performance technique dimension is used as input, and the above data is generated through multi-modal data collection and feature deconstruction processing. Using the context association model, the features in the set of symbolic metadata (such as pitch encoding, ornament symbols, pitch deviation, etc.) are calculated for similarity with the features in the ethnic culture feature library (such as architectural patterns, timbre spectra, etc.) to generate a cultural association weight value. According to the cultural association weight value, the acoustic parameter set, body movement parameter set, and scene element parameter set are processed for multi-modal fusion to generate a composite symbolic description framework. The subsequent generated composite symbolic description framework contains the relevance between music symbols and cultural elements, and can provide support for subsequent virtual scene construction and teaching applications.

[0067] The cross-cultural dynamic mapping module 103 is used to input the composite symbolic description framework into the virtual reality engine, and perform cross-cultural mapping processing through the pitch conversion layer and the cultural element adaptation algorithm to generate a set of mapping parameters for the target teaching scene.

[0068] Specifically, a composite symbolic description framework containing the relevance between music symbols and cultural elements is used as input, and the above framework is generated through the context association model processing. Perform differential tone compensation processing on the acoustic parameters in the composite symbolic description framework to adapt to the pitch characteristics of non-Western music. Perform temperament compatibility processing to ensure that the pitches of different cultural music systems can be accurately presented in virtual reality. Perform architectural sound field adaptation processing on the scene element parameters in the composite symbolic description framework to simulate the acoustic characteristics under a specific cultural environment. Perform visual element matching processing to ensure that the visual elements in the virtual scene match the cultural background of the music symbols. Input the acoustic parameter set and cultural adaptation parameter set processed by the pitch conversion layer and the cultural element adaptation algorithm into the spatial mapping module of the virtual reality engine for parameter fusion processing to generate a set of mapping parameters for the target teaching scene.

[0069] The immersive scene construction module 104 is used to, based on the set of mapping parameters, call the three-dimensional scene rendering module to perform immersive environment construction processing to generate a cultural theme teaching scene, and the cultural teaching scene includes a three-dimensional visualization model of music symbols.

[0070] Specifically, perform sound field parameter analysis on the sound field reflection coefficient and temperament compatibility parameter in the mapping parameter set to generate an acoustic environment model; at the same time, perform visual parameter extraction on the architectural structure characteristics to generate a three-dimensional model of the historical building. Convert the pitch dimension and time value dimension features in the composite symbol description framework into an interactive geometric topology structure to generate a three-dimensional visualization model of musical symbols. Input the acoustic environment model, the three-dimensional model of the historical building, and the three-dimensional visualization model of musical symbols into the three-dimensional scene rendering module for spatial fusion processing to generate a cultural theme teaching scene including multi-modal teaching elements. Through the processing of the three-dimensional scene rendering module, an immersive cultural theme teaching scene is generated later, providing a multi-sensory linked music learning experience for learners.

[0071] The behavior feedback and parameter correction module 105 is used to obtain the performance behavior data in the cultural teaching scene through an action capture device, perform symbol-behavior matching analysis on the performance behavior data to generate a behavior matching analysis result, trigger a virtual tutor prompt system based on the behavior matching analysis result, and generate a dynamically corrected mapping parameter set.

[0072] Specifically, use an optical, inertial, or electromagnetic action capture device to capture the action data of the performer in the cultural teaching scene, including the sequence of hand joint angle changes and the pressure distribution data on the instrument contact surface. Process the performance behavior data, extract the hand movement trajectory and pressure distribution characteristics, and generate a multi-modal behavior feature vector. Calculate the similarity between the behavior feature vector and the standard action parameters in the composite symbol description framework to generate a set of action deviation scores. According to the action deviation scores, trigger a virtual tutor prompt system to generate multi-modal prompt information including limb movement correction guidelines. Based on the key node data of the multi-modal prompt information, incrementally adjust the sound field reflection coefficient and temperament compatibility parameter in the mapping parameter set to generate a dynamically corrected mapping parameter set. Through the above steps, it is possible to analyze the performance behavior in real time, provide relatively accurate feedback, and dynamically optimize the parameters of the virtual teaching scene to improve the teaching effect.

[0073] An embodiment of the present application provides a virtual reality music teaching system based on the scenario teaching method, including: a multimodal data acquisition and feature deconstruction module, which is used to perform multimodal data acquisition and processing on non-Western music symbol systems, and perform symbol feature deconstruction processing on the acquired data to generate a set of symbol metadata. The set of symbol metadata includes a pitch dimension, a duration dimension, and a performance technique dimension; a cultural context association module, which is used to perform cultural feature matching processing through a context association model based on the set of symbol metadata to generate a composite symbol description framework; a cross-cultural dynamic mapping module, which is used to input the composite symbol description framework into a virtual reality engine and perform cross-cultural mapping processing through a pitch conversion layer and a cultural element adaptation algorithm to generate a set of mapping parameters for the target teaching scenario; an immersive scenario construction module, which is used to call a three-dimensional scene rendering module to perform immersive environment construction processing based on the set of mapping parameters to generate a cultural theme teaching scenario. The cultural teaching scenario includes a three-dimensional visualization model of music symbols; a behavior feedback and parameter correction module, which is used to obtain performance behavior data in the cultural teaching scenario through a motion capture device, perform symbol-behavior matching analysis processing on the performance behavior data to generate a behavior matching analysis result, and trigger a virtual tutor prompt system based on the behavior matching analysis result to generate a dynamically corrected set of mapping parameters, so as to solve the problems of insufficient symbol digitization compatibility and cultural context fragmentation, thereby improving the parsing accuracy of non-Western music microtones and special symbols, enhancing the relevance between music symbols and cultural elements in the virtual teaching scenario, and reducing the distortion of the original characteristics of ethnic music and the cognitive gap of learners.

[0074] As Figure 2 shown, in one embodiment, performance behavior data in the cultural teaching scenario is obtained through a motion capture device, and symbol-behavior matching analysis processing is performed on the performance behavior data to generate a behavior matching analysis result, including:

[0075] S201: Perform hand movement trajectory extraction processing on the performance behavior data to generate a sequence of hand joint angle changes; at the same time, perform spatial discretization processing on the instrument contact surface pressure distribution data to generate a contact point pressure distribution matrix;

[0076] S202: Perform spatio-temporal alignment processing on the sequence of hand joint angle changes and the contact point pressure distribution matrix to generate a multimodal behavior feature vector;

[0077] S203: Perform similarity calculation processing on the multimodal behavior feature vector and the standard action parameters in the composite symbol description framework to generate a set of action deviation scores;

[0078] S204: Perform weighted summation processing on the set of action deviation scores to generate a behavior matching analysis result.

[0079] Specifically, the performance behavior data in the cultural teaching scenario is obtained through a motion capture device, and the hand motion trajectory in the performance behavior data is extracted and processed to generate a sequence of hand joint angle changes; at the same time, the spatial discretization process is performed on the pressure distribution data of the instrument contact surface to generate a contact point pressure distribution matrix. Then, the sequence of hand joint angle changes and the contact point pressure distribution matrix are subjected to spatio-temporal alignment processing to generate a multi-modal behavior feature vector. After that, the similarity calculation process is performed between the multi-modal behavior feature vector and the standard action parameters in the composite symbol description framework to generate a set of action deviation scores. Then, the weighted summation process is performed on the set of action deviation scores to generate the result of behavior matching analysis. The above process ensures the accurate analysis of the performance behavior data and provides data support for subsequent virtual tutor prompts and parameter correction.

[0080] Further, the similarity calculation process is performed between the multi-modal behavior feature vector and the standard action parameters in the composite symbol description framework to generate a set of action deviation scores, including:

[0081] (1) Perform spatio-temporal feature decoupling processing on the standard action parameters to generate a hand motion reference vector and a contact point pressure reference matrix;

[0082] (2) Perform dynamic time warping processing on the sequence of hand joint angle changes in the multi-modal behavior feature vector and the hand motion reference vector to generate a motion trajectory deviation score;

[0083] (3) Perform spatial convolution processing on the contact point pressure distribution matrix in the multi-modal behavior feature vector and the contact point pressure reference matrix to generate a pressure distribution deviation score;

[0084] (4) Perform weighted fusion processing on the motion trajectory deviation score and the pressure distribution deviation score to generate a set of action deviation scores.

[0085] Specifically, the spatio-temporal feature decoupling process is performed on the standard action parameters to generate a hand movement reference vector and a contact pressure reference matrix. This step decomposes complex action parameters into more basic features for subsequent comparison and analysis. The dynamic time warping process is performed on the hand joint angle change sequence in the multi-modal behavior feature vector and the hand movement reference vector to generate a motion trajectory deviation score. Dynamic time warping is an algorithm for comparing two time series, capable of handling the stretching and compression of time series on the time axis, thus accurately evaluating the similarity of motion trajectories. The spatial convolution process is performed on the contact pressure distribution matrix in the multi-modal behavior feature vector and the contact pressure reference matrix to generate a pressure distribution deviation score. The spatial convolution process is used to analyze and compare the spatial distribution characteristics of two matrices, effectively evaluating the similarity of pressure distributions. The weighted fusion process is performed on the motion trajectory deviation score and the pressure distribution deviation score to generate a set of action deviation scores. The weighted fusion process combines the scores of different features and comprehensively evaluates the overall deviation degree of the action according to their respective weights, thus generating a more comprehensive set of action deviation scores.

[0086] Furthermore, based on the behavior matching analysis results, the virtual tutor prompt system is triggered to generate a set of dynamically corrected mapping parameters, including:

[0087] (1) Perform a threshold comparison process on the action deviation scores in the behavior matching analysis results to generate a prompt trigger identifier;

[0088] (2) According to the type code of the prompt trigger identifier, match the corresponding correction strategy template from the composite symbol description framework to generate a prompt type identifier;

[0089] (3) Based on the prompt type identifier, call the 3D animation database of the virtual tutor prompt system to generate multi-modal prompt information including limb movement correction guidelines;

[0090] (4) According to the key node data of the multi-modal prompt information, perform an incremental adjustment process on the sound field reflection coefficient and the temperament compatibility parameter in the mapping parameter set to generate a set of dynamically corrected mapping parameters.

[0091] Specifically, a threshold comparison process is performed on the action deviation score in the behavior matching analysis result to determine whether a prompt needs to be triggered. If the deviation exceeds the preset threshold, a prompt trigger identifier is generated. According to the type code of the prompt trigger identifier, the corresponding correction strategy template is retrieved and matched from the composite symbol description framework to generate a prompt type identifier. This step ensures that corresponding correction strategies can be provided according to specific error types. Based on the prompt type identifier, the three-dimensional animation database of the virtual tutor prompt system is called to generate multimodal prompt information including limb movement correction guidelines. The above prompt information includes multiple modalities such as vision and hearing to adapt to different learning styles and needs. According to the key node data in the multimodal prompt information, incremental adjustment processing is performed on the sound field reflection coefficient and temperament compatibility parameter in the mapping parameter set. This step optimizes the virtual teaching scenario by fine-tuning the parameters to make it more conform to the correct performance behavior and cultural context. Through the above steps, the learner's performance behavior can be responded to in real time, providing personalized feedback and correction suggestions, thereby improving the teaching effect and learning experience.

[0092] Furthermore, according to the key node data of the multimodal prompt information, incremental adjustment processing is performed on the sound field reflection coefficient and temperament compatibility parameter in the mapping parameter set to generate a dynamically corrected mapping parameter set, including:

[0093] (1) Perform bone key point analysis processing on the limb movement correction guidelines in the multimodal prompt information to generate a sound field reflection influence area identifier and a temperament adjustment priority identifier;

[0094] (2) Based on the sound field reflection influence area identifier, perform spatial sound field simulation processing on the sound field reflection coefficient in the mapping parameter set to generate an adjusted sound field reflection coefficient;

[0095] (3) According to the temperament adjustment priority identifier, perform scale interpolation compensation processing on the temperament compatibility parameter to generate an adjusted temperament compatibility parameter;

[0096] (4) Integrate the adjusted sound field reflection coefficient and the adjusted temperament compatibility parameter into the mapping parameter set to generate a dynamically corrected mapping parameter set.

[0097] Specifically, analyze the limb movement correction guidelines in the multi-modal prompt information to extract key skeletal key point data. The above data identifies the key nodes of the performance movement and provides a basis for subsequent parameter adjustment. According to the skeletal key point data, determine the sound field reflection influence area. Different movement postures and positions will have different effects on the sound field reflection. By analyzing the above key points, the affected area of the sound field reflection can be accurately located. Also based on the skeletal key point data, determine the priority of temperament adjustment. Adjust the sound field reflection coefficient in the mapping parameter set. Use spatial sound field simulation technology to adjust the sound field reflection coefficient according to the determined influence area to more realistically simulate the sound field environment. Adjust the temperament compatibility parameters. Use the scale interpolation compensation technology to fine-tune the temperament compatibility parameters according to the determined priority to ensure that the musical scales of different cultural music systems can be accurately presented in virtual reality. Integrate the adjusted sound field reflection coefficient and temperament compatibility parameters back into the mapping parameter set to generate a dynamically corrected mapping parameter set. This step ensures that the virtual teaching scenario can be optimized according to the real-time feedback of the performance behavior and provide more accurate teaching guidance.

[0098] Furthermore, based on the symbol metadata set, perform cultural feature matching processing through the context association model to generate a composite symbol description framework, including:

[0099] (1) Calculate the similarity between the decorative symbol features in the symbol metadata set and the architectural pattern features in the ethnic culture feature library to generate a cultural association weight value;

[0100] (2) Extract the architectural structure features associated with the decorative symbol features from the ethnic culture feature library to generate scene element parameters;

[0101] (3) Perform timbre spectrum analysis on the pitch dimension features in the symbol metadata set to generate an acoustic parameter set;

[0102] (4) Based on the performance technique dimension features in the symbol metadata set, perform spatio-temporal alignment processing of the movement trajectory and musical symbols to generate a limb movement parameter set;

[0103] (5) According to the cultural association weight value, perform multi-modal fusion processing on the acoustic parameter set, limb movement parameter set, and scene element parameters to generate a composite symbol description framework.

[0104] Specifically, the decorative symbol features in the symbol metadata set are calculated for similarity with the architectural pattern features in the national culture feature library to generate a cultural association weight value. This step provides a basis for subsequent fusion processing by comparing the similarities between different features. Architectural structure features associated with the decorative symbol features are extracted from the national culture feature library to generate scene element parameters. The above parameters will be used to construct a virtual scene related to the cultural background. The pitch dimension features in the symbol metadata set are processed by timbre spectrum analysis to generate an acoustic parameter set. This step converts pitch information into quantifiable acoustic parameters to accurately present music features in the virtual scene. Based on the performance technique dimension features in the symbol metadata set, spatio-temporal alignment processing of the motion trajectory and music symbols is performed to generate a set of limb motion parameters. This step ensures the synchronization of performance actions and music symbols in time and space. According to the cultural association weight value, multi-modal fusion processing is performed on the acoustic parameter set, limb motion parameter set, and scene element parameters to generate a composite symbol description framework. Multi-modal fusion generates a symbol description framework by combining information from different modalities, providing support for subsequent virtual reality applications.

[0105] Furthermore, the composite symbol description framework is input into the virtual reality engine, and cross-cultural mapping processing is performed through the pitch conversion layer and cultural element adaptation algorithm to generate a set of mapping parameters for the target teaching scene, including:

[0106] (1) Through the pitch conversion layer, differential tone compensation processing and temperament compatibility processing are performed on the acoustic parameters in the composite symbol description framework to generate an adjusted acoustic parameter set;

[0107] (2) Through the cultural element adaptation algorithm, architectural sound field adaptation processing and visual element matching processing are performed on the scene element parameters in the composite symbol description framework to generate a set of cultural adaptation parameters;

[0108] (3) The adjusted acoustic parameter set and the cultural adaptation parameter set are input into the space mapping module of the virtual reality engine for parameter fusion processing to generate a set of mapping parameters for the target teaching scene.

[0109] Specifically, after the composite symbol description framework is input into the virtual reality engine, the microtonal compensation and temperament compatibility processing are performed on the acoustic parameters through the temperament conversion layer to adapt to the temperament characteristics of different cultural music systems, and a set of adjusted acoustic parameters is generated. At the same time, the cultural element adaptation algorithm is used to perform architectural sound field adaptation and visual element matching processing on the scene element parameters to ensure that the acoustic and visual elements of the virtual scene match the target culture, and a set of cultural adaptation parameters is generated. Then, the above two sets of parameters are input into the space mapping module of the virtual reality engine for fusion processing to generate a set of mapping parameters for the target teaching scene, thereby realizing cross-cultural mapping and providing an immersive cultural theme teaching scene for learners.

[0110] Furthermore, based on the set of mapping parameters, the three-dimensional scene rendering module is called for immersive environment construction processing to generate a cultural theme teaching scene, and the cultural teaching scene includes a three-dimensional visualization model of music symbols, including:

[0111] (1) Perform acoustic field parameter analysis processing on the sound field reflection coefficient and temperament compatibility parameters in the set of mapping parameters to generate an acoustic environment model; at the same time, perform visual parameter extraction processing on the architectural structure characteristics to generate a three-dimensional model of the historical building;

[0112] (2) Convert the pitch dimension and time value dimension characteristics in the composite symbol description framework into an interactive geometric topology structure to generate a three-dimensional visualization model of music symbols;

[0113] (3) Input the acoustic environment model, the three-dimensional model of the historical building, and the three-dimensional visualization model of music symbols into the three-dimensional scene rendering module for space fusion processing to generate a cultural theme teaching scene including multi-modal teaching elements.

[0114] Specifically, perform acoustic field parameter analysis on the sound field reflection coefficients and temperament compatibility parameters in the mapping parameter set to generate an acoustic environment model for simulating a real cultural sound field environment. At the same time, perform visual parameter extraction on the architectural structure features to generate a 3D model of the historical building, ensuring that the visual elements of the virtual scene match the historical buildings of the target culture. Convert the pitch dimension and time value dimension features in the composite symbol description framework into an interactive geometric topology structure to generate a three-dimensional visualization model of musical symbols. The above models support real-time interaction between users and musical symbols, enhancing the interactivity of teaching. Input the acoustic environment model, the 3D model of the historical building, and the three-dimensional visualization model of musical symbols into the 3D scene rendering module for spatial fusion processing. Through spatial fusion technology, integrate acoustic, visual, and interactive elements into a unified immersive environment. Then generate a cultural theme teaching scene including multi-modal teaching elements, providing learners with an immersive learning experience that combines hearing, vision, and interaction. The above process ensures that the virtual teaching scene is not only acoustically and visually faithful to the target culture but also enhances the learning participation and effect through an interactive musical symbol model through the integration and rendering of multi-modal data.

[0115] Furthermore, perform multi-modal data acquisition on non-Western musical symbol systems and perform symbol feature deconstruction on the acquired data to generate a set of symbol metadata, including:

[0116] (1) Perform high-resolution image scanning on Gongchepu characters to generate a standardized symbol image data set;

[0117] (2) Perform multi-channel recording on the performance process of Rag scales to generate an original audio data set including microtone features;

[0118] (3) Obtain the limb movement trajectory data during the performance of Guqin Yinnao fingering through wearable sensors to generate a three-dimensional motion capture data set;

[0119] (4) Perform stroke topology analysis on the symbol image data set to generate pitch encoding features and decorative symbol features; perform differential audio spectrum analysis on the original audio data set to generate temperament offset features; perform motion trajectory modeling on the three-dimensional motion capture data set to generate performance technique correlation parameters;

[0120] (5) Perform multi-modal alignment on the pitch encoding features, decorative symbol features, temperament offset features, and performance technique correlation parameters to generate a set of symbol metadata.

[0121] Specifically, high-resolution image scanning is performed on Gongche score characters to generate a standardized symbol image dataset. Multi-channel recording is carried out on the performance process of Rag scales to generate an original audio dataset including microtonal features. The limb movement trajectory data during the performance of Guqin Yinnao fingering is obtained through wearable sensors to generate a three-dimensional motion capture dataset. Stroke topology analysis is performed on the symbol image dataset to generate pitch encoding features and decorative symbol features. Differential audio spectrum analysis is performed on the original audio dataset to generate pitch deviation features. Motion trajectory modeling is performed on the three-dimensional motion capture dataset to generate performance technique correlation parameters. The pitch encoding features, decorative symbol features, pitch deviation features, and performance technique correlation parameters are subjected to multi-modal alignment to ensure the synchronization of different modal data. The aligned features are integrated into a symbol metadata set, including pitch dimension, duration dimension, and performance technique dimension. The above process generates a structured symbol metadata set through the acquisition and feature deconstruction of multi-modal data, providing a basis for subsequent cultural context association and virtual scene construction.

[0122] Furthermore, according to the type code of the prompt trigger identifier, the corresponding correction strategy template is matched from the composite symbol description framework to generate a prompt type identifier, including:

[0123] (1) Perform semantic segmentation on the type code of the prompt trigger identifier to generate an error type identifier and a severity identifier;

[0124] (2) Based on the error type identifier, retrieve a candidate correction strategy template set from the performance rule library of the composite symbol description framework;

[0125] (3) Perform applicability weight sorting on the candidate correction strategy template set according to the severity identifier to generate an optimal correction strategy template;

[0126] (4) Convert the metadata encoding of the optimal correction strategy template into a standardized prompt type identifier.

[0127] Specifically, semantic segmentation processing is performed on the type code of the prompt trigger identifier to generate an error type identifier and a severity identifier. This step extracts the parts representing the error type and severity by analyzing the structure of the type code, providing a basis for subsequent matching and sorting. Based on the error type identifier, all relevant candidate correction strategy templates are retrieved from the performance rule library of the composite symbol description framework to form a set. This step ensures that all possible correction strategies related to a specific error type can be found. The candidate correction strategy template set is sorted according to the applicability weight based on the severity identifier to generate an optimal correction strategy template. This step determines the most suitable correction strategy according to the severity of the error by evaluating the applicability of each candidate template, ensuring that the most effective feedback can be provided. The metadata encoding of the optimal correction strategy template is converted into a standardized prompt type identifier.

[0128] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the indication of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order limit for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or steps or stages in other steps.

[0129] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present disclosure solution. Those of ordinary skill in the art can understand and implement it without creative work.

[0130] The above-described embodiments only represent several implementation manners of the embodiments of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the embodiments of the application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the embodiments of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the embodiments of the present application.

Claims

1. A virtual reality music teaching system based on the scenario teaching method, characterized in that, The system includes: A multi-modal data acquisition and feature deconstruction module, which is used to perform multi-modal data acquisition and processing on non-Western music symbol systems, and perform symbol feature deconstruction processing on the acquired data to generate a set of symbol metadata. The set of symbol metadata includes pitch dimension, duration dimension, and performance technique dimension; A cultural context association module, which is used to perform cultural feature matching processing through a context association model based on the set of symbol metadata to generate a composite symbol description framework; A cross-cultural dynamic mapping module, which is used to input the composite symbol description framework into a virtual reality engine and perform cross-cultural mapping processing through a pitch conversion layer and a cultural element adaptation algorithm to generate a set of mapping parameters for the target teaching scenario; An immersive scenario construction module, which is used to call a three-dimensional scene rendering module to perform immersive environment construction processing based on the set of mapping parameters to generate a cultural theme teaching scenario. The cultural teaching scenario includes a three-dimensional visualization model of music symbols; A behavior feedback and parameter correction module, which is used to obtain performance behavior data in the cultural teaching scenario through a motion capture device, perform symbol-behavior matching analysis processing on the performance behavior data to generate a behavior matching analysis result, and trigger a virtual tutor prompt system based on the behavior matching analysis result to generate a dynamically corrected set of mapping parameters.

2. The virtual reality music teaching system based on the scenario teaching method according to claim 1, wherein The obtaining of the performance behavior data in the cultural teaching scenario through the motion capture device and the performing of symbol-behavior matching analysis processing on the performance behavior data to generate a behavior matching analysis result includes: Performing hand movement trajectory extraction processing on the performance behavior data to generate a sequence of hand joint angle changes; at the same time, performing spatial discretization processing on the pressure distribution data of the instrument contact surface to generate a contact point pressure distribution matrix; Performing spatio-temporal alignment processing on the sequence of hand joint angle changes and the contact point pressure distribution matrix to generate a multi-modal behavior feature vector; Performing similarity calculation processing on the multi-modal behavior feature vector and the standard action parameters in the composite symbol description framework to generate a set of action deviation scores; Performing weighted summation processing on the set of action deviation scores to generate the behavior matching analysis result.

3. A virtual reality music teaching system based on the scenario teaching method according to claim 2, characterized in that, The performing of similarity calculation processing on the multi-modal behavior feature vector and the standard action parameters in the composite symbol description framework to generate a set of action deviation scores includes: Performing spatio-temporal feature decoupling processing on the standard action parameters to generate a hand movement reference vector and a contact point pressure reference matrix; Performing dynamic time warping processing on the sequence of hand joint angle changes in the multi-modal behavior feature vector and the hand movement reference vector to generate a motion trajectory deviation score; Performing spatial convolution processing on the contact point pressure distribution matrix in the multi-modal behavior feature vector and the contact point pressure reference matrix to generate a pressure distribution deviation score; Performing weighted fusion processing on the motion trajectory deviation score and the pressure distribution deviation score to generate the set of action deviation scores.

4. A virtual reality music teaching system based on the scenario teaching method according to claim 1, characterized in that, The triggering of the virtual tutor prompt system based on the behavior matching analysis result to generate a dynamically corrected set of mapping parameters includes: Perform a threshold comparison process on the action deviation score in the behavior matching analysis result to generate a prompt trigger identifier; According to the type code of the prompt trigger identifier, match the corresponding correction strategy template from the composite symbol description framework to generate a prompt type identifier; Based on the prompt type identifier, call the three-dimensional animation database of the virtual tutor prompt system to generate multimodal prompt information including limb movement correction guidelines; According to the key node data of the multimodal prompt information, perform an incremental adjustment process on the sound field reflection coefficient and temperament compatibility parameter in the mapping parameter set to generate the dynamically corrected mapping parameter set.

5. A virtual reality music teaching system based on the scenario teaching method according to claim 4, characterized in that, The step of performing an incremental adjustment process on the sound field reflection coefficient and temperament compatibility parameter in the mapping parameter set according to the key node data of the multimodal prompt information to generate the dynamically corrected mapping parameter set includes: Perform bone key point analysis on the limb movement correction guidelines in the multimodal prompt information to generate a sound field reflection influence area identifier and a temperament adjustment priority identifier; Based on the sound field reflection influence area identifier, perform spatial sound field simulation on the sound field reflection coefficient in the mapping parameter set to generate an adjusted sound field reflection coefficient; According to the temperament adjustment priority identifier, perform scale interpolation compensation on the temperament compatibility parameter to generate an adjusted temperament compatibility parameter; Integrate the adjusted sound field reflection coefficient and the adjusted temperament compatibility parameter into the mapping parameter set to generate the dynamically corrected mapping parameter set.

6. A virtual reality music teaching system based on the scenario teaching method according to claim 1, characterized in that, The step of generating a composite symbol description framework by performing cultural feature matching through a context association model based on the symbol metadata set includes: Perform a similarity calculation on the decorative symbol features in the symbol metadata set and the architectural pattern features in the ethnic culture feature library to generate a cultural association weight value; Extract the architectural structure features associated with the decorative symbol features from the ethnic culture feature library to generate scene element parameters; Perform timbre spectrum analysis on the pitch dimension features in the symbol metadata set to generate an acoustic parameter set; Based on the performance technique dimension features in the symbol metadata set, perform spatio-temporal alignment of the action trajectory and musical symbols to generate a limb movement parameter set; According to the cultural association weight value, perform multimodal fusion on the acoustic parameter set, the limb movement parameter set, and the scene element parameters to generate the composite symbol description framework.

7. A virtual reality music teaching system based on the scenario teaching method according to claim 1, characterized in that, The step of inputting the composite symbol description framework into a virtual reality engine and performing cross-cultural mapping through a temperament conversion layer and a cultural element adaptation algorithm to generate a mapping parameter set for the target teaching scene includes: Perform differential tone compensation and temperament compatibility processing on the acoustic parameters in the composite symbol description framework through the temperament conversion layer to generate an adjusted acoustic parameter set; Perform architectural sound field adaptation and visual element matching on the scene element parameters in the composite symbol description framework through the cultural element adaptation algorithm to generate a cultural adaptation parameter set; Input the adjusted set of acoustic parameters and the set of culture adaptation parameters into the spatial mapping module of the virtual reality engine for parameter fusion processing to generate the set of mapping parameters for the target teaching scenario.

8. A virtual reality music teaching system based on the scenario teaching method according to claim 1, characterized in that, Based on the set of mapping parameters, call the three-dimensional scene rendering module to perform immersive environment construction processing to generate a culture-themed teaching scenario, and the culture teaching scenario includes a three-dimensional visualization model of musical symbols, including: Perform acoustic parameter analysis processing on the sound field reflection coefficient and temperament compatibility parameter in the set of mapping parameters to generate an acoustic environment model; at the same time, perform visual parameter extraction processing on the architectural structure features to generate a three-dimensional model of a historical building; Convert the pitch dimension and time value dimension features in the composite symbol description framework into an interactive geometric topology structure to generate a three-dimensional visualization model of musical symbols; Input the acoustic environment model, the three-dimensional model of the historical building, and the three-dimensional visualization model of musical symbols into the three-dimensional scene rendering module for spatial fusion processing to generate the culture-themed teaching scenario including multi-modal teaching elements.

9. A virtual reality music teaching system based on the scenario teaching method according to claim 1, characterized in that, Perform multi-modal data acquisition processing on the non-Western musical symbol system, and perform symbol feature deconstruction processing on the collected data to generate a set of symbol metadata, including: Perform high-resolution image scanning processing on Gongche score characters to generate a standardized symbol image data set; Perform multi-channel recording processing on the performance process of the Rag scale to generate an original audio data set including microtone features; Obtain the limb movement trajectory data during the performance of the Guqin Yinnao fingering through a wearable sensor to generate a three-dimensional motion capture data set; Perform stroke topology analysis processing on the symbol image data set to generate pitch encoding features and decorative symbol features; perform differential audio spectrum analysis processing on the original audio data set to generate pitch deviation features; perform motion trajectory modeling processing on the three-dimensional motion capture data set to generate performance technique correlation parameters; Perform multi-modal alignment processing on the pitch encoding features, the decorative symbol features, the pitch deviation features, and the performance technique correlation parameters to generate the set of symbol metadata.

10. A virtual reality music teaching system based on the scenario teaching method according to claim 4, characterized in that, According to the type code of the prompt trigger identifier, match the corresponding correction strategy template from the composite symbol description framework to generate a prompt type identifier, including: Perform semantic segmentation processing on the type code of the prompt trigger identifier to generate an error type identifier and a severity identifier; Based on the error type identifier, retrieve a set of candidate correction strategy templates from the performance rule library of the composite symbol description framework; Perform applicability weight sorting processing on the set of candidate correction strategy templates according to the severity identifier to generate an optimal correction strategy template; Convert the metadata encoding of the optimal correction strategy template into the standardized prompt type identifier.

Images