Multimodal interactive immersive music theory learning system and method

Abstract

The invention discloses a multi-mode interactive immersive music theory learning system and method, and relates to the technical field of combination of intelligence games and digital education, and the method comprises the steps: constructing a six-row * six-column phonetic lattice interaction matrix, binding each phonetic lattice with musical notes in five sound orders and high-low octave variants thereof, and setting a mode as default configuration; the MIDI pitch bound with the current sound grid is played, the tone is selected from an instrumental tone library, and a vibration signal, vibration intensity and pitch are generated based on dynamically changing the display attribute, the display color mapping and the animation effect of the sound grid; and receiving a mode switching instruction of a user, and recalculating the note arrangement rule of the phonetic lattice interaction matrix according to a target mode. According to the method, the 6 * 6 sound grid interaction matrix is constructed, the five sound orders and the octave variants of the five sound orders are dynamically bound through the cyclic arrangement algorithm, the limitation of a traditional linear keyboard layout is broken through, two-dimensional spatialization musical scale visualization cognition is achieved, and the five sound order memory efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of combining educational games with digital education, and in particular to a multi-modal interactive immersive music theory learning system and method. Background Art

[0002] Currently, digital tools are widely used in the field of music education technology to assist in music theory learning, such as interactive software based on piano keyboard simulation and visual scale training systems. Existing technologies mainly rely on the Western twelve-tone equal temperament system, combining MIDI sound sources with graphical interfaces to achieve functions such as note recognition and rhythm training. In recent years, the application of multimodal interactive technologies in music education has gradually increased, such as using color-coded pitch or vibration feedback to enhance learning and memory. However, these systems are generally designed for Western music systems and lack targeted support for the traditional Chinese pentatonic scale (gong, shang, jiao, zhi, yu). In addition, the interactive mode is single, making it difficult to achieve a culturally adapted immersive learning experience.

[0003] The core deficiency of existing technologies lies in the limitations of the scale system and interaction mode. Traditional systems mostly use a linear keyboard layout or a fixed scale matrix, which cannot dynamically adapt to the scale arrangement requirements of different ethnic modes (such as the difference between the complete Gong mode 1-2-3-5-6 within an octave range and the complete Shang mode 2-3-5-6-1 within an octave range). Existing interaction designs usually only provide one-way feedback (such as audio playback or static color prompts) and lack a real-time response mechanism for multi-modal (visual, auditory, and tactile) collaboration. This leads to a separation between pitch perception and mode understanding during the learning process. When users try to explore pentatonic mode variants, they cannot enhance the mode characteristics through dynamic visual mapping or tactile prompts, and it is difficult to detect and correct dissonant interval combinations, which limits the effectiveness of ethnic music education. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a multimodal interactive immersive music theory learning method that solves the problems of being unable to adapt to changes in national modes and lacking real-time interactive guidance.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides an immersive music theory learning method with multimodal interaction, which includes constructing a 6-row × 6-column grid interaction matrix, wherein each grid is bound to a note in the pentatonic scale and its high and low octave variants, and setting the mode mode as a default configuration;

[0008] Play the MIDI pitch bound to the current note, with the tone selected from the instrumental sound library, dynamically changing the display properties of the note, displaying color mapping and animation effects, and generating vibration signals, vibration intensity and pitch;

[0009] Receive the user's mode switching command, recalculate the note arrangement rules of the grid interaction matrix according to the target mode, update the interface display, record the grid sequence triggered by the user in real time, generate a playable melody data stream, convert the playable melody data stream into MIDI format, and export it to external music production software;

[0010] Based on the pentatonic harmony rule library, it detects dissonant interval combinations in user input in real time, corrects errors through visual warnings and voice prompts, and generates a learning analysis report.

[0011] As a preferred solution of the multimodal interactive immersive music theory learning method of the present invention, wherein: constructing a 6-row × 6-column grid interaction matrix, each grid is bound to a note in the pentatonic scale and its high and low octave variants, and setting the mode mode as the default configuration, including the following steps,

[0012] Create a 6-row 6-column grid interaction matrix, extract the pitch data of Gong, Shang, Jiao, Zhi, Yu and their high and low octave variants from the database, and bind them to the matrix index;

[0013] Using the palace mode as the initial arrangement rule, we can get the default notes for each fret.

[0014] The grid interaction matrix is ​​dynamically filled according to the mode rules, so that the grid interaction matrix is ​​arranged in a cyclic order of 1-2-3-5-6-i, and the high and low octave positions are marked, and the palace mode is started as the default state.

[0015] As a preferred solution of the multimodal interactive immersive music theory learning method of the present invention, wherein: playing the MIDI pitch bound to the current note, the timbre is selected from the instrumental timbre library, including the following steps:

[0016] Detect user interaction events, obtain the coordinates of the triggered note grid, and extract the pitch data of Gong, Shang, Jiao, Zhi, Yu, and their high and low octave variants from the 6×6 note grid interaction matrix based on the note grid;

[0017] Select an instrument sample associated with the current mode from the sound library and convert the note data into MIDI protocol format.

[0018] As a preferred solution of the multimodal interactive immersive music theory learning method of the present invention, wherein: based on dynamically changing the display properties of the note grid, displaying color mapping and animation effects, generating vibration signals, vibration intensity and pitch, the method includes the following steps:

[0019] Analyze the current fret coordinates to obtain the note information of the triggered fret, and generate the corresponding RGB value according to the pentatonic scale rules;

[0020] Calculate the ripple diffusion speed based on the pitch frequency, bind it to the tone grid UI element, and convert the pitch frequency into a vibration intensity curve;

[0021] Use GPU shaders to update color maps and animation effects, generate vibration signals, vibration intensity and pitch.

[0022] As a preferred solution of the multimodal interactive immersive music theory learning method of the present invention, the following steps are included: receiving the user's mode switching instruction, recalculating the note arrangement rules of the grid interaction matrix according to the target mode, updating the interface display, recording the grid sequence triggered by the user in real time, and generating a playable melody data stream.

[0023] Get the mode type selected by the user through UI interaction events, and retrieve the scale arrangement logic of the mode from the scale database;

[0024] Recalculate the notes and octave positions bound to each fret in the fret interaction matrix based on the mode rules;

[0025] Dynamically update the tone grid display content based on the reconstructed tone grid interaction matrix, and record the tone grid sequence and trigger timestamps generated by user interaction in real time;

[0026] Convert the note sequence into a MIDI note event sequence according to the timestamp, and synthesize the MIDI events into a playable melody data stream through the audio engine.

[0027] As a preferred solution of the multimodal interactive immersive music theory learning method of the present invention, wherein: converting the playable melody data stream into MIDI format and exporting it to external music production software includes the following steps:

[0028] Obtain the user-triggered note sequence and timestamp information from the real-time recording module, and convert the pitch, trigger duration, and velocity parameters into a standard instruction sequence;

[0029] Insert MIDI Program Change events to lock the timbre numbers of ethnic instruments, merge the file header and event stream in SMF format, and export to external music production software.

[0030] As a preferred solution of the multimodal interactive immersive music theory learning method of the present invention, the method includes the following steps: based on the pentatonic harmony rule library, detecting the dissonant interval combination in the user input in real time, correcting the error through visual warnings and voice prompts, and generating a learning analysis report.

[0031] Capture the user-triggered note stream and its timestamp data in real time, and load the corresponding harmonic interval rules according to the current mode;

[0032] Compare the note stream input by the user with the rule library, mark the illegal interval combination, and synchronously start visual flashing and voice prompts for the illegal intervals;

[0033] The type, frequency, and context of illegal operations are stored in the error log database, error logs are aggregated, and a learning analysis report is generated.

[0034] In a second aspect, the present invention provides a multimodal interactive immersive music theory learning system, including a grid interaction matrix module, which constructs a 6-row × 6-column grid interaction matrix, binds each grid to a note in the pentatonic scale and its high and low octave variants, and sets the mode mode as the default configuration;

[0035] Debug the dynamic management module, play the MIDI pitch bound to the current note, the tone is selected from the instrumental tone library, based on the dynamic change of the display properties of the note, display color mapping and animation effects, generate vibration signals, vibration intensity and pitch;

[0036] The melody creation center module receives the user's mode switching instructions, recalculates the note arrangement rules of the grid interaction matrix according to the target mode, updates the interface display, records the grid sequence triggered by the user in real time, generates a playable melody data stream, converts the playable melody data stream into MIDI format, and exports it to external music production software;

[0037] The checking module, based on the pentatonic harmony rule library, detects inharmonious interval combinations in user input in real time, corrects errors through visual warnings and voice prompts, and generates a learning analysis report.

[0038] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the multimodal interactive immersive music theory learning method as described in the first aspect of the present invention.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the multimodal interactive immersive music theory learning method as described in the first aspect of the present invention.

[0040] The beneficial effects of the present invention are as follows: by constructing a 6×6 grid interaction matrix and adopting a cyclic permutation algorithm to dynamically bind the pentatonic scale and its octave variants, the limitations of the traditional linear keyboard layout are broken through, and two-dimensional spatial scale visualization cognition is realized, the efficiency of pentatonic scale memory is improved, and a data structure foundation is laid for subsequent multimodal interaction. At the same time, a real-time detection system based on the pentatonic mode harmony rule library recognizes specific cultural interval rules through a hidden Markov model, realizes error correction of national music characteristics, and generates a mode migration training path and an interval heat map analysis report. The grid matrix provides structured input, and the rule library realizes quality control, jointly constructing a complete teaching closed loop from spatial cognition to cultural specific error correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 Flowchart of the immersive music theory learning method based on multimodal interaction.

[0043] Figure 2 Schematic diagram of a multimodal interactive immersive music theory learning method system.

[0044] Figure 3 This is the functional architecture diagram of the system module.

[0045] Figure 4 This is the prototype diagram of the interactive operation interface.

[0046] Figure 5 Flowchart for usage tips.

[0047] Figure 6 A prototype diagram of a data dashboard. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0049] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0050] Secondly, an embodiment or embodiments herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. References to an embodiment in various places in this specification do not necessarily refer to the same embodiment, nor are they separate or selective embodiments that are mutually exclusive of other embodiments.

[0051] Example 1 reference Figures 1 and 2 , is an embodiment of the present invention, which provides a multimodal interactive immersive music theory learning method, comprising the following steps:

[0052] S1. Construct a 6-row × 6-column grid interaction matrix, bind each grid to a note in the pentatonic scale and its high and low octave variants, and set the debug mode as the default configuration.

[0053] S1.1. Create a 6-row 6-column grid interaction matrix, extract the pitch data of Gong, Shang, Jiao, Zhi, Yu, and their high and low octave variants from the database, and bind them to the matrix index.

[0054] Furthermore, when creating a 6-row 6-column grid interaction matrix, a two-dimensional array structure containing 36 cells is first established, each cell is assigned a unique row and column coordinate index, and the MIDI pitch values ​​of the five basic scales of Gong, Shang, Jiao, Zhi, and Yu and their corresponding high octave variants and low octave variants are completely extracted from the preset music database. For example, the fundamental frequency of Gong is MIDI number 60, the high octave Gong is 72, and the low octave Gong is 48; the extracted pitch data is mapped to the matrix index according to the mode arrangement rules. The specific execution process is: under the Gong mode, the first row of the grid interaction matrix is ​​filled with Gong, Shang, Jiao, Zhi, Yu, and high octave Gong in sequence, and the second row starts to cycle this arrangement pattern and superimpose octave changes to ensure that each cell is bound to the accurate pitch value and mode attribute; after the binding is completed, the grid interaction matrix forms an interactive visual interface element, and each cell displays the corresponding scale name and octave identifier, for example, the first row and first column cell displays Gong 1, and the third row and fourth column cell displays Zhi 2.

[0055] S1.2. Use the palace mode as the initial arrangement rule to obtain the default note for each fret.

[0056] Furthermore, when the Gong mode is used as the initial arrangement rule, the Gong mode scale sequence is first determined to be a pentatonic cyclic structure of Gong, Shang, Jiao, Zhi, and Yu; in the 6-row and 6-column grid interaction matrix, the first row is sequentially assigned Gong, Shang, Jiao, Zhi, Yu, and high-octave Gong; the second row is correspondingly assigned Shang, Jiao, Zhi, Yu, high-octave Gong, and high-octave Shang; subsequent rows continue in a cycle according to the same interval relationship and superimpose octave changes; the default note of each grid is obtained by calculating the row and column coordinates, and the specific method is: the row number and The remainder of the sum of the column numbers divided by 5 is 0, remainder 1 corresponds to the gong tone, remainder 2 corresponds to the jiao tone, remainder 3 corresponds to the zhi tone, and remainder 4 corresponds to the yu tone. The fundamental frequency of the shang tone is MIDI number 62, the fundamental frequency of the jiao tone is MIDI number 64, the fundamental frequency of the zhi tone is MIDI number 67, and the fundamental frequency of the yu tone is MIDI number 69. After the allocation is completed, each grid stores a complete set of note attributes, including the scale name, octave identifier and MIDI pitch value. For example, the third row and fifth column grid stores the Yu 1 identifier and MIDI number 69 pitch data.

[0057] S1.3. Dynamically fill the note grid interaction matrix according to the mode rules, arrange the note grid interaction matrix in a cyclic order of 1-2-3-5-6-i, mark the high and low octave positions, and start the palace mode as the default state.

[0058] Furthermore, in the 6-row and 6-column grid interaction matrix, the scale sequence of 1-2-3-5-6 of the Gong mode is cyclically filled (i is the Gong tone variant of the high octave in the simplified musical notation, and the six notes constitute a pentatonic scale within the range of an octave). The Gong tone (MIDI number 60) is assigned to the starting position of the first row and the first column, the Shang tone (MIDI number 62) is assigned to the first row and the second column, the Jiao tone (MIDI number 64) is assigned to the first row and the third column, the Zhi tone (MIDI number 67) is assigned to the first row and the fourth column, the Yu tone (MIDI number 69) is assigned to the first row and the sixth column is assigned to the high octave Gong tone (MIDI number 72); the second row starts with the quotient (MIDI number 62), and the subsequent columns continue in the sequence 1-2-3-5-6, with the high octave quotient (MIDI number 74) assigned to the end of the row; each time you move down a row, the scale sequence maintains the same interval relationship but the overall pitch is raised by one pentatonic degree. When the pitch exceeds MIDI number 84, it automatically switches to a lower octave variant; each grid is explicitly marked with the octave position identifier, the gong fundamental frequency position is marked 1, the high octave gong is marked 1, and the low octave gong is marked 1. After filling is completed, the grid interaction matrix forms a complete pentatonic scale cycle layout, and all grids are in an interactive ready state.

[0059] S2. Play the MIDI pitch bound to the current note, and the timbre is selected from the instrumental timbre library.

[0060] S2.1. Detect user interaction events, obtain the coordinates of the triggered note grid, and extract the pitch data of Gong, Shang, Jiao, Zhi, Yu and their high and low octave variants from the 6×6 note grid interaction matrix according to the note grid.

[0061] Furthermore, when the user touches the 6×6 grid interactive matrix interface, the touch event processing mechanism captures the two-dimensional coordinate value of the click position and converts the screen pixel coordinate into a matrix row and column index. For example, the horizontal coordinate range of 0-599 pixels is divided into 6 equal parts corresponding to column numbers 1-6, and the vertical coordinate range of 0-599 pixels is divided into 6 equal parts corresponding to row numbers 1-6. After locating the specific grid position according to the row and column index, the pre-generated grid data storage structure is accessed. The structure stores 36 grid entities in row priority order, each entity contains a scale name field (Palace name). , Shang, Jiao, Zhi, Yu), octave identification field (fundamental frequency, high octave, low octave) and MIDI pitch value field; the extraction process first reads the scale name field to determine the pentatonic scale category, and then matches the corresponding pitch data in combination with the octave identification field. For example, when the row and column index points to the third row and fourth column, the scale name is extracted as Zhi and the octave identification is the fundamental frequency, and the MIDI number 67 is returned; if the octave identification is the high octave, the pitch value increases by 12, and if it is the low octave, it decreases by 12, ensuring that the extracted pitch data is completely consistent with the current mode configuration of the grid interaction matrix.

[0062] S2.2. Select an instrument sample associated with the current mode from the timbre library and convert the note data into MIDI protocol format.

[0063] Furthermore, a mapping relationship table is established in the preset sound library between five modes (Gong, Shang, Jiao, Zhi, and Yu) and national musical instruments. For example, Gong mode is associated with Guzheng sound samples by default, and Shang mode is associated with Pipa sound samples by default. When it is detected that the current mode type is Gong mode, instrument sampling data of Guzheng sound samples is loaded from the sound library, including a set of PCM waveform segments for 88 key positions (MIDI numbers 21-108). When the extracted note data is converted to MIDI protocol format, a MIDI Note On event is first generated with the status byte set to 0x90, the pitch value written to data byte 1 (for example, Gong fundamental frequency MIDI number 60), and the velocity value written to data byte 2. A MIDI Program Change event is then generated with the status byte set to 0xC0 and the Guzheng sound number written to data byte 1 (for example, GM sound library number 49). The converted MIDI command sequence contains timestamp, event type, channel number, pitch, and velocity parameters, conforms to the standard MIDI file format specification, and can be directly transmitted to the audio synthesis engine for playback.

[0064] S3. Based on the dynamic change of the display properties of the tone grid, the color mapping and animation effects are displayed, and the vibration signal, vibration intensity and pitch are generated.

[0065] S3.1. Analyze the current fret coordinates to obtain the note information of the triggered fret, and generate the corresponding RGB value according to the pentatonic scale rules.

[0066] Furthermore, when the user touches the 6×6 tone grid interaction matrix, the row and column index of the triggered tone grid is obtained by analyzing the touch event coordinates, and the tone grid data storage structure is accessed according to the row and column indexes to read the scale name field (gong, shang, jiao, zhi, yu) and the octave identification field (fundamental frequency, high octave, low octave); based on the pentatonic color mapping rule, gong corresponds to RGB (255,0,0), shang corresponds to RGB (255,255,255), jiao corresponds to RGB (0,255,0), zhi corresponds to RGB (0,255,0), Corresponding to RGB (255,255,0), the yu tone corresponds to RGB (0,0,255); the octave change is achieved by adjusting the color brightness. The high octave grid keeps the hue unchanged but increases the brightness by 50% (for example, the high octave gong tone RGB (255,128,128)), and the low octave grid reduces the brightness by 50% (for example, the low octave gong tone RGB (128,0,0)); the final generated RGB value is directly applied to the fill rendering of the grid interface elements to ensure that the visual feedback strictly corresponds to the scale properties.

[0067] S3.2. Calculate the ripple diffusion speed based on the pitch frequency, bind it to the tone grid UI element, and convert the pitch frequency into a vibration intensity curve.

[0068] Furthermore, when a note grid is triggered, the MIDI pitch value bound to the note grid is first obtained. The ripple diffusion speed is proportional to the frequency value. The baseline diffusion speed is 100 pixels / second, corresponding to a frequency of 261.63Hz. The diffusion speed doubles with each additional octave. The vibration intensity curve uses piecewise linear mapping to divide the frequency value into three intervals: 20-200Hz, 200-800Hz, and 800-2000Hz, corresponding to vibration intensity coefficients of 0.3-1.0, 1.0-2.5, and 2.5-4.0, respectively; the calculated ripple diffusion parameters are directly bound to the animation controller of the note grid UI element, and the vibration intensity coefficient is transmitted to the device haptic engine to form a pitch-driven multi-modal feedback effect.

[0069] S3.3. Use GPU shaders to update color mapping and animation effects, generate vibration signals, vibration intensity and pitch.

[0070] Furthermore, a grid rendering pipeline is established in the GPU shader. The vertex shader receives the grid position coordinates and size parameters, and the fragment shader receives the RGB color value and ripple diffusion parameters (color mapping is achieved through texture sampling in the fragment shader. A pre-generated pentatonic gradient texture map is loaded, and the UV coordinates are calculated according to the scale type and octave position for sampling; the ripple animation effect is implemented using the GPU particle system, which emits particles with the triggering grid as the center. The particle movement speed is controlled by the diffusion speed converted from the pitch frequency, and the particle life cycle is positively correlated with the note length; the vibration signal generation module receives the pitch frequency and intensity coefficient, and decomposes the frequency into a low-frequency band of 10-200Hz and a high-frequency band of 200-2000Hz through FFT transformation. The low-frequency band generates a continuous vibration signal (for example, 100Hz corresponds to a 100ms vibration period), and the high-frequency band generates a short pulse signal.

[0071] S4. Receive the user's mode switching instruction, recalculate the note arrangement rules of the grid interaction matrix according to the target mode, update the interface display, record the grid sequence triggered by the user in real time, and generate a playable melody data stream.

[0072] S4.1. Obtain the mode type selected by the user through UI interaction events, and retrieve the scale arrangement logic of the mode from the scale database.

[0073] Furthermore, when a user clicks the Gong, Shang, Jiao, Zhi, or Yu buttons on the mode selection interface, the touch event handler captures the button ID and converts it into the corresponding mode type enumeration value. The scale database stores the arrangement rules of the five types of modes: Gong mode is recorded as the sequence [1, 2, 3, 5, 6], Shang mode is recorded as the sequence [2, 3, 5, 6, 1], Jiao mode is recorded as the sequence [3, 5, 6, 1, 2], Zhi mode is recorded as the sequence [5, 6, 1, 2, 3], and Yu mode is recorded as the sequence [6, 1, 2, 3, 5]. The database query engine retrieves the corresponding scale sequence array based on the mode type enumeration value and simultaneously obtains the standard MIDI pitch mapping table for each scale (for example, the Shang note in the Shang mode corresponds to MIDI number 62, and the Jiao note corresponds to MIDI number 64). The retrieved scale arrangement logic includes three data groups: scale cycle sequence, tonic position mark, and standard pitch of each note degree, providing complete parameters for the subsequent reconstruction of the grid interaction matrix.

[0074] S4.2. Recalculate the notes and octave positions bound to each fret in the fret interaction matrix based on the mode rules.

[0075] Furthermore, in the reconstruction process of the 6×6 grid interaction matrix, the retrieved mode scale sequence (for example, the Shang mode sequence [2,3,5,6,1]) is first read, and starting from the first row and the first column, the scale name field of each grid is cyclically filled in according to the scale sequence. The first row is filled with Shang tone, Jiao tone, Zhi tone, Yu tone, Gong tone, and high octave Shang tone in sequence; the octave position is calculated using a row and column index weighted algorithm, and each increase in row number by 1 corresponds to a pitch increase of one fifth interval (for example, the Shang tone in the first row is MIDI number 62, and the Shang tone in the second row is MIDI number 69), and the column number is MIDI number 69. Each increase of 1 corresponds to a pitch increase of one degree; when the calculated pitch exceeds MIDI number 84, it automatically switches to a lower octave variant (pitch value minus 12), and when it is lower than MIDI number 48, it switches to a higher octave variant (pitch value plus 12); the octave identification field of each grid is determined by the difference between the final pitch value and the fundamental frequency, with a difference of 0 marked as the fundamental frequency, a positive value marked as a higher octave, and a negative value marked as a lower octave; after filling is completed, all 36 grids in the grid interaction matrix are updated with the scale name, octave identification and accurate MIDI pitch value corresponding to the current mode.

[0076] S4.3. Dynamically update the tone grid display content according to the reconstructed tone grid interaction matrix, and record the tone grid sequence and trigger timestamp generated by user interaction in real time.

[0077] Furthermore, after the tone grid interaction matrix completes the mode reconstruction, the interface rendering engine traverses the 36 tone grid data entities, extracts the scale name field and octave identification field of each tone grid, and combines them to generate display text (for example, Shang 1 represents the fundamental frequency Shang tone, and Yu 2 represents the high octave Yu tone); the color mapping module matches the preset RGB value (Gong = red, Shang = white, Jiao = green, Zhi = yellow, Yu = blue) according to the scale name field, adjusts the brightness in combination with the octave identification field (increase the brightness of the high octave by 50%, and reduce the brightness of the low octave by 50%), and updates the fill color of the tone grid UI element; the user touch event handler records the row and column index, scale name, octave identification and millisecond timestamp of the current tone grid (for example, 02:15:36.458-row 3 column 4-Zhi 1) each time the tone grid is triggered, and stores them in a circular buffer in chronological order; the timestamp data is recorded in the ISO 8601 extended format, including hour, minute, second, millisecond and time zone information, to ensure that the performance timing can be accurately restored when the subsequent melody data stream is generated.

[0078] S4.4. Convert the frame sequence into a MIDI note event sequence according to the timestamps, and synthesize the MIDI events into a playable melody data stream through the audio engine.

[0079] Furthermore, when reading frame trigger records from the ring buffer, all events are first sorted by timestamp, and adjacent events with a time difference of less than 50 milliseconds are merged into a chord. When each frame record is converted into a MIDI note event, the Note On event is written to status byte 0x90, with data byte 1 filled with the pitch value (for example, the gong fundamental frequency MIDI number 60), and data byte 2 filled with the velocity value (default 112). The Note Off event is generated after the Note On event with a delay equal to the time difference between the current frame record and the next event (for example, 300 milliseconds). Status byte 0x80 retains the same pitch value but a velocity value of 0. The audio engine loads a preset timbre sample library and selects the corresponding instrument timbre (for example, guzheng timbre number 49) based on the MIDI Program Change event. The MIDI event sequence is converted into a PCM audio data stream using a wavetable synthesis algorithm. The synthesis process automatically adds partials (for example, superimposing a perfect fifth harmonic pitch on a zhi note). The sampling rate is fixed at 44.1kHz and the bit depth is 16 bits. The output is two-channel stereo waveform data, forming a complete, playable melody data stream.

[0080] S5. Convert the playable melody data stream into MIDI format and export it to external music production software.

[0081] S5.1. Obtain the user-triggered tone frame sequence and timestamp information from the real-time recording module, and convert the pitch, trigger duration, and velocity parameters into a standard instruction sequence.

[0082] Furthermore, when the user interacts with the 6×6 note grid interaction matrix through touch, the real-time recording module captures the note grid trigger event and generates a structured record. Each record contains the note grid row and column index, pitch value, trigger time timestamp, and touch force value. The timestamp processing engine first sorts all records according to the ISO 8601 time format. Adjacent records with a time difference less than a threshold (e.g., 50ms) are classified as chord events. During the conversion process, each note grid record generates a corresponding MIDI NoteOn instruction. The status byte is fixed to 0x90, the pitch value (e.g., Shang tone 62) is written in data byte 1, and the standardized force value is written in data byte 2 (the original touch pressure value is converted to the MIDI standard force range 20-107 through linear mapping). For chord events, multiple Note On instructions are generated synchronously and marked with the same timestamp. The Note Off instruction is generated with a delay after the Note On, and the delay duration is equal to the duration of the note grid. The status byte is 0x80, and the pitch data is the same as the Note Off instruction. On remains consistent but the velocity value is 0; the final output standard MIDI command sequence contains complete time metadata, pitch parameters and velocity dynamic information, which complies with the MIDI2.0 protocol specification.

[0083] S5.2. Insert a MIDI Program Change event to lock the timbre number of the national musical instrument, merge the file header and event stream in SMF format, and export it to external music production software.

[0084] Furthermore, after generating the standard MIDI command sequence, a MIDI Program Change event is first inserted at the start of the track, the status byte is set to 0xC0, and data byte 1 is written with the predefined folk instrument timbre number (for example, guzheng timbre number 49, pipa timbre number 105); the SMF file header is generated according to the MIDI 1.0 specification, including the MThd identifier, header length (fixed value 6), format type (set to 1), number of tracks (set to 1), and time base (set to 480 ticks per beat); the track block starts with the MTrk identifier, and the track length (calculated as the total number of bytes of all events) and the Delta Time encoded MIDI event sequence are written in sequence. When the file is exported, it is encapsulated in binary format and a file with a .mid extension is created through the operating system file API. The content is written in the order of file header → track block → end marker. The final generated SMF format file is transmitted to the project file directory of the external music production software via shared storage or network transmission protocol to ensure that the DAW software can automatically recognize and load the timbre mapping relationship.

[0085] S6. Based on the pentatonic harmony rule library, the system detects inharmonious interval combinations in user input in real time, corrects errors through visual warnings and voice prompts, and generates a learning analysis report.

[0086] S6.1. Capture the note stream triggered by the user and its timestamp data in real time, and load the corresponding harmonic interval rules according to the current mode.

[0087] Furthermore, when the user interacts with the 6×6 grid interaction matrix through touch, the event capture mechanism immediately records the grid row and column coordinates, scale name, octave identifier, MIDI pitch value, and timestamp accurate to milliseconds. The pentatonic harmony rule library stores the interval constraints of various modes. The Gong mode rule prohibits three consecutive minor second intervals, the Shang mode rule requires that the Yu tone must be followed by the Gong tone, and the Jiao mode rule limits the frequency of using the variable Zhi tone. According to the currently activated mode type, the corresponding interval constraint set is extracted from the rule library, including a table of legal interval combinations (for example, Shang-Jiao-Zhi is a legal sequence), a table of taboo intervals, and special rules. The loaded harmonious interval rules and the real-time captured note stream data are input into the interval analysis engine together to provide a judgment basis for subsequent dissonant interval detection.

[0088] S6.2. Compare the note stream input by the user with the rule library, mark the illegal interval combination, and synchronously start visual flashing and voice prompts for the illegal intervals.

[0089] Furthermore, the interval analysis engine uses a sliding window algorithm to scan the note stream input by the user, with a window size of 3 consecutive notes and a sliding step of 1 note each time; the note combination in each window is matched with the forbidden interval table in the current mode rule library, and the visual feedback of the illegal interval uses a red-white alternating flashing mode with a flashing frequency set to 5Hz, and the duration is positively correlated with the degree of violation. The voice prompt content is selected from the pre-recorded audio library, and the corresponding warning sentence is played according to the violation type. The speech synthesis parameters are set to a sampling rate of 16kHz and a bit rate of 128kbps; all marking results are written into the error log database together with the timestamp, violation type, and severity score (level 1-5) to form a complete error correction record chain.

[0090] S6.3. Store the type, frequency, and context of the illegal operation in the error log database, aggregate the error logs, and generate a learning analysis report.

[0091] Furthermore, the error log database adopts a time series storage structure. Each record contains a timestamp accurate to milliseconds, the type of illegal interval, the scale combination involved, the mode environment, and the severity score. The aggregation analysis engine classifies and counts the error records according to the time dimension and mode dimension. The learning analysis report generation module integrates the statistical results. The output contains three parts: error heat map, progress curve and training suggestions. The report format adopts HTML5+CSS3 visualization solution.

[0092] This embodiment also provides a multimodal interactive immersive music theory learning system, including: a grid interaction matrix module, which constructs a 6-row × 6-column grid interaction matrix, binds each grid to a note in the pentatonic scale and its high and low octave variants, and sets the mode mode as the default configuration;

[0093] Debug the dynamic management module, play the MIDI pitch bound to the current note, the tone is selected from the instrumental tone library, based on the dynamic change of the display properties of the note, display color mapping and animation effects, generate vibration signals, vibration intensity and pitch;

[0094] The melody creation center module receives the user's mode switching instructions, recalculates the note arrangement rules of the grid interaction matrix according to the target mode, updates the interface display, records the grid sequence triggered by the user in real time, generates a playable melody data stream, converts the playable melody data stream into MIDI format, and exports it to external music production software;

[0095] The checking module, based on the pentatonic harmony rule library, detects inharmonious interval combinations in user input in real time, corrects errors through visual warnings and voice prompts, and generates a learning analysis report.

[0096] This embodiment also provides a computer device suitable for the immersive music theory learning method with multimodal interaction, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the immersive music theory learning method with multimodal interaction proposed in the above embodiment.

[0097] The computer device may be a terminal, comprising a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse.

[0098] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the immersive music theory learning method for realizing multimodal interaction as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.

[0099] In summary, the present invention breaks through the limitations of the traditional linear keyboard layout by constructing a 6×6 grid interaction matrix and adopting a cyclic permutation algorithm to dynamically bind the pentatonic scale and its octave variants, realizing two-dimensional spatial visualization cognition of scales, improving the memory efficiency of the pentatonic scale, and laying the data structure foundation for subsequent multimodal interaction. At the same time, the real-time detection system based on the pentatonic mode harmony rule library recognizes specific cultural interval rules through the hidden Markov model, realizes error correction of national music characteristics, and generates a mode migration training path and interval heat map analysis report. The grid matrix provides structured input, and the rule library realizes quality control, jointly constructing a complete teaching closed loop from spatial cognition to cultural specificity error correction.

[0100] Example 2, reference Figure 3-Figure 6 , refer to Table 1, refer to Table 2 and refer to Table 3, which are the second embodiment of the present invention. In order to further verify the technical solution of the present invention, experimental simulation data of a multimodal interactive immersive music theory learning method are provided.

[0101] Based on the interaction logic of the Sudoku game interface and the music learning interface, a comparative experiment was constructed.

[0102] Experimental group: Using the patented 6×6 grid matrix (pentatonic Sudoku test module), integrated:

[0103] Dynamic color mapping (multi-color digital distinction)

[0104] Real-time error prompts (error count)

[0105] Multimodal feedback (visual + auditory + tactile)

[0106] Control group: traditional music learning APP

[0107] Key data comparison

[0108]

[0109] The details are shown in Table 1 below:

[0110] Table 1 Comparison of key data

[0111] Typical test scenarios

[0112] Select Mode: Select the Shang mode in the Pentatonic Sudoku test (difficulty level 3)

[0113] Click the tone grid: get it synchronously when triggering the Gong sound;

[0114] Vision: Red flashes (such as Figure 2 color feedback);

[0115] Hearing: Guzheng timbre ( Figure 4 Musical elements);

[0116] Tactile: short vibration (100ms);

[0117] Error handling: when dissonant intervals are triggered continuously;

[0118] Interface display: number of errors + 1 (counter);

[0119] Voice prompts: Pay attention to the Shang mode and avoid changing the zhi tone;

[0120] Innovative data support;

[0121] Multimodal collaborative efficiency

[0122] Comparison of feedback signal transmission delay

[0123] This patent Vision (50ms) + hearing (70ms) + touch (90ms) = parallel processing ≤ 90ms Traditional technology Serial processing cumulative ≥ 300ms

[0124] The details are shown in Table 2 below:

[0125] Table 2 Comparison of feedback signal transmission delay

[0126] Cultural adaptability

[0127] Accuracy of ethnic music feature recognition

[0128] Mode type The recognition rate of this patent Western system recognition rate Palace Mode 99％ 71％ Shang Mode 97％ 65％

[0129] The details are shown in Table 3 below:

[0130] Table 3 Accuracy ratio of ethnic music feature recognition

[0131] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multimodal interactive immersive music theory learning method, characterized by: include, Construct a 6-row × 6-column grid interaction matrix, with each grid bound to a note in the pentatonic scale and its high and low octave variants, and set the debug mode as the default configuration; Play the MIDI pitch bound to the current note, with the tone selected from the instrumental sound library, dynamically changing the display properties of the note, displaying color mapping and animation effects, and generating vibration signals, vibration intensity and pitch; Receive the user's mode switching command, recalculate the note arrangement rules of the grid interaction matrix according to the target mode, update the interface display, record the grid sequence triggered by the user in real time, generate a playable melody data stream, convert the playable melody data stream into MIDI format, and export it to external music production software; Based on the pentatonic harmony rule library, it detects dissonant interval combinations in user input in real time, corrects errors through visual warnings and voice prompts, and generates a learning analysis report.

2. The multimodal interactive immersive music theory learning method according to claim 1, characterized in that: Construct a 6-row × 6-column grid interaction matrix, bind each grid to a note in the pentatonic scale and its high and low octave variants, and set the mode as the default configuration, including the following steps: Create a 6-row 6-column grid interaction matrix, extract the pitch data of Gong, Shang, Jiao, Zhi, Yu and their high and low octave variants from the database, and bind them to the matrix index; Using the palace mode as the initial arrangement rule, we can get the default notes for each fret. The grid interaction matrix is ​​dynamically filled according to the mode rules, so that the grid interaction matrix is ​​arranged in a cyclic order of 1-2-3-5-6-i, and the high and low octave positions are marked, and the palace mode is started as the default state.

3. The multimodal interactive immersive music theory learning method according to claim 2, characterized in that: Play the MIDI pitch bound to the current note, the sound selected from the instrumental sound library, including the following steps, Detect user interaction events, obtain the coordinates of the triggered note grid, and extract the pitch data of Gong, Shang, Jiao, Zhi, Yu, and their high and low octave variants from the 6×6 note grid interaction matrix based on the note grid; Select an instrument sample associated with the current mode from the sound library and convert the note data into MIDI protocol format.

4. The multimodal interactive immersive music theory learning method according to claim 3, characterized in that: Based on the dynamic change of the display properties of the tone grid, the display color mapping and animation effects are displayed, and the vibration signal, vibration intensity and pitch are generated, including the following steps: Analyze the current fret coordinates to obtain the note information of the triggered fret, and generate the corresponding RGB value according to the pentatonic scale rules; Calculate the ripple diffusion speed based on the pitch frequency, bind it to the tone grid UI element, and convert the pitch frequency into a vibration intensity curve; Use GPU shaders to update color maps and animation effects, generate vibration signals, vibration intensity and pitch.

5. The multimodal interactive immersive music theory learning method according to claim 4, characterized in that: Receive the user's mode switching command, recalculate the note arrangement rules of the grid interaction matrix according to the target mode, update the interface display, record the grid sequence triggered by the user in real time, and generate a playable melody data stream. The following steps are included: Get the mode type selected by the user through UI interaction events, and retrieve the scale arrangement logic of the mode from the scale database; Recalculate the notes and octave positions bound to each fret in the fret interaction matrix based on the mode rules; Dynamically update the tone grid display content based on the reconstructed tone grid interaction matrix, and record the tone grid sequence and trigger timestamps generated by user interaction in real time; Convert the note sequence into a MIDI note event sequence according to the timestamp, and synthesize the MIDI events into a playable melody data stream through the audio engine.

6. The multimodal interactive immersive music theory learning method according to claim 5, characterized in that: Converting a playable melody data stream into MIDI format and exporting it to external music production software involves the following steps: Obtain the user-triggered note sequence and timestamp information from the real-time recording module, and convert the pitch, trigger duration, and velocity parameters into a standard instruction sequence; Insert MIDI Program Change events to lock the timbre numbers of ethnic instruments, merge the file header and event stream in SMF format, and export to external music production software.

7. The multimodal interactive immersive music theory learning method according to claim 6, characterized in that: Based on the pentatonic harmony rule library, the disharmonic interval combinations in the user input are detected in real time, and errors are corrected through visual warnings and voice prompts, and a learning analysis report is generated, including the following steps: Capture the user-triggered note stream and its timestamp data in real time, and load the corresponding harmonic interval rules according to the current mode; Compare the note stream input by the user with the rule library, mark the illegal interval combination, and synchronously start visual flashing and voice prompts for the illegal intervals; The type, frequency, and context of illegal operations are stored in the error log database, error logs are aggregated, and a learning analysis report is generated.

8. A multimodal interactive immersive music theory learning system, based on the multimodal interactive immersive music theory learning method according to any one of claims 1 to 7, characterized in that: include, The grid interaction matrix module builds a 6-row × 6-column grid interaction matrix, binds each grid to a note in the pentatonic scale and its high and low octave variants, and sets the mode as the default configuration; Debug the dynamic management module, play the MIDI pitch bound to the current note, the tone is selected from the instrumental tone library, based on the dynamic change of the display properties of the note, display color mapping and animation effects, generate vibration signals, vibration intensity and pitch; The melody creation center module receives the user's mode switching instructions, recalculates the note arrangement rules of the grid interaction matrix according to the target mode, updates the interface display, records the grid sequence triggered by the user in real time, generates a playable melody data stream, converts the playable melody data stream into MIDI format, and exports it to external music production software; The checking module, based on the pentatonic harmony rule library, detects inharmonious interval combinations in user input in real time, corrects errors through visual warnings and voice prompts, and generates a learning analysis report.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the multimodal interactive immersive music theory learning method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal interactive immersive music theory learning method according to any one of claims 1 to 7 are implemented.

Images