Three-dimensional notation coding method and system based on spatio-temporal data fusion

Abstract

The invention discloses a three-dimensional notation coding method and system based on spatio-temporal data fusion, and belongs to the technical field of data fusion, and the method comprises the steps: collecting geographic coordinates and environment voiceprints through Beidou, GPS positioning and COMSOL acoustic modeling, and constructing a sound-space correlation model; through OpenPose and MEMS sensors, limb motions, vocal organ motions and foot stamping intensity parameters are quantified, digital conversion of body memory is completed, and a three-dimensional music score visualization framework is constructed through AR and VR systems; the AR system uses the scanning mark as an interaction entrance, body movement guidance of the Shaanbei folk song and a stage movement design visual interface are matched through a space positioning technology, and dynamic association of a performer and a virtual scene is matched; the VR system generates personalized teaching feedback based on the motion capture data, and optimizes a learning path through real-time data analysis; and the virtual human model inputs action parameters of a three-dimensional notation method to realize automatic performance of non-peoples and folk songs.

Description

Technical Field

[0001] The present invention belongs to the technical field of data fusion, and specifically refers to a three-dimensional notation encoding method and system based on spatiotemporal data fusion. Background Art

[0002] Traditional music notation mainly records pitch and rhythm, which makes it difficult to fully present the multi-dimensional performance characteristics of intangible cultural heritage folk songs;

[0003] However, the existing three-dimensional notation coding system based on spatiotemporal data fusion still has certain defects. The existing system relies on high-precision sensors and complex algorithms, resulting in high system costs and high technical barriers. AR, VR equipment and virtual human technology need to rely on specific hardware, which reduces the applicability of equipment-free scenarios. The standardization of intangible cultural heritage sound database may weaken the diversity of regional characteristics. Therefore, a three-dimensional notation coding method and system based on spatiotemporal data fusion is proposed. Summary of the Invention

[0004] The purpose of the present invention is to provide a three-dimensional notation encoding method and system based on spatiotemporal data fusion to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a three-dimensional notation encoding method based on spatiotemporal data fusion, comprising the following steps:

[0006] S1. Through Beidou, GPS positioning and COMSOL acoustic modeling, we collect geographic coordinates and environmental soundprints to build a "sound-space" correlation model;

[0007] S2. Quantify body movements, vocal organ movements, and stomping intensity parameters using OpenPose and MEMS sensors to complete the digital transformation of body memory.

[0008] S3, based on Fourier transform, calibrates the D-key Kuyin Yanyue scale, quantifies microtonal, laryngeal vibrato and choir-throwing techniques, and generates a timbre template for intangible cultural heritage singing;

[0009] S4. Integrate pitch, physical movement, and geographical parameters to construct a three-axis three-dimensional notation coding system that includes pitch, movement, and spatial mapping;

[0010] S5, integrating 27-dimensional parameters to generate a living conservation model that holographically records the details of folk song performances;

[0011] S6. Through AR, VR systems and virtual human driving technology, a three-dimensional music score visualization framework is constructed, intelligent teaching feedback data is generated, and immersive cultural experience scenes in the metaverse are presented.

[0012] S7. Implement digital museum archiving, cultural tourism immersion experience, and musical instrument acoustic design to protect and industrialize intangible cultural heritage.

[0013] Among them, the S1 obtains the geographical coordinates of the performance venue through Beidou and GPS positioning technology, combines it with environmental soundprint collection, and records environmental parameters; uses COMSOL sound wave reflection modeling technology to simulate the cave reverberation parameters and terrain acoustic effects, and constructs a "sound-space" association model; geographical coordinates and environmental soundprint data are synchronously collected through high-precision sensors, and acoustic modeling generates a three-dimensional spatial sound field distribution based on a physical simulation algorithm, forming a dynamic mapping relationship between geography and sound.

[0014] Among them, the S2 quantifies the performer's body movements at multiple key points through OpenPose skeletal tracking technology, combines with MEMS throat sensors to record the movement parameters of the vocal organs, and quantifies the stomping intensity parameters through pressure sensors, such as light tapping, heavy stomping, and jumping stomping patterns and vibration frequencies, which are synchronized with the rhythm of the waist drum. After the body movements and vocal parameters are aligned through timestamps, a digital body memory of the dynamic action sequence is generated to form a computable action.

[0015] Among them, the S3 accurately calibrates the D-yu-key bitter tone Yanyue scale based on Fourier transform, records the fundamental frequency error, extracts the intangible cultural heritage singing skills through the breathy voice ratio and dialect tone parameters, and combines the pitch change curve of the swing-style technique to generate the intangible cultural heritage singing timbre template; the mapping relationship between the pitch microtones and the Yanyue scale is analyzed by algorithm, and the dynamic characteristics of laryngeal vibrato and swing-style technique are quantified through time-frequency analysis to form a reproducible intangible cultural heritage timbre database.

[0016] Among them, the S4 integrates the pitch dimension, physical dimension, and geographical dimension to construct a three-axis three-dimensional notation coding system; the pitch trajectory is generated through the Fourier transform algorithm, the action sequence is integrated with MEMS sensor data through OpenPose, and the spatial mapping is integrated with Beidou and GPS positioning data through COMSOL modeling; after the three-axis data is aligned by timestamp, a three-dimensional coding framework including pitch, action, and space is formed.

[0017] Among them, the S5 integrates 27-dimensional parameters such as pitch, rhythm, movement, geography, and acoustics to generate a living protection model that holographically records the details of folk song performances; the parameter space is synchronously collected and analyzed by algorithms from multiple sources to construct a complete data set that includes body coordination, geographical soundscape correlation, and dynamic changes in timbre; the model is integrated through time series alignment and spatial mapping algorithms to form a storable and retrievable living folk song archive.

[0018] Among them, the S6 constructs a three-dimensional music score visualization framework through AR and VR systems, combines virtual human driving technology to generate intelligent teaching feedback data, and forms a multi-dimensional cultural experience system; the AR system uses scanning marks as the interactive entrance, synchronizes the body movement guidance and stage movement design of the three-dimensional music score to the visualization interface, and matches the dynamic association between the performer and the virtual scene through spatial positioning technology; the VR system generates personalized teaching feedback based on motion capture data, including body coordination scoring, movement trajectory deviation analysis and rhythm matching evaluation, and optimizes the learning path through real-time data analysis.

[0019] Among them, the S6, virtual human model uses the action parameters of three-dimensional notation as input to drive the digital performance subject in the metaverse scene; the system combines the environmental soundprint with the acoustic characteristics of the virtual space through the geographical soundscape restoration algorithm; the construction of the immersive cultural experience scene relies on the dynamic rendering technology of three-dimensional music scores, synchronizing the pitch, action and geographical parameters to the virtual environment, allowing users to perceive the multi-dimensional characteristics of traditional performance skills in the interaction; this framework deeply integrates teaching feedback and cultural display through the closed-loop design of the data chain, forming a complete chain from skill learning to scene experience.

[0020] Among them, the S7 establishes a three-dimensional music score database through digital museum archives to permanently preserve the performance details of Shaanxi folk songs; in the cultural and tourism immersive experience scene, tourists use AR devices to trigger the three-dimensional music scores of Shaanxi Loess Plateau folk songs, superimpose environmental sound patterns and performance movements, and conduct an immersive cultural experience; through the acoustic design of musical instruments, the construction of the concert hall is optimized by referring to the reverberation parameters of cave dwellings, and ergonomic musical instrument accessories are designed to deeply integrate the protection of intangible cultural heritage with industrial applications.

[0021] Among them, the three-dimensional notation coding system based on spatiotemporal data fusion includes:

[0022] Geographic voiceprint and spatial modeling module: Through Beidou, GPS positioning and COMSOL modeling, it collects geographic coordinates and environmental voiceprints, builds a dynamic "sound-space" mapping relationship, and generates a three-dimensional sound field distribution model;

[0023] Body motion digitization module: This module uses OpenPose skeletal tracking and MEMS sensors to quantify body movements, vocal organ movements, and stomping intensity parameters, generating a digital body memory of dynamic motion sequences.

[0024] Pitch and Timbre Modeling Module: This module calibrates the Yan music scale based on Fourier transform, quantifies microtonal and laryngeal vibrato characteristics, and combines dialect tones and singing techniques to generate a database of intangible cultural heritage singing timbres.

[0025] 3D coding framework construction module: Integrates pitch trajectory, motion sequence and geospatial data, and constructs a three-axis 3D notation coding system including pitch, motion and spatial mapping through timestamp alignment;

[0026] Multi-dimensional parameter fusion module: Fusion of 27-dimensional parameters to generate holographic folk song performance archives, forming a storable and retrievable living protection model;

[0027] 3D visualization and teaching module: Build a visualization framework through AR and VR systems, combine virtual human driving technology to generate intelligent teaching feedback, and support the construction of immersive cultural experience scenes;

[0028] Intangible Cultural Heritage Protection and Application Module: Establish a three-dimensional music score database to support digital museum archiving, cultural tourism AR, VR experience and musical instrument acoustic design, and promote the inheritance and industrial application of intangible cultural heritage.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. This invention utilizes OpenPose skeletal tracking and MEMS throat sensors to quantify limb movements, vocal organ movements, and stomping intensity parameters. Through timestamp alignment technology, it generates digital body memories of dynamic movement sequences, forming computable movements. This method breaks through the ambiguity of traditional oral transmission and converts the body language of intangible cultural heritage performances into precise mathematical parameters. The synchronization analysis of stomping intensity and waist drum rhythm can help optimize the collaborative design of interaction between musical instruments and the human body, improving the standardized inheritance efficiency of traditional skills.

[0031] 2. The present invention uses AR and VR systems to construct a three-dimensional music score visualization framework, combines it with virtual human driving technology to generate intelligent teaching feedback data, and forms a multi-dimensional cultural experience system. The AR system uses scanning marks as the interactive entrance to synchronize the body movement guidance and stage movement design of Shaanxi folk songs to the visualization interface, and uses spatial positioning technology to match the dynamic association between the performer and the virtual scene. The VR system generates personalized teaching feedback based on motion capture data and optimizes the learning path through real-time data analysis. This method breaks through the limitations of oral transmission between teachers and apprentices in traditional teaching and improves the efficiency of musical expression training. After the virtual human model inputs the movement parameters of the three-dimensional notation, it drives the digital performance subject in the metaverse scene to realize the automated performance of intangible cultural heritage folk songs.

[0032] 3. This invention establishes a three-dimensional music score database through digital museum archiving, permanently preserving the performance details of folk songs from northern Shaanxi and solving the problem of information loss in oral transmission. In the cultural tourism immersive experience scenario, tourists use AR devices to trigger the three-dimensional music scores of folk songs from the Loess Plateau of northern Shaanxi, superimposing environmental sound patterns and performance movements to achieve an immersive cultural experience. By referencing the reverberation parameters of cave dwellings in the design of musical instrument acoustics, the construction of concert halls is optimized, and ergonomic musical instrument accessories are designed to promote the deep integration of intangible cultural heritage protection and industrial application. This method provides a complete chain for intangible cultural heritage from digital archiving to commercial implementation, promoting the sustainable dissemination of traditional culture in the context of globalization.

[0033] 4. The present invention calibrates the D-key bitter tone Yanyue scale based on Fourier transform, quantifies microtones, laryngeal vibrato and slinging techniques, and generates a non-legacy singing timbre template; extracts non-legacy singing techniques through breathy voice ratio and dialect tone parameters, and constructs a reproducible non-legacy timbre database based on the pitch change curve of slinging techniques; this method solves the problem that traditional simplified notation cannot mark microtones, and retains the subtle pitch changes of traditional folk songs; the time-frequency analysis and quantification of laryngeal vibrato and slinging techniques enable the non-legacy timbre characteristics to be analyzed and reproduced by the algorithm; the mapping relationship between pitch microtones and the Yanyue scale is analyzed by the algorithm, providing a standardized timbre reference for intelligent teaching software, reducing the difficulty for learners to master complex singing techniques. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The operating process of the present invention based on the three-dimensional notation coding method of spatiotemporal data fusion Figure 1 ;

[0035] Figure 2 The operating process of the present invention based on the three-dimensional notation coding method of spatiotemporal data fusion Figure 2 ;

[0036] Figure 3 It is a structural diagram of the three-dimensional notation coding system based on spatiotemporal data fusion of the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] Example

[0039] See also Figure 1-Figure 3 As shown, the present invention provides a technical solution: comprising the following steps:

[0040] S1. Through Beidou, GPS positioning and COMSOL acoustic modeling, we collect geographic coordinates and environmental soundprints to build a "sound-space" correlation model;

[0041] S2. Quantify body movements, vocal organ movements, and stomping intensity parameters using OpenPose and MEMS sensors to complete the digital transformation of body memory.

[0042] S3, based on Fourier transform, calibrates the D-key Kuyin Yanyue scale, quantifies microtonal, laryngeal vibrato and choir-throwing techniques, and generates a timbre template for intangible cultural heritage singing;

[0043] S4. Integrate pitch, physical movement, and geographical parameters to construct a three-axis three-dimensional notation coding system that includes pitch, movement, and spatial mapping;

[0044] S5, integrating 27-dimensional parameters to generate a living conservation model that holographically records the details of folk song performances;

[0045] S6. Through AR, VR systems and virtual human driving technology, a three-dimensional music score visualization framework is constructed, intelligent teaching feedback data is generated, and immersive cultural experience scenes in the metaverse are presented.

[0046] S7. Implement digital museum archiving, cultural tourism immersion experience, and musical instrument acoustic design to protect and industrialize intangible cultural heritage.

[0047] Among them, the S1 obtains the geographical coordinates of the performance venue through Beidou and GPS positioning technology, combines it with environmental soundprint collection, and records environmental parameters; uses COMSOL sound wave reflection modeling technology to simulate the cave reverberation parameters and terrain acoustic effects, and constructs a "sound-space" association model; geographical coordinates and environmental soundprint data are synchronously collected through high-precision sensors, and acoustic modeling generates a three-dimensional spatial sound field distribution based on a physical simulation algorithm, forming a dynamic mapping relationship between geography and sound.

[0048] Among them, the S2 quantifies the performer's body movements at multiple key points through OpenPose skeletal tracking technology, combines with MEMS throat sensors to record the movement parameters of the vocal organs, and quantifies the stomping intensity parameters through pressure sensors, such as light tapping, heavy stomping, and jumping stomping patterns and vibration frequencies, which are synchronized with the rhythm of the waist drum. After the body movements and vocal parameters are aligned through timestamps, a digital body memory of the dynamic action sequence is generated to form a computable action.

[0049] Among them, the S3 accurately calibrates the D-yu-key bitter tone Yanyue scale based on Fourier transform, records the fundamental frequency error, extracts the intangible cultural heritage singing skills through the breathy voice ratio and dialect tone parameters, and combines the pitch change curve of the swing-style technique to generate the intangible cultural heritage singing timbre template; the mapping relationship between the pitch microtones and the Yanyue scale is analyzed by algorithm, and the dynamic characteristics of laryngeal vibrato and swing-style technique are quantified through time-frequency analysis to form a reproducible intangible cultural heritage timbre database.

[0050] Among them, the S4 integrates the pitch dimension, physical dimension, and geographical dimension to construct a three-axis three-dimensional notation coding system; the pitch trajectory is generated through the Fourier transform algorithm, the action sequence is integrated with MEMS sensor data through OpenPose, and the spatial mapping is integrated with Beidou and GPS positioning data through COMSOL modeling; after the three-axis data is aligned by timestamp, a three-dimensional coding framework including pitch, action, and space is formed.

[0051] Among them, the S5 integrates 27-dimensional parameters such as pitch, rhythm, movement, geography, and acoustics to generate a living protection model that holographically records the details of folk song performances; the parameter space is synchronously collected and analyzed by algorithms from multiple sources to construct a complete data set that includes body coordination, geographical soundscape correlation, and dynamic changes in timbre; the model is integrated through time series alignment and spatial mapping algorithms to form a storable and retrievable living folk song archive.

[0052] Among them, the S6 constructs a three-dimensional music score visualization framework through AR and VR systems, combines virtual human driving technology to generate intelligent teaching feedback data, and forms a multi-dimensional cultural experience system; the AR system uses scanning marks as the interactive entrance, synchronizes the body movement guidance and stage movement design of the three-dimensional music score to the visualization interface, and matches the dynamic association between the performer and the virtual scene through spatial positioning technology; the VR system generates personalized teaching feedback based on motion capture data, including body coordination scoring, movement trajectory deviation analysis and rhythm matching evaluation, and optimizes the learning path through real-time data analysis.

[0053] Among them, the S6, virtual human model uses the action parameters of three-dimensional notation as input to drive the digital performance subject in the metaverse scene; the system combines the environmental soundprint with the acoustic characteristics of the virtual space through the geographical soundscape restoration algorithm; the construction of the immersive cultural experience scene relies on the dynamic rendering technology of three-dimensional music scores, synchronizing the pitch, action and geographical parameters to the virtual environment, allowing users to perceive the multi-dimensional characteristics of traditional performance skills in the interaction; this framework deeply integrates teaching feedback and cultural display through the closed-loop design of the data chain, forming a complete chain from skill learning to scene experience.

[0054] Among them, the S7 establishes a three-dimensional music score database through digital museum archives to permanently preserve the performance details of Shaanxi folk songs; in the cultural and tourism immersive experience scene, tourists use AR devices to trigger the three-dimensional music scores of Shaanxi Loess Plateau folk songs, superimpose environmental sound patterns and performance movements, and conduct an immersive cultural experience; through the acoustic design of musical instruments, the construction of the concert hall is optimized by referring to the reverberation parameters of cave dwellings, and ergonomic musical instrument accessories are designed to deeply integrate the protection of intangible cultural heritage with industrial applications.

[0055] Among them, the three-dimensional notation coding system based on spatiotemporal data fusion includes:

[0056] Geographic voiceprint and spatial modeling module: Through Beidou, GPS positioning and COMSOL modeling, it collects geographic coordinates and environmental voiceprints, builds a dynamic "sound-space" mapping relationship, and generates a three-dimensional sound field distribution model;

[0057] Body motion digitization module: This module uses OpenPose skeletal tracking and MEMS sensors to quantify body movements, vocal organ movements, and stomping intensity parameters, generating a digital body memory of dynamic motion sequences.

[0058] Pitch and Timbre Modeling Module: This module calibrates the Yan music scale based on Fourier transform, quantifies microtonal and laryngeal vibrato characteristics, and combines dialect tones and singing techniques to generate a database of intangible cultural heritage singing timbres.

[0059] 3D coding framework construction module: Integrates pitch trajectory, motion sequence and geospatial data, and constructs a three-axis 3D notation coding system including pitch, motion and spatial mapping through timestamp alignment;

[0060] Multi-dimensional parameter fusion module: Fusion of 27-dimensional parameters to generate holographic folk song performance archives, forming a storable and retrievable living protection model;

[0061] 3D visualization and teaching module: Build a visualization framework through AR and VR systems, combine virtual human driving technology to generate intelligent teaching feedback, and support the construction of immersive cultural experience scenes;

[0062] Intangible Cultural Heritage Protection and Application Module: Establish a three-dimensional music score database to support digital museum archiving, cultural tourism AR, VR experience and musical instrument acoustic design, and promote the inheritance and industrial application of intangible cultural heritage.

[0063] Working principle: Through multi-source data collection and algorithm analysis, the holographic digital protection and dissemination of intangible cultural heritage folk songs are realized; the system first uses Beidou, GPS positioning and COMSOL modeling technology to collect geographic coordinates and environmental soundprint data, build a "sound-space" dynamic mapping relationship, and generate a three-dimensional sound field distribution model; synchronously combine OpenPose skeletal tracking and MEMS sensors to quantify limb movements, vocal organ movements and stomping intensity parameters, and generate digital body memories of dynamic movement sequences; based on Fourier transform, the Yan music scale is calibrated, microtones, laryngeal vibrato and choir techniques are extracted to form an intangible cultural heritage singing timbre database; by integrating pitch trajectories, movement sequences and geographic space data, a three-axis three-dimensional notation coding system including pitch, movement and space mapping is constructed, and 27-dimensional parameters are integrated to generate a holographic folk song performance archive; finally, through AR, VR systems and virtual human driving technology, a three-dimensional music score visualization framework and intelligent teaching feedback mechanism are constructed, combined with digital museum archiving, cultural tourism immersive experience and musical instrument acoustic design to promote the protection and industrial application of intangible cultural heritage.

[0064] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0065] The present invention and its embodiments are described above. This description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if a person skilled in the art is inspired by this and, without departing from the purpose of the present invention, designs structures and embodiments similar to this technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. A three-dimensional notation encoding method based on spatiotemporal data fusion, characterized in that: The following steps are involved: S1. Through Beidou, GPS positioning and COMSOL acoustic modeling, we collect geographic coordinates and environmental soundprints to build a "sound-space" correlation model; S2. Quantify body movements, vocal organ movements, and stomping intensity parameters using OpenPose and MEMS sensors to complete the digital transformation of body memory. S3, based on Fourier transform, calibrates the D-key Kuyin Yanyue scale, quantifies microtonal, laryngeal vibrato and choir-throwing techniques, and generates a timbre template for intangible cultural heritage singing; S4. Integrate pitch, physical movement, and geographical parameters to construct a three-axis three-dimensional notation coding system that includes pitch, movement, and spatial mapping; S5, integrating 27-dimensional parameters to generate a living conservation model that holographically records the details of folk song performances; S6. Through AR, VR systems and virtual human driving technology, a three-dimensional music score visualization framework is constructed, intelligent teaching feedback data is generated, and immersive cultural experience scenes in the metaverse are presented. S7. Implement digital museum archiving, cultural tourism immersion experience, and musical instrument acoustic design to protect and industrialize intangible cultural heritage.

2. The three-dimensional notation encoding method based on spatiotemporal data fusion according to claim 1, characterized in that: The S1 uses Beidou and GPS positioning technologies to obtain the geographic coordinates of the performance venue, combines them with environmental soundprint collection, and records environmental parameters; uses COMSOL sound wave reflection modeling technology to simulate cave reverberation parameters and terrain acoustic effects, and constructs a "sound-space" association model; geographic coordinates and environmental soundprint data are synchronously collected through high-precision sensors, and acoustic modeling generates a three-dimensional spatial sound field distribution based on a physical simulation algorithm, forming a dynamic mapping relationship between geography and sound.

3. The three-dimensional notation encoding method based on spatiotemporal data fusion according to claim 1, characterized in that: The S2 uses OpenPose skeletal tracking technology to quantify the performer's body movements at multiple key points, combined with a MEMS throat sensor to record the motion parameters of the vocal organs. The stomping intensity parameters are quantified through a pressure sensor, including light tapping, heavy stomping, and jumping stomping patterns and vibration frequencies, which are synchronized with the rhythm of the waist drum. After the body movements and vocal parameters are aligned through timestamps, a digital body memory of the dynamic action sequence is generated, forming a computable action.

4. The three-dimensional notation encoding method based on spatiotemporal data fusion according to claim 1, characterized in that: The S3 accurately calibrates the D-yu-key bitter-tone Yanyue scale based on Fourier transform, records the fundamental frequency error, extracts the intangible cultural heritage singing skills through the breathy voice ratio and dialect tone parameters, and generates the intangible cultural heritage singing timbre template in combination with the pitch change curve of the swing-style technique; the mapping relationship between the pitch microtones and the Yanyue scale is analyzed by an algorithm, and the dynamic characteristics of the laryngeal vibrato and the swing-style technique are quantified through time-frequency analysis to form a reproducible intangible cultural heritage timbre database.

5. The three-dimensional notation encoding method based on spatiotemporal data fusion according to claim 1, characterized in that: The S4 integrates the pitch dimension, physical dimension, and geographical dimension to construct a three-axis three-dimensional notation coding system; the pitch trajectory is generated through the Fourier transform algorithm, the action sequence is integrated with MEMS sensor data through OpenPose, and the spatial mapping is integrated with Beidou and GPS positioning data through COMSOL modeling; after the three-axis data are aligned by timestamp, a three-dimensional coding framework that includes pitch, action, and space is formed.

6. The three-dimensional notation encoding method based on spatiotemporal data fusion according to claim 1, characterized in that: The S5 integrates 27 parameters including pitch, rhythm, movement, geography, and acoustics to generate a living conservation model that holographically records the details of folk song performances. The parameter space is analyzed through simultaneous multi-source data collection and algorithm analysis to construct a complete data set that includes body coordination, geographical soundscape correlation, and dynamic changes in timbre. The model is integrated through time series alignment and spatial mapping algorithms to form a storable and retrievable living folk song archive.

7. The three-dimensional notation encoding method based on spatiotemporal data fusion according to claim 1, characterized in that: The S6 constructs a three-dimensional music score visualization framework through AR and VR systems, and combines virtual human driving technology to generate intelligent teaching feedback data, forming a multi-dimensional cultural experience system; the AR system uses scanning marks as the interactive entrance, synchronizes the body movement guidance and stage movement design of the three-dimensional music score to the visualization interface, and matches the dynamic association between the performer and the virtual scene through spatial positioning technology; the VR system generates personalized teaching feedback based on motion capture data, including body coordination scoring, movement trajectory deviation analysis and rhythm matching assessment, and optimizes the learning path through real-time data analysis.

8. The three-dimensional notation encoding method based on spatiotemporal data fusion according to claim 1, characterized in that: In S6, the virtual human model uses the movement parameters of the three-dimensional notation as input to drive the digital performance subject in the metaverse scene; the system combines the environmental soundprint with the acoustic characteristics of the virtual space through the geographical soundscape restoration algorithm; the construction of the immersive cultural experience scene relies on the dynamic rendering technology of the three-dimensional musical score, synchronizing the pitch, movement and geographical parameters to the virtual environment; this framework deeply integrates teaching feedback and cultural display through the closed-loop design of the data chain.

9. The three-dimensional notation encoding method based on spatiotemporal data fusion according to claim 1, characterized in that: The S7 establishes a three-dimensional music score database through digital museum archives to permanently preserve the performance details of Shaanxi folk songs; in the cultural and tourism immersive experience scene, tourists use AR devices to trigger the three-dimensional music scores of Shaanxi Loess Plateau folk songs, superimpose environmental sound patterns and performance movements, and conduct an immersive cultural experience; through the acoustic design of musical instruments, the construction of the concert hall is optimized with reference to the reverberation parameters of cave dwellings, and ergonomic musical instrument accessories are designed to achieve a deep integration of intangible cultural heritage protection and industrial application.

10. A three-dimensional notation coding system based on spatiotemporal data fusion, characterized by: Geographic voiceprint and spatial modeling module: Through Beidou, GPS positioning and COMSOL modeling, it collects geographic coordinates and environmental voiceprints, builds a dynamic "sound-space" mapping relationship, and generates a three-dimensional sound field distribution model; Body motion digitization module: This module uses OpenPose skeletal tracking and MEMS sensors to quantify body movements, vocal organ movements, and stomping intensity parameters, generating a digital body memory of dynamic motion sequences. Pitch and Timbre Modeling Module: This module calibrates the Yan music scale based on Fourier transform, quantifies microtonal and laryngeal vibrato characteristics, and combines dialect tones and singing techniques to generate a database of intangible cultural heritage singing timbres. 3D coding framework construction module: Integrates pitch trajectory, motion sequence and geospatial data, and constructs a three-axis 3D notation coding system including pitch, motion and spatial mapping through timestamp alignment; Multi-dimensional parameter fusion module: Fusion of 27-dimensional parameters to generate holographic folk song performance archives, forming a storable and retrievable living protection model; 3D visualization and teaching module: Build a visualization framework through AR and VR systems, combine virtual human driving technology to generate intelligent teaching feedback, and support the construction of immersive cultural experience scenes; Intangible Cultural Heritage Protection and Application Module: Establish a three-dimensional music score database to support digital museum archiving, cultural tourism AR, VR experience and musical instrument acoustic design, and promote the inheritance and industrial application of intangible cultural heritage.

Images