3D-based traditional instrumental interactive three-dimensional teaching resource generation system

Abstract

The present application relates to the technical field of instrumental music teaching resources, in particular to a traditional instrumental music interactive three-dimensional teaching resource generation system based on 3D, comprising: a system acquires a traditional instrumental music two-dimensional image set, depth point cloud data and physical material attribute data through a data acquisition module. A model reconstruction module uses the two-dimensional image set and the depth point cloud data to run a three-dimensional reconstruction algorithm to generate a traditional instrumental music basic three-dimensional geometric model. A model dynamic module uses an improved structure and morphing animation generation algorithm, combines entity structure characteristics and performance physical laws to optimize animation skeletons and morphing parameters, and generates a dynamic three-dimensional instrumental music model with movable joints and morphing parameters. A rendering generation module renders the physical material attribute data to the dynamic three-dimensional instrumental music model to generate an interactive three-dimensional teaching resource with a realistic appearance.

Description

Technical Field

[0001] This invention relates to the field of instrumental music teaching resources, and in particular to a 3D-based interactive 3D teaching resource generation system for traditional instrumental music. Background Technology

[0002] Traditional instrumental music teaching relies heavily on single teaching methods such as two-dimensional graphics and static images, lacking standardized, multi-dimensional methods for collecting information about the physical components of traditional instruments. Conventional methods within the industry only record single visual image data, failing to simultaneously collect 2D image sets, depth point cloud data, and physical material property data. This fails to fully preserve the spatial structure, depth information, and inherent material parameters of the instrument. Relying solely on a single image for 3D model reconstruction, without incorporating depth point cloud data into the modeling calculations, results in 3D geometric models with blurred structural details, low spatial 3D structural accuracy, and insufficient completeness in physical reconstruction.

[0003] Existing 3D instrumental models are mostly static, holistic models, lacking the ability to adapt to dynamic performance movements. There is a lack of algorithms for generating animations based on the structure and deformation of traditional instruments. It is impossible to optimize the animation skeleton layout and deformation parameter configuration according to the instrument's physical structure characteristics and the physical laws of performance, making it difficult to construct dynamic models with movable joints and controllable deformation parameters, and thus unable to reproduce the changes in motion structure during instrumental performance.

[0004] Conventional 3D resource creation only involves simple material mapping, failing to accurately map and render the real physical material properties onto the dynamic model surface. This results in a significant difference in the model's appearance and texture compared to the actual musical instrument. The completed 3D model lacks interactive functionality, cannot support multi-angle structural viewing, joint movement demonstrations, and other teaching needs, and is difficult to adapt to the standards for the creation and application of traditional digital and interactive 3D teaching resources for musical instruments. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a 3D-based interactive three-dimensional teaching resource generation system for traditional instrumental music.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a 3D-based interactive three-dimensional teaching resource generation system for traditional instrumental music, comprising: The data acquisition module acquires real instrumental data of the target traditional instrument, which includes a set of two-dimensional images, depth point cloud data, and physical material attribute data of the target traditional instrument. The model reconstruction module performs a three-dimensional reconstruction algorithm on the two-dimensional image set and the depth point cloud data to generate a basic three-dimensional geometric model of the target traditional musical instrument. The model dynamization module executes an improved structure and deformation animation generation algorithm on the basic three-dimensional geometric model. The improved structure and deformation animation generation algorithm optimizes the animation skeleton and deformation parameters based on the physical structure characteristics and performance physics of the target traditional musical instrument, thereby generating a dynamic three-dimensional musical instrument model with movable joints and deformation parameters. The rendering generation module renders the physical material attribute data onto the dynamic 3D instrument model to generate interactive 3D teaching resources with a realistic appearance.

[0007] As a further aspect of the present invention, a three-dimensional reconstruction algorithm is performed on the two-dimensional image set and the depth point cloud data to generate a basic three-dimensional geometric model of the target traditional musical instrument, including: Multi-view feature point matching and dense reconstruction are performed on the two-dimensional image set to generate a color three-dimensional point cloud that is aligned with the scale of the depth point cloud data; The color 3D point cloud and the depth point cloud data are fused and denoised to form complete point cloud data of the target traditional musical instrument. Poisson surface reconstruction is performed on the point cloud data to generate a mesh model containing topological connections; The mesh model is simplified, holes are repaired, and smoothing is performed to obtain the basic three-dimensional geometric model with a complete surface and accurate structure.

[0008] As a further aspect of the present invention, the mesh model is simplified, hole repaired, and smoothed, including: simplifying the mesh using an edge folding algorithm based on a quadratic error metric, and completing the surface of the hole region using the Poisson equation.

[0009] As a further aspect of the present invention, an improved structure and deformation animation generation algorithm is applied to the basic three-dimensional geometric model. This improved algorithm optimizes the animation skeleton and deformation parameters based on the physical structural characteristics and performance physics of the target traditional musical instrument, thereby generating a dynamic three-dimensional musical instrument model with movable joints and deformation parameters, including: Based on the prior knowledge of the structure of the target traditional musical instrument, a series of key feature points are defined on the basic three-dimensional geometric model. These key feature points correspond to the physical connection points and movable points between the instrument components. Based on the key feature points, a hierarchical animation skeleton structure is constructed inside the basic three-dimensional geometric model. The animation skeleton structure consists of a root node, multiple joint nodes, and corresponding bone chains, which are used to drive the movement of the basic three-dimensional geometric model. Based on the physical laws of performance, motion constraint parameters are set for each joint node in the animation skeleton structure. The motion constraint parameters include the range of degrees of freedom of joint rotation, the speed limit of movement, and the linkage relationship between joints. Using a physics simulation engine, physical properties are set for the basic three-dimensional geometric model, including the mass, elasticity, and damping of the components, and the deformation of the surface mesh of the basic three-dimensional geometric model is calculated when external forces or joint movements are applied. The animated skeleton structure is bound to the calculated deformation data to generate the dynamic three-dimensional instrumental model. This model can respond to external interactive commands and perform animation demonstrations based on the motion constraint parameters and physical properties.

[0010] As a further aspect of the present invention, based on the aforementioned physical laws of performance, motion constraint parameters are set for each joint node in the animation skeleton structure, including: Collect multiple sets of motion capture data of the target traditional musical instrument under standard performance techniques. The motion capture data records the real-time three-dimensional position and rotation angle of each component of the instrument during performance; The kinematic features of each movable joint are extracted from the motion capture data, including rotation axis, maximum and minimum angle, and angular velocity range; The correlation between the movements of different joints in the motion capture data is analyzed, and the motion linkage rules between joints are defined. The motion linkage rules describe the quantitative proportion and time delay of the driven joint moving when the driven joint moves. The extracted kinematic features and the defined motion linkage rules are configured together as the motion constraint parameters of the corresponding joint nodes in the animation skeleton structure.

[0011] As a further aspect of the present invention, the kinematic features of each movable joint are extracted from the motion capture data, including: determining the main rotation axis of the joint by principal component analysis, and calculating the mean and variance of the rotation angle using statistical methods to determine its range of motion.

[0012] As a further aspect of the present invention, rendering the physical material attribute data onto the dynamic three-dimensional instrumental model includes: The physical material property data includes high-resolution texture maps, normal maps, roughness maps, and metallicity maps collected from the surface of real musical instruments; The high-resolution texture map is mapped onto the corresponding surface of the dynamic 3D instrument model to determine its base color and pattern; The normal map is applied to the dynamic 3D instrument model to simulate the changes in light and shadow caused by the microscopic geometric details of its surface; The roughness map and metallicity map are respectively assigned to different material regions of the dynamic 3D instrument model to define the specular and diffuse reflection characteristics of its surface to light. In a real-time rendering environment, the physically based rendering pipeline calculates and outputs the pixel color of each frame of the dynamic 3D instrumental model based on all the textures, ambient lighting information and viewing angle, thereby generating teaching resource images with a realistic appearance.

[0013] As a further aspect of the present invention, the improved structure and deformation animation generation algorithm also includes the generation of teaching interactive action sequences: Receive teaching instruction input, wherein the teaching instruction specifies the name of the performance technique to be demonstrated; The preset performance technique movement library is queried. The performance technique movement library stores standard joint movement trajectory data and deformation parameter change sequences corresponding to the name of the performance technique. Retrieve standard joint motion trajectory data and deformation parameter change sequences that match the teaching instructions from the performance technique motion library; The standard joint motion trajectory data and deformation parameter change sequence are invoked to drive the animation skeleton structure and surface mesh of the dynamic three-dimensional instrumental model, generating a continuous and smooth sequence of teaching interactive actions.

[0014] As a further aspect of the present invention, the process of constructing the performance technique movement library includes: Record data from inertial measurement units attached to key parts of the instrument when a professional performer operates a real instrument, or capture three-dimensional motion data of marked points through an optical motion capture system. The recorded raw motion data is cleaned, filtered, and redirected, and then adapted to the standardized animation skeleton structure to generate standardized joint rotation data. During the performance, local deformation data of the instrument surface, collected by a high-speed camera or tactile sensor, is recorded synchronously and aligned with the standardized joint rotation data on the timeline. The standardized joint rotation data and the aligned local deformation data of the instrument surface are packaged together and associated with the performance technique tag, and stored as a complete technique record in the performance technique action library.

[0015] As a further aspect of the present invention, after generating interactive 3D teaching resources with a realistic appearance, it further includes: An interactive function integration module adds interactive response logic to different components of the dynamic three-dimensional instrumental model. The interactive response logic defines the animation or state change that the corresponding component of the dynamic three-dimensional instrumental model should execute when the user triggers an interactive event through an input device. Multiple interactive hotspot areas are set on the dynamic three-dimensional instrument model, and each hotspot area is associated with corresponding knowledge explanation content; The dynamic 3D instrument model, the interactive response logic, the hotspot areas, and the associated knowledge explanations are encapsulated into a stand-alone interactive 3D application or a 3D scene file that can be embedded in a webpage; The interactive response logic includes component decomposition and display logic and simulated performance response logic: The component decomposition and display logic is configured to respond to the user's separation command for the overall instrument model, control the specified joint nodes in the animation skeleton structure to move according to the preset separation path and animation duration, so that the specified instrument component is separated from the overall model and hovers, while highlighting the component. The simulated performance response logic is configured to respond to the user's touch or click command on the virtual performance interface, map the command to a drive signal for a specific part of the dynamic three-dimensional instrument model, the drive signal triggers the dynamic three-dimensional instrument model to perform corresponding joint movements and surface deformations, and calls the audio engine to play pre-recorded or real-time synthesized sound corresponding to the performance action.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are as follows: Simultaneously, a set of two-dimensional images of traditional musical instruments, depth point cloud data, and physical material attribute data are collected. Based on the two-dimensional image set and the depth point cloud data, a three-dimensional reconstruction algorithm is run to construct the basic three-dimensional geometric model of the traditional musical instrument. The combination of multiple data types supplements the shape contour and spatial depth information, compensating for the lack of spatial dimension parameters in single-image modeling. This improves the overall structure and detail of the basic model, ensuring that the model's shape conforms to the structural dimensions and shape characteristics of the actual traditional musical instrument, providing a regular and reliable model base for subsequent dynamic processing and rendering.

[0017] An improved structural and deformation animation generation algorithm is applied to the basic 3D geometric model. Based on the structural characteristics of traditional musical instruments and the physical laws of performance, the arrangement of the animation skeleton and deformation parameters are optimized and adjusted to generate a dynamic 3D musical instrument model with movable joints and deformation parameters. The range of motion of the joints and the standard of body deformation are set according to the instrument's own structural logic and actual movement rules, so that the static model has the basic conditions for structural movement and shape change, and is adapted to the dynamic operation logic required for performance simulation.

[0018] The collected physical material attribute data is fully rendered onto the surface of a dynamic 3D instrumental model, restoring the material texture and appearance of traditional instruments. Based on real material parameters, the model's appearance is meticulously replicated, ensuring the physical properties of the model surface closely resemble the actual instrument. This creates a 3D teaching resource that supports free-view browsing, structural disassembly demonstrations, and dynamic motion simulations. The complete preservation of form, dynamic attributes, and material characteristics meets the resource usage needs for self-viewing, structural learning, and motion observation in the digital teaching of traditional instruments. Attached Figure Description

[0019] Figure 1 This is a sequence diagram of the traditional instrumental music interactive three-dimensional teaching resource generation system based on 3D as described in this invention; Figure 2 A flowchart generated from a basic 3D geometric model; Figure 3 A flowchart for setting motion constraint parameters and rendering physical materials. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0022] See Figure 1 This invention provides a 3D-based interactive 3D teaching resource generation system for traditional musical instruments, the overall implementation of which is as follows: The data acquisition module acquires real instrumental data of the target traditional instrument, including a set of two-dimensional images, depth point cloud data, and physical material attribute data of the target traditional instrument. The model reconstruction module performs a three-dimensional reconstruction algorithm on the set of two-dimensional images and the depth point cloud data to generate a basic three-dimensional geometric model of the target traditional instrument. The model dynamization module performs an improved structure and deformation animation generation algorithm on the basic three-dimensional geometric model. The improved structure and deformation animation generation algorithm optimizes the animation skeleton and deformation parameters based on the physical structural characteristics and performance physics of the target traditional instrument, thereby generating a dynamic three-dimensional instrumental model with movable joints and deformation parameters. The rendering generation module renders the physical material attribute data onto the dynamic three-dimensional instrumental model to generate an interactive three-dimensional teaching resource with a realistic appearance.

[0023] In one embodiment of the present invention, see [reference] Figure 2 The process involves: performing multi-view feature point matching and dense reconstruction on the two-dimensional image set to generate a colored three-dimensional point cloud aligned with the scale of the depth point cloud data; fusing and denoising the colored three-dimensional point cloud with the depth point cloud data to form complete point cloud data for the target traditional musical instrument; performing Poisson surface reconstruction on the point cloud data to generate a mesh model containing topological connections; and performing mesh simplification, hole repair, and smoothing on the mesh model to obtain a basic three-dimensional geometric model with a complete surface and accurate structure. When performing mesh simplification, hole repair, and smoothing on the mesh model, an edge folding algorithm based on a quadratic error metric is used for mesh simplification, and the Poisson equation is used for surface completion of the hole regions.

[0024] In the specific implementation, the guqin, a traditional Chinese musical instrument, is used as the target instrument. The data acquisition module obtains a set of two-dimensional images, depth point cloud data, and physical material property data of the guqin. The model reconstruction module performs a three-dimensional reconstruction algorithm on the 2D image set and depth point cloud data of the guqin to generate a basic three-dimensional geometric model of the guqin. Specifically, the model reconstruction module performs multi-view feature point matching and dense reconstruction on the 2D image set of the guqin to generate a colored 3D point cloud that is scaled to the depth point cloud data of the guqin. In the specific implementation, scale-invariant feature transformation feature points in images of the guqin from different perspectives are extracted and matched using a multi-view stereo vision algorithm. The camera parameters and 3D point coordinates are optimized using bundle adjustment to generate a dense colored 3D point cloud. The model reconstruction module fuses and denoises the colored 3D point cloud of the guqin with the depth point cloud data of the guqin to form complete point cloud data of the guqin. The fusion process uses voxel mesh filtering to unify the two point cloud data into the same coordinate system and uses statistical filtering to remove isolated outliers. Poisson surface reconstruction is performed on the point cloud data of the guqin to generate a mesh model containing topological connections. Poisson surface reconstruction converts the point cloud data of the guqin into a continuous triangular mesh surface by solving the Poisson equation. This triangular mesh surface has complete vertex connectivity.

[0025] In some embodiments, the mesh model of the guqin is simplified, holes are repaired, and smoothed to obtain a basic three-dimensional geometric model of the guqin with a complete surface and accurate structure. Specifically, mesh simplification is performed using an edge folding algorithm based on quadratic error metric. This algorithm calculates the quadratic error metric value generated by each edge after folding to the new vertex position, and folds edges sequentially in ascending order of error metric value, reducing the number of triangular faces while maintaining the overall shape characteristics of the guqin model. Hole repair utilizes the Poisson equation to complete the curved surface of the hole region. For the hole boundaries caused by reconstruction defects in the guqin mesh model, the vertices of the hole boundaries are extracted, and then boundary polygons are constructed. The hole region is projected onto a two-dimensional plane for triangulation, and then the triangulated vertices are mapped back to three-dimensional space. The optimal position of the vertices inside the hole region is solved using the Poisson equation, ensuring the curvature continuity between the completed surface and the surrounding mesh. Smoothing uses the Laplace smoothing algorithm to iteratively adjust the vertex positions of the guqin mesh model, eliminating noise and jagged edges generated during reconstruction.

[0026] Optionally, during the mesh simplification process using an edge folding algorithm based on a quadratic error metric, a weight is assigned to each vertex to preserve the geometric details of fine feature areas on the surface of the guqin, such as strings and frets. The weights are set according to the curvature of the region where the vertex is located; regions with greater curvature are given higher weights, ensuring that the edges of those regions are preferentially preserved during the folding process.

[0027] Optionally, when using the Poisson equation to complete the surface of a hole region, the normal vector constraint at the hole boundary can be used as the boundary condition of the Poisson equation to ensure that the completed surface and the original mesh maintain geometric continuity at the boundary. The Poisson equation is in the form of: in: Represents the Laplace operator. Let be the surface height function of the hole region to be solved. The gradient vector field of the mesh vertices in the neighborhood of the hole boundary. Represents a vector field Calculate the divergence. The function value for each vertex inside the hole region is obtained by solving the above equation. This allows us to determine the coordinates of the vertex in three-dimensional space.

[0028] In some embodiments, after smoothing the mesh model of the guqin, thickness detection is performed on the surface and bottom of the guqin model. Thickness detection is achieved by emitting rays from the vertices of the surface mesh along the normal direction and finding their intersection with the bottom mesh, calculating the distance between the intersection points as the local thickness value. If the local thickness value is lower than a preset guqin thickness threshold, the vertices of the corresponding region are adjusted outward along the normal direction. It can be understood that when performing multi-view feature point matching and dense reconstruction on the 2D image set of the guqin, a hierarchical clustering matching strategy is adopted to improve feature point matching efficiency. The 2D image set of the guqin is grouped according to the similarity of viewpoints, feature point matching is performed within each group, and global matching is achieved through common feature points between groups. It can also be understood that after fusing and denoising the generated 3D color point cloud of the guqin with the depth point cloud data of the guqin, uniform resampling is performed on the complete point cloud data of the guqin. Uniform resampling is achieved by setting a fixed voxel size and retaining the point closest to the voxel center within each voxel as the sampling point, thus homogenizing the point cloud density distribution.

[0029] In one embodiment of the present invention, based on prior structural knowledge of the target traditional musical instrument, a series of key feature points are defined on the basic three-dimensional geometric model. These key feature points correspond to physical connection points and movable points between instrument components. Based on these key feature points, a hierarchical animation skeleton structure is constructed within the basic three-dimensional geometric model. This animation skeleton structure consists of a root node, multiple joint nodes, and corresponding skeletal chains, used to drive the movement of the basic three-dimensional geometric model. According to the physical laws of performance, motion constraint parameters are set for each joint node in the animation skeleton structure. These motion constraint parameters include the range of degrees of freedom for joint rotation, speed limits, and inter-joint linkages. Using a physics simulation engine, physical properties are set for the basic three-dimensional geometric model, including the mass, elasticity, and damping of components, and the deformation of the surface mesh of the basic three-dimensional geometric model is calculated when external forces are applied or joint movements occur. The animation skeleton structure is bound to the calculated deformation data to generate the dynamic three-dimensional musical instrument model. This model can respond to external interactive commands and perform animation demonstrations based on the motion constraint parameters and physical properties.

[0030] In practical implementation, the guqin, a traditional Chinese musical instrument, is used as the target instrument. The model dynamization module applies an improved structural and deformation animation generation algorithm to the basic 3D geometric model of the guqin. Based on the physical structural characteristics and playing physics of the guqin, the model dynamization module optimizes the animation skeleton and deformation parameters to generate a dynamic 3D instrument model with movable joints and deformation parameters. In practical implementation, based on prior knowledge of the guqin's structure, a series of key feature points are defined on the basic 3D geometric model of the guqin. These key feature points correspond to the physical connection points and movable points between the guqin's components. The components of the guqin include the soundboard, the bottom, the strings, the bridge, the dragon's teeth (a type of joint), the goose feet (another type of joint), and seven fret positions. The physical connection points of the guqin include the contact points between the strings and the bridge, the contact points between the strings and the dragon's teeth, and the fixed points between the goose feet and the bottom. The movable points of the guqin include the pressing points of the strings and the rotation points of the goose feet.

[0031] In some embodiments, a hierarchical animation skeleton structure is constructed within the basic three-dimensional geometric model of the guqin based on the key feature points of the guqin. The animation skeleton structure of the guqin consists of a root node, multiple joint nodes, and corresponding skeletal chains, used to drive the movement of the basic three-dimensional geometric model of the guqin. The root node of the guqin is located at the geometric center of the instrument body. The joint nodes of the guqin include the yueshan node, the longyin node, seven string nodes, and two yanzu nodes. The skeletal chains of the guqin extend from the root node to the yueshan node and the longyin node, and then connect from the yueshan node and the longyin node to each string node respectively. Each string node of the guqin is further subdivided into the string root node, the string middle node, and the string pressing node to simulate the vibration and deformation of the string at different positions.

[0032] In practical implementation, based on the physical laws of performance, motion constraint parameters are set for each joint node in the animation skeleton structure of the guqin. These motion constraint parameters include the range of degrees of freedom for joint rotation, speed limits, and the linkage between joints. For the string nodes, the range of degrees of freedom for joint rotation is limited to a torsional angle around the string axis not exceeding ±5 degrees, and a bending angle around the axis perpendicular to the string axis not exceeding ±15 degrees. For the goose-foot nodes, the range of degrees of freedom for joint rotation is limited to a rotation angle around the goose-foot axis from 0 degrees to 360 degrees. The speed limit for the string nodes is set to an angular velocity not exceeding 60 degrees per second. The linkage between the bridge and the dragon-tooth nodes is set such that when the bridge node moves, the dragon-tooth nodes move synchronously at a 1:1 displacement ratio.

[0033] Optionally, a physics simulation engine is used to set physical properties for the basic 3D geometric model of the guqin. These physical properties include the mass, elasticity, and damping of the components. The mass of the guqin's soundboard is set to 1.5 kg, the mass of the soundboard's base is set to 0.8 kg, and the mass of each string is set to 5 g. The elastic modulus of the strings is set to 800 Newtons per meter, and the elastic modulus of the soundboard and soundboard is set to 12,000 Newtons per meter. The damping coefficient of the strings is set to 0.05 N / s, and the damping coefficient of the soundboard and soundboard is set to 0.3 N / s. The physics simulation engine calculates the deformation of the surface mesh of the basic 3D geometric model of the guqin when external forces or joint movements are applied. For the string surface mesh, the physics simulation engine calculates the displacement of each mesh vertex under string tension based on a point-mass spring model. For the surface mesh of the guqin, the physics simulation engine calculates the local concave deformation of the guqin surface caused by finger pressing based on the finite element method.

[0034] Optionally, in the physics simulation engine, the displacement of the mesh vertices on the surface of the guqin's strings is calculated using the following formula: in: This represents the elastic force acting on the vertices of the grid on the surface of the strings of the guqin. This refers to the elasticity coefficient of the strings in the guqin. This represents the current position coordinates of the vertices of the grid on the surface of the strings of the guqin. This represents the initial position coordinates of the vertices of the grid on the surface of the guqin's strings. This indicates the damping coefficient of the string components of the guqin. This indicates the speed at which the vertices of the grid on the surface of the strings of the guqin move.

[0035] In some embodiments, the animated skeleton structure of the guqin is bound to the calculated deformation data to generate a dynamic 3D instrumental model of the guqin. The binding process maps the transformation matrix of each joint node in the animated skeleton structure of the guqin to the corresponding vertex weights of the guqin's surface mesh. The vertex weights of the guqin's surface mesh are determined using a linear hybrid skinning algorithm. Each vertex receives the weight of one or more joint nodes based on its distance to the nearest joint node, and the sum of all weights is normalized to 1. The dynamic 3D instrumental model of the guqin responds to external interaction commands and performs animation demonstrations based on the motion constraint parameters and physical properties. When the external interaction command specifies pressing the 5th fret of the 7th string of the guqin, the dynamic 3D instrumental model of the guqin drives the corresponding string pressing node to move in a direction perpendicular to the surface of the instrument downwards. The range of motion is limited by the degrees of freedom of the joint rotation, and the speed of motion is limited by the speed limit. Simultaneously, the physics simulation engine calculates the stretching deformation of the string surface mesh and the local concave deformation of the instrument surface, generating a continuous sequence of animation frames.

[0036] It can be understood that after constructing a hierarchical animation skeleton structure within the basic 3D geometric model of the guqin, inverse kinematics is performed on this animation skeleton structure. The inverse kinematics solution deduces the rotation angle of each joint node based on the position of the end effector of the guqin, which is used to achieve automatic adaptation of the string joints when the fingers press any string position in guqin playing techniques. It can also be understood that after setting physically based properties for the basic 3D geometric model of the guqin in the physics simulation engine, an initial tension preload is applied to the string components of the guqin. This initial tension preload is achieved by applying opposite tension vectors at both ends of the string nodes of the guqin, keeping the surface mesh of the guqin strings taut when there is no external force.

[0037] In one embodiment of the present invention, see [reference] Figure 3The process involves collecting multiple sets of motion capture data of the target traditional musical instrument under standard performance techniques. This motion capture data records the real-time three-dimensional position and rotation angle of each component of the instrument during performance. Kinematic features of each movable joint are extracted from the motion capture data, including rotation axis, maximum and minimum angles, and angular velocity range. Principal component analysis is used to determine the main rotation axis of the joint, and statistical methods are used to calculate the mean and variance of the rotation angle to determine its range of motion. The correlation between the movements of different joints in the motion capture data is analyzed, and motion linkage rules between joints are defined. These rules describe the quantitative proportion and time delay of the movement of the driven joint when the driving joint moves. The extracted kinematic features and the defined motion linkage rules are then configured as the motion constraint parameters for the corresponding joint nodes in the animation skeleton structure. The physical material attribute data includes high-resolution texture maps, normal maps, roughness maps, and metallic maps collected from the surface of real musical instruments. The high-resolution texture maps are mapped onto the corresponding surfaces of the dynamic 3D musical instrument model to determine its base color and pattern. The normal maps are applied to the dynamic 3D musical instrument model to simulate the light and shadow changes caused by the microscopic geometric details of its surface. The roughness maps and metallic maps are assigned to different material regions of the dynamic 3D musical instrument model to define its surface's specular and diffuse reflection characteristics. In a real-time rendering environment, the physically based rendering pipeline calculates and outputs the pixel color of each frame of the dynamic 3D musical instrument model based on all the above maps, ambient lighting information, and viewing angle, thereby generating teaching resource images with a realistic appearance.

[0038] In the specific implementation, the guqin, a traditional musical instrument, is used as the target instrument. Before the rendering generation module renders the physical material attribute data to the dynamic 3D instrument model, the model dynamization module needs to set motion constraint parameters for each joint node in the guqin's animated skeleton structure. Multiple sets of motion capture data of the guqin under standard playing techniques are collected. This motion capture data records the real-time 3D position and rotation angle of each component of the guqin during performance. In the specific implementation, an optical motion capture system is used to record the performance process of professional guqin players. Passive marker balls are fixed on the guqin's surface, bottom, bridge, dragon teeth, the roots and middle of the seven strings, and the two goose feet. The 3D spatial coordinates of the marker balls are recorded at a sampling frequency of 120 frames per second. Simultaneously, a 6-axis inertial measurement unit is attached to the vicinity of the string pressing points to record local rotation angles.

[0039] In some embodiments, kinematic features of each movable joint are extracted from the motion capture data of the guqin. These kinematic features include rotation axes, maximum and minimum angles, and angular velocity ranges. Principal component analysis (PCA) is used to determine the principal rotation axis of each movable joint. PCA takes the three-dimensional rotation matrix sequence of the guqin joint in the motion capture data as input, calculates the eigenvectors of the covariance matrix, and uses the direction of the eigenvector corresponding to the largest eigenvalue as the direction of the principal rotation axis of the guqin joint. Statistical methods are used to calculate the mean and variance of the rotation angles to determine the range of motion of the guqin joint. Specifically, all rotation angle values ​​of the guqin joint during the motion capture period are sorted, the 5th percentile is taken as the minimum angle, the 95th percentile as the maximum angle, and the difference is taken as the range of motion. For the string pressing joints of the guqin, the extracted kinematic features include a mean deflection angle of the pressing direction relative to the normal to the instrument surface of 82 degrees and a variance of 3 degrees, with a rotational angular velocity range of 0 to 120 degrees per second (see Table 1).

[0040] Table 1: Kinematic Characteristics of Joints in the Guqin In practical implementation, the correlation of different joint movements in the motion capture data of the guqin was analyzed, and the motion linkage rules between joints were defined. These motion linkage rules describe the quantitative ratio and time delay of the driven joint's movement when the driving joint moves. Cross-correlation analysis was performed on the displacement time series data of the guqin's Yueshan and Longyin joints; the delay time corresponding to the peak value is the time delay, and the quantitative ratio was determined using linear regression coefficients. The motion linkage rule between the Yueshan and Longyin joints was set such that when the Yueshan joint moves, the Longyin joint moves in the same direction at a displacement ratio of 0.9:1 after 0.02 seconds. The motion linkage rule between the string pressing joints and the surface deformation area was set such that for every 1-degree increase in the rotation angle of the string pressing joints, the corresponding grid vertex of the surface deformation area sinks 0.03 mm along the normal direction.

[0041] Optionally, the extracted kinematic features of the guqin and the defined motion linkage rules of the guqin are configured together as the motion constraint parameters of the corresponding joint nodes in the animation skeleton structure of the guqin. The motion constraint parameters of the guqin's Yueshan joint node include the main rotation axis direction around the world coordinate system X-axis, with a minimum angle of -5 degrees, a maximum angle of 8 degrees, an angular velocity range of 0 degrees per second to 45 degrees per second, and linkage rule identifiers belonging to the Yueshan joint. The motion constraint parameters of the guqin's string pressing joint nodes include the main rotation axis direction around the string local coordinate system Z-axis, with a minimum angle of -15 degrees to -5 degrees (set according to different string positions), a maximum angle of 3 degrees to 5 degrees, an angular velocity range of 0 degrees per second to 135 degrees per second, and linkage rule parameters driving the deformation of the instrument surface.

[0042] In practice, the rendering module acquires the physical material properties data of the guqin, including high-resolution texture maps, normal maps, roughness maps, and metallic maps collected from the surface of a real guqin. Using a DSLR camera with a macro lens, images of the guqin's surface, bottom, bridge, feet, and string areas are captured under cross-polarized lighting conditions. Eight images of each area are collected from different angles, and a diffuse texture map with a resolution of 300 dots per inch is generated using photogrammetry software. A 3D scanner is used to acquire the normal map of the guqin's surface, and the surface normal deviation of each pixel is calculated from images taken from multiple light sources. A handheld roughness meter is used to collect roughness values ​​of different material areas of the guqin in a contact manner, and the measured values ​​are mapped to grayscale images in the range of 0 to 1 as roughness maps. In the metallic map of the guqin, the string area is set to a metallic value of 0.9, the lacquered surface area is set to a metallic value of 0.05, and the metal parts of the feet are set to a metallic value of 0.8.

[0043] Optionally, a high-resolution texture map of the guqin is mapped onto the corresponding surface of the dynamic 3D instrumental model of the guqin to determine the base color and patterns of the guqin. The mapping process uses texture sampling based on UV coordinates. For the surface area of ​​the guqin, the color value of the corresponding pixel in the diffuse texture map is used as the base color of the guqin model surface, preserving the crack texture and fret marking patterns of the guqin. The normal map of the guqin is applied to the dynamic 3D instrumental model of the guqin to simulate the light and shadow changes brought about by the microscopic geometric details of the guqin surface. The three-channel normal vector of each texel stored in the normal map replaces the original normal of the model surface, producing a bumpy visual effect in the lighting calculation. The roughness map and metallicity map of the guqin are assigned to different material areas of the dynamic 3D instrumental model of the guqin, respectively. The roughness map value of the lacquer surface area of ​​the guqin is 0.7, and the metallicity map value is 0.05. The roughness map value of the string area of ​​the guqin is 0.2, and the metallicity map value is 0.9.

[0044] In some embodiments, in a real-time rendering environment, the physically based rendering pipeline calculates and outputs the pixel color of each frame of the guqin's dynamic 3D instrument model based on all the aforementioned textures, ambient lighting information, and viewing angle. The physically based rendering pipeline for the guqin uses a microfacet-based lighting model, and its calculation method is as follows: in: This indicates the point on the surface of the dynamic three-dimensional instrument model of the guqin. Along the direction of observation The emitted radiance, Representing the surface points of the guqin The bidirectional reflection distribution function at that location, This indicates a channel-wise multiplication operation. Indicates direction Incident point on the surface of the guqin Radiance This represents the dot product of the incident direction and the normal to the surface of the guqin. Indicated by the surface normal of the guqin The central hemispherical spatial integration domain is used. In the real-time rendering environment, image-based lighting technology is used to pre-calculate the irradiance integral of the environment cube map. At runtime, the peak width of specular reflection and the intensity of diffuse reflection are dynamically adjusted by sampling the values ​​of roughness map and metallicity map to generate teaching resource images with a realistic appearance.

[0045] It is understandable that when extracting the kinematic features of each movable joint from the motion capture data of the guqin, the rotation angle data is filtered using a quaternion spherical linear interpolation method to eliminate high-frequency noise introduced by marker point jitter. It is also understandable that after assigning the roughness map and metallicity map of the guqin to different material regions of the dynamic 3D instrument model, an anisotropic lighting model is additionally enabled for the string region of the guqin in the real-time rendering environment to simulate the optical reflection characteristics of the brushed texture direction on the string surface.

[0046] In one embodiment of the present invention, a teaching instruction is received, which specifies the name of a performance technique to be demonstrated; a preset performance technique action library is queried, which stores standard joint motion trajectory data and deformation parameter change sequences corresponding to the name of the performance technique; the standard joint motion trajectory data and deformation parameter change sequences matching the teaching instruction are retrieved from the performance technique action library; the retrieved standard joint motion trajectory data and deformation parameter change sequences are used to drive the animation skeleton structure and surface mesh of the dynamic three-dimensional instrumental model to generate a continuous and smooth teaching interactive action sequence. The construction process of the performance technique motion library includes: recording inertial measurement unit data attached to key parts of the instrument when a professional performer operates a real instrument, or capturing three-dimensional motion data of marked points through an optical motion capture system; cleaning, filtering, and retargeting the recorded raw motion data, adapting it to the standardized animation skeleton structure, and generating standardized joint rotation data; synchronously recording local deformation data of the instrument surface collected by a high-speed camera or tactile sensor during the performance, and aligning it with the standardized joint rotation data on the timeline; packaging the standardized joint rotation data and the aligned local deformation data of the instrument surface, as well as associating them with performance technique tags, and storing them as a complete technique record in the performance technique motion library.

[0047] In practical implementation, the guqin, a traditional Chinese musical instrument, is used as the target instrument. The improved structure and deformation animation generation algorithm also includes the generation of teaching interactive action sequences. The model dynamization module receives teaching instruction input, which specifies the name of the playing technique to be demonstrated. Guqin playing technique names include "open strings," "pressed notes," "harmonics," "mo," "tiao," "gou," "ti," "bo," and "tuo," among others. In practical implementation, the user selects the "harmonics" playing technique through the interactive interface, and the system generates the corresponding teaching interactive action sequence. The model dynamization module queries a preset playing technique action library, which stores standard joint motion trajectory data and deformation parameter change sequences corresponding to the guqin playing technique names. The playing technique action library is stored in a structured file format, with each technique entry containing a technique identifier, time axis sampling points of the joint rotation sequence, and a deformation parameter mapping table for the corresponding time.

[0048] In some embodiments, standard joint motion trajectory data and deformation parameter change sequences matching the teaching instructions are retrieved from the guqin playing technique motion library. When the teaching instruction is "harmonics," the system retrieves the entry with the technique identifier "HY," reads the stored three-dimensional trajectory point sequence of the left-hand index finger joint vertically rising and falling above the seven frets (each trajectory point contains a timestamp, spatial coordinates, and rotation quaternions), and the motion trajectory data of the right-hand plucking joint as it plucks the strings in the middle section. Simultaneously, the sequence of deformation parameter changes on the guqin surface synchronized with the joint movements is read, including the displacement amplitude and attenuation curve of the string vibration node at the moment of harmonic triggering. The retrieved standard joint motion trajectory data and deformation parameter change sequences of the guqin drive the animation skeleton structure and surface mesh of the guqin's dynamic three-dimensional instrument model, generating a continuous and smooth sequence of interactive teaching actions. The driving process uses spline interpolation to smoothly connect discrete trajectory points, updates the local transformation matrix of each joint node in the animation skeleton structure each frame, and adjusts the displacement of the string surface mesh vertices according to the deformation parameter change sequence.

[0049] Optionally, the construction of the guqin playing technique motion library includes recording inertial measurement unit (IMU) data attached to key parts of the guqin when a professional performer operates it, or capturing three-dimensional motion data of marked points using an optical motion capture system. Twenty-eight optical markers were placed on the guqin's bridge, dragon's teeth, the pressing points of the seven strings (five commonly used fret positions for each string), the two goose feet, and the four corners of the body. An optical motion capture system consisting of 12 infrared cameras recorded motion data of a professional performer executing 32 commonly used playing techniques at a sampling rate of 240 frames per second. Simultaneously, six-axis IMUs were attached to the root and middle of each string, recording local angular velocity and acceleration data at a sampling rate of 180 times per second (see Table 2).

[0050] Table 2: List of Guqin Playing Techniques and Movements In practice, the recorded raw motion data undergoes cleaning, filtering, and redirection processing to adapt it to the standardized animation skeleton structure of the guqin, generating standardized joint rotation data. The cleaning process removes flying point data (points whose 3D coordinates exceed the normal range by 2 standard deviations) generated when marker points are occluded. The filtering process uses a low-pass Butterworth filter with a cutoff frequency of 20 Hz to remove high-frequency noise. The redirection process transforms the marker point coordinates from the optical motion capture coordinate system to the local coordinate system of the guqin's animation skeleton structure. During redirection, the optimal rigid body transformation matrix between the marker points and corresponding nodes of the animation skeleton is solved to map the raw motion data onto standardized joint nodes, generating a rotation Euler angle sequence for each joint node in the world coordinate system. Simultaneously, local deformation data of the guqin surface, collected by a high-speed camera or tactile sensor during performance, is recorded and aligned with the standardized joint rotation data on the timeline. Sixteen thin-film pressure sensors (sampling frequency 500 times per second) were placed beneath the surface of the guqin. Simultaneously, two high-speed cameras (1000 frames per second) captured the string vibration and surface concavity processes from the front and sides, extracting the vertical displacement of the grid vertices as local deformation data. A dynamic time warping algorithm was used to align the time axis of the standardized joint rotation data with the time axis of the local deformation data, ensuring that each joint movement corresponds to the correct deformation parameters.

[0051] Optionally, standardized joint rotation data and aligned local deformation data of the guqin surface are packaged together and associated with performance technique tags, stored as a complete technique record in the guqin performance technique motion library. The packaging process uses a hierarchical data structure. The root node contains technique tags and metadata (recording date, performer ID, technique category), and child nodes contain arrays of joint names, rotation trajectories (each element being a timestamp plus a rotation vector), and deformation parameters (each element being a timestamp plus a vertex index and displacement). The performance technique motion library is stored as a non-relational database document. Each technique record is stored independently and a hash index of the technique name is created to support fast querying. This can be understood as follows: when recording motion capture data of professional performers operating a real guqin, the string markers are spring-loaded to prevent the quality of the markers from affecting the natural frequency of the strings' free vibration. Similarly, after packaging the standardized joint rotation data and aligned local deformation data of the guqin surface into the performance technique motion library, each technique record in the library is compressed and encoded. The quaternion compression algorithm is used to compress the joint rotation data from 4 floating-point numbers to 2 floating-point numbers. Differential coding technology is used to store the difference values ​​between adjacent frames of the deformation parameter sequence to reduce data redundancy.

[0052] In one embodiment of the present invention, the interactive function integration module adds interactive response logic to different components of the dynamic three-dimensional instrumental model. The interactive response logic defines the animation or state change that the corresponding component of the dynamic three-dimensional instrumental model should execute when the user triggers an interactive event through an input device. Multiple interactive hotspot areas are set on the dynamic three-dimensional instrumental model, and each hotspot area is associated with corresponding knowledge explanation content. The dynamic three-dimensional instrumental model, the interactive response logic, the hotspot areas, and the associated knowledge explanation content are encapsulated into an independently runnable interactive three-dimensional application or a three-dimensional scene file that can be embedded in a webpage. The interactive response logic includes component decomposition and display logic and simulated performance response logic: The component decomposition and display logic is configured to respond to the user's separation command for the overall instrument model, control the specified joint nodes in the animation skeleton structure to move according to a preset separation path and animation duration, so that the specified instrument component is separated from the overall model and hovers, while highlighting the component; The simulated performance response logic is configured to respond to the user's touch or click command on the virtual performance interface, map the command to a drive signal for a specific component of the dynamic three-dimensional instrument model, the drive signal triggers the dynamic three-dimensional instrument model to perform corresponding joint movements and surface deformations, and calls the audio engine to play pre-recorded or real-time synthesized sound corresponding to the performance action.

[0053] In practical implementation, the guqin, a traditional Chinese musical instrument, is used as the target instrument. After the rendering module generates interactive 3D teaching resources with a realistic appearance, the interactive function integration module adds interactive response logic to different components of the guqin's dynamic 3D instrument model. This interactive response logic defines the animation or state change that the corresponding component of the guqin's dynamic 3D instrument model should execute when a user triggers an interactive event through an input device. In practice, the user clicks with a mouse on a personal computer or touches the screen with their finger. The interactive function integration module receives ray detection hit events from the input device and looks up the corresponding interactive response logic table based on the component name of the hit point's mesh. The guqin's interactive response logic table is indexed by component name, and each entry contains trigger conditions, execution action type, and action parameters. The guqin's string component is configured to play the corresponding pitch when clicked; the guqin's goose foot component is configured to trigger a rotation animation when clicked; and the guqin's bridge component is configured to highlight and display a text description when clicked.

[0054] In some embodiments, multiple interactive hotspot areas are set on the dynamic three-dimensional instrument model of the guqin, each hotspot area being associated with corresponding explanatory content. Circular hotspot areas are set at the seven fret positions of the guqin, each with a radius of 3 mm. These hotspot areas display a semi-transparent halo effect when the user approaches. The explanatory content associated with the hotspot area of ​​the thirteenth fret position (outer side of the soundboard) is "Guqin fret positions are used to mark harmonic positions; there are 13 fret positions in total." The explanatory content associated with the hotspot area of ​​the guqin's bridge is "The bridge is the highest part of the guqin, used to support the strings; its height affects string vibration." The explanatory content associated with the hotspot area of ​​the guqin's dragon's teeth is "The dragon's teeth are located at the end of the soundboard, through which the strings pass and are fixed." The explanatory content associated with the hotspot area of ​​the guqin's goose feet is "The goose feet are used to wrap around the end of the strings, adjusting string tension." The explanatory content associated with the hotspot area of ​​each string of the guqin includes the name of the string (e.g., first string, second string) and its corresponding open string pitch.

[0055] In practice, the dynamic 3D instrument model of the guqin, its interactive response logic, its hotspot areas, and related explanatory content are encapsulated into a standalone interactive 3D application or a 3D scene file that can be embedded in a webpage. The encapsulation process uses WebGL technology to export the dynamic 3D instrument model of the guqin as a glTF binary file. The interactive response logic of the guqin is written in JavaScript and embedded in an HTML file. The hotspot area data of the guqin is stored as an additional information layer in JSON format and bound to the model file. The standalone interactive 3D application is published as a portable executable file for the Windows platform, and the 3D scene file that can be embedded in a webpage is generated as a single HTML file, supporting direct opening and running in mainstream browsers.

[0056] Optionally, the interactive response logic of the guqin includes component decomposition and display logic. This logic is configured to respond to user commands to separate the guqin's overall model, controlling specified joint nodes in the guqin's animated skeleton structure to move according to a preset separation path and animation duration. This separates the specified guqin component from the overall model and hovers it, while simultaneously highlighting the component. The user issues a separation command by clicking the "Decompose" button on the interface, specifying the guqin component to be separated as "Yueshan". The Yueshan joint node in the guqin's animated skeleton structure moves along a preset separation path (moving 15 cm from its original position along the positive Y-axis of the world coordinate system, then 8 cm along the positive X-axis), with an animation duration of 0.8 seconds. An easing function is used to make the initial and final speeds zero. After the Yueshan joint node reaches its endpoint, it hovers for 2 seconds, while the surface material's self-illumination intensity increases to 0.7, and its edge outline is highlighted in yellow. When the user clicks the reset button, the Yueshan joint node moves back to its initial position along the reverse path, and the highlight disappears.

[0057] Optionally, the interactive response logic of the guqin includes simulated performance response logic. This simulated performance response logic is configured to respond to touch or click commands from the user on the virtual performance interface, mapping these commands to drive signals for specific components of the guqin's dynamic 3D instrumental model. These drive signals trigger the guqin's dynamic 3D instrumental model to perform corresponding joint movements and surface deformations, and invoke the audio engine to play pre-recorded or real-time synthesized sounds corresponding to the performance action. For example, when a user touches the fifth fret of the seventh string on the virtual performance interface, the touch command is converted into a drive signal after coordinate mapping. This drive signal includes a target component identifier (seventh string pressing joint), the direction of movement (vertically downward), and the displacement (5 mm). The seventh string pressing joint node of the guqin's dynamic 3D instrumental model moves from its initial position to the target position within 4 frames (approximately 0.067 seconds). Simultaneously, the physics simulation engine calculates the stretching deformation of the string surface mesh and the local concave deformation of the instrument surface, generating a continuous deformation animation. The audio engine reads the audio segment corresponding to the fifth harmonic of the seventh string from the pre-recorded waveform file. The audio segment is 2 seconds long and is played in stereo output mode.

[0058] In some embodiments, when encapsulating the dynamic 3D instrument model of the guqin, the interactive response logic of the guqin, the hot spots of the guqin, and the associated knowledge explanation content, the animation parameters in the interactive response logic are optimized for performance. The interpolation calculation of the animation parameters uses the following formula: in: This represents the joint nodes in the animation skeleton structure of the guqin (a seven-stringed zither) in time. Location at any given moment Indicates the initial position of the joint node. Indicates the target position of the joint node. Indicates about time The smooth step function and defined as Using this formula for interpolation ensures that the instantaneous velocity of the joint nodes is zero at the start and end of the animation, avoiding abrupt changes in motion.

[0059] It's understandable that when setting interactive hotspot areas on the dynamic 3D instrument model of the guqin, the activation detection of these hotspot areas uses a ray-and-bounding-box hierarchy intersection test method to improve response efficiency in multi-touch scenarios. It's also understandable that after configuring the simulated performance response logic in the guqin's interactive response logic, the audio engine's loading strategy is asynchronously processed, preloading all pre-recorded audio segments of the guqin into the memory buffer when the application starts, thus keeping the audio playback latency when the user triggers a performance command within 50 milliseconds.

[0060] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A 3D-based interactive 3D teaching resource generation system for traditional instrumental music, characterized in that: The system includes: The data acquisition module acquires real instrumental data of the target traditional instrument, which includes a set of two-dimensional images, depth point cloud data, and physical material attribute data of the target traditional instrument. The model reconstruction module performs a three-dimensional reconstruction algorithm on the two-dimensional image set and the depth point cloud data to generate a basic three-dimensional geometric model of the target traditional musical instrument. The model dynamization module executes an improved structure and deformation animation generation algorithm on the basic three-dimensional geometric model. The improved structure and deformation animation generation algorithm optimizes the animation skeleton and deformation parameters based on the physical structure characteristics and performance physics of the target traditional musical instrument, thereby generating a dynamic three-dimensional musical instrument model with movable joints and deformation parameters. The rendering generation module renders the physical material attribute data onto the dynamic 3D instrument model to generate interactive 3D teaching resources with a realistic appearance.

2. The 3D-based interactive three-dimensional teaching resource generation system for traditional instrumental music according to claim 1, characterized in that, A 3D reconstruction algorithm is performed on the set of 2D images and the depth point cloud data to generate a basic 3D geometric model of the target traditional musical instrument, including: Multi-view feature point matching and dense reconstruction are performed on the two-dimensional image set to generate a color three-dimensional point cloud that is aligned with the scale of the depth point cloud data; The color 3D point cloud and the depth point cloud data are fused and denoised to form complete point cloud data of the target traditional musical instrument. Poisson surface reconstruction is performed on the point cloud data to generate a mesh model containing topological connections; The mesh model is simplified, holes are repaired, and smoothing is performed to obtain the basic three-dimensional geometric model with a complete surface and accurate structure.

3. The 3D-based interactive three-dimensional teaching resource generation system for traditional instrumental music according to claim 2, characterized in that, The mesh model is simplified, hole repaired and smoothed, including: using an edge folding algorithm based on a quadratic error metric for mesh simplification, and using the Poisson equation for surface completion of the hole region.

4. The 3D-based interactive three-dimensional teaching resource generation system for traditional instrumental music according to claim 1, characterized in that, An improved structure and deformation animation generation algorithm is applied to the basic 3D geometric model. This algorithm optimizes the animation skeleton and deformation parameters based on the physical structural characteristics and performance physics of the target traditional musical instrument, thereby generating a dynamic 3D musical instrument model with movable joints and deformation parameters, including: Based on the prior knowledge of the structure of the target traditional musical instrument, a series of key feature points are defined on the basic three-dimensional geometric model. These key feature points correspond to the physical connection points and movable points between the instrument components. Based on the key feature points, a hierarchical animation skeleton structure is constructed inside the basic three-dimensional geometric model. The animation skeleton structure consists of a root node, multiple joint nodes, and corresponding bone chains, which are used to drive the movement of the basic three-dimensional geometric model. Based on the physical laws of performance, motion constraint parameters are set for each joint node in the animation skeleton structure. The motion constraint parameters include the range of degrees of freedom of joint rotation, the speed limit of movement, and the linkage relationship between joints. Using a physics simulation engine, physical properties are set for the basic three-dimensional geometric model, including the mass, elasticity, and damping of the components, and the deformation of the surface mesh of the basic three-dimensional geometric model is calculated when external forces or joint movements are applied. The animated skeleton structure is bound to the calculated deformation data to generate the dynamic three-dimensional instrumental model. This model can respond to external interactive commands and perform animation demonstrations based on the motion constraint parameters and physical properties.

5. The 3D-based interactive three-dimensional teaching resource generation system for traditional instrumental music according to claim 4, characterized in that, Based on the aforementioned physical laws of performance, motion constraint parameters are set for each joint node in the animation skeleton structure, including: Collect multiple sets of motion capture data of the target traditional musical instrument under standard performance techniques. The motion capture data records the real-time three-dimensional position and rotation angle of each component of the instrument during performance; The kinematic features of each movable joint are extracted from the motion capture data, including rotation axis, maximum and minimum angle, and angular velocity range; The correlation between the movements of different joints in the motion capture data is analyzed, and the motion linkage rules between joints are defined. The motion linkage rules describe the quantitative proportion and time delay of the driven joint moving when the driven joint moves. The extracted kinematic features and the defined motion linkage rules are configured together as the motion constraint parameters of the corresponding joint nodes in the animation skeleton structure.

6. The 3D-based interactive three-dimensional teaching resource generation system for traditional instrumental music according to claim 5, characterized in that, Extracting the kinematic features of each movable joint from the motion capture data includes: determining the main rotation axis of the joint using principal component analysis, and calculating the mean and variance of the rotation angle using statistical methods to determine its range of motion.

7. The 3D-based interactive three-dimensional teaching resource generation system for traditional instrumental music according to claim 5, characterized in that, Rendering the physical material property data onto the dynamic 3D instrumental model includes: The physical material property data includes high-resolution texture maps, normal maps, roughness maps, and metallicity maps collected from the surface of real musical instruments; The high-resolution texture map is mapped onto the corresponding surface of the dynamic 3D instrument model to determine its base color and pattern; The normal map is applied to the dynamic 3D instrument model to simulate the changes in light and shadow caused by the microscopic geometric details of its surface; The roughness map and metallicity map are respectively assigned to different material regions of the dynamic 3D instrument model to define the specular and diffuse reflection characteristics of its surface to light. In a real-time rendering environment, the physically based rendering pipeline calculates and outputs the pixel color of each frame of the dynamic 3D instrumental model based on all the textures, ambient lighting information and viewing angle, thereby generating teaching resource images with a realistic appearance.

8. The 3D-based interactive three-dimensional teaching resource generation system for traditional instrumental music according to claim 4, characterized in that, The improved structure and deformation animation generation algorithm also includes the generation of teaching interactive action sequences: Receive teaching instruction input, wherein the teaching instruction specifies the name of the performance technique to be demonstrated; The preset performance technique movement library is queried. The performance technique movement library stores standard joint movement trajectory data and deformation parameter change sequences corresponding to the name of the performance technique. Retrieve standard joint motion trajectory data and deformation parameter change sequences that match the teaching instructions from the performance technique motion library; The standard joint motion trajectory data and deformation parameter change sequence are invoked to drive the animation skeleton structure and surface mesh of the dynamic three-dimensional instrumental model, generating a continuous and smooth sequence of teaching interactive actions.

9. The 3D-based interactive three-dimensional teaching resource generation system for traditional instrumental music according to claim 8, characterized in that, The process of constructing the performance technique motion library includes: Record data from inertial measurement units attached to key parts of the instrument when a professional performer operates a real instrument, or capture three-dimensional motion data of marked points through an optical motion capture system. The recorded raw motion data is cleaned, filtered, and redirected, and then adapted to the standardized animation skeleton structure to generate standardized joint rotation data. During the performance, local deformation data of the instrument surface, collected by a high-speed camera or tactile sensor, is recorded synchronously and aligned with the standardized joint rotation data on the timeline. The standardized joint rotation data and the aligned local deformation data of the instrument surface are packaged together and associated with the performance technique tag, and stored as a complete technique record in the performance technique action library.

10. The 3D-based interactive three-dimensional teaching resource generation system for traditional instrumental music according to claim 7, characterized in that, After generating interactive 3D teaching resources with a realistic appearance, it also includes: An interactive function integration module adds interactive response logic to different components of the dynamic three-dimensional instrumental model. The interactive response logic defines the animation or state change that the corresponding component of the dynamic three-dimensional instrumental model should execute when the user triggers an interactive event through an input device. Multiple interactive hotspot areas are set on the dynamic three-dimensional instrument model, and each hotspot area is associated with corresponding knowledge explanation content; The dynamic 3D instrument model, the interactive response logic, the hotspot areas, and the associated knowledge explanations are encapsulated into a stand-alone interactive 3D application or a 3D scene file that can be embedded in a webpage; The interactive response logic includes component decomposition and display logic and simulated performance response logic: The component decomposition and display logic is configured to respond to the user's separation command for the overall instrument model, control the specified joint nodes in the animation skeleton structure to move according to the preset separation path and animation duration, so that the specified instrument component is separated from the overall model and hovers, while highlighting the component. The simulated performance response logic is configured to respond to the user's touch or click command on the virtual performance interface, map the command to a drive signal for a specific part of the dynamic three-dimensional instrument model, the drive signal triggers the dynamic three-dimensional instrument model to perform corresponding joint movements and surface deformations, and calls the audio engine to play pre-recorded or real-time synthesized sound corresponding to the performance action.

Images