An artificial intelligence-based system for piano learning in augmented reality (XR)

Abstract

A system for closed-loop piano learning in augmented reality (XR), consisting of: an optical music recognition (OMR) engine configured to receive note inputs, preprocess the note inputs through rotation correction and image enhancement, segment the preprocessed notes into staves and musical symbols using dual neural network architectures, analyze the position and relationship of the segmented musical symbols, and generate a structured MusicXML file based on the analyzed musical symbols; a virtual reality (VR) headset configured to display a rendered virtual environment including a virtual piano, track a user's hand movements, and register the user's tracked hand movements; a virtual reality piano interface module connected to the VR headset, the virtual reality piano interface module configured to: render a three-dimensional virtual piano model with individual interactive keys; dynamically highlight specific piano keys based on the sequential piano key instructions derived from the MusicXML file generated by the OMR engine; detect collisions between virtual representations of the user's fingers and the virtual piano keys as registered via the VR headset; provide real-time auditory, visual, and haptic feedback based on the detected collisions; and adjust the difficulty level of the piano lesson based on the user's performance; and a computing device comprising a processor and a GPU, connected to a memory, and configured to control the operation of a virtual reality piano interface module and a VR headset, wherein the computing device is configured to implement a virtual reality piano interface, process the MusicXML file to create sequential piano key instructions, transmit visual rendering data to the VR headset via the interface module, and process hand tracking data received from the VR headset.

Description

FIELD OF THE INVENTION

[0001] The present disclosure relates to an artificial intelligence-based system for closed-loop piano learning in augmented reality (XR). More specifically, the present invention relates to a system that integrates optical music recognition (OMR) and virtual reality (VR) to create an immersive piano learning environment. BACKGROUND OF THE INVENTION

[0002] Traditional piano learning techniques pose significant challenges for beginners, particularly when learning sight-reading, which requires complex coordination of visual, cognitive, and motor processes. Studies by Arthur et al. and Lehmann et al. show that novice piano players struggle with eye-hand coordination and real-time interpretation of music scores, often leading to fragmented practice and slow progress. The lack of immediate feedback in conventional teaching methods further limits learners' ability to correct errors in rhythm, pitch, and dynamics during practice sessions.

[0003] Recent advances in optical music recognition (OMR) have fundamentally transformed musical notation processing. Deep learning approaches significantly improve symbol recognition and sequence modeling compared to traditional rule-based systems. At the same time, artificial intelligence (AI) has enhanced music education through real-time performance analysis and adaptive feedback mechanisms that can detect errors in timing, finger position, and dynamics.

[0004] Extended Reality (XR) technologies offer unique opportunities for immersive music instruction by simulating realistic practice environments while leveraging spatial audio and hand-tracking capabilities. Despite these individual technological advances, existing systems typically address isolated aspects of the piano learning process rather than providing an integrated solution.

[0005] The present invention addresses these limitations by integrating OMR, AI, and XR technologies into a unified closed-loop learning system that digitizes notation, delivers AI-driven feedback, and creates an immersive practice environment, transforming the way piano skills are acquired and reinforced. Summary of the invention

[0006] The present disclosure relates to an AI-based system for closed-loop piano learning in augmented reality (XR). The present invention provides an AI-powered closed-loop piano learning system in augmented reality (XR) that integrates optical music recognition (OMR), artificial intelligence (AI), and XR technologies. The system transforms traditional sheet music into an immersive, interactive learning experience by automatically processing music, presenting it in a virtual environment, and providing adaptive real-time feedback based on user performance.The core components include an OMR engine that converts sheet music into structured MusicXML data using dual neural network architectures, a computing device that processes this data and user interactions, a virtual reality (VR) headset with hand tracking capabilities, and a VR piano interface module that renders a virtual piano with dynamically highlighted keys while analyzing user performance using AI algorithms to provide personalized feedback and guidance.

[0007] The present disclosure aims to provide an artificial intelligence-based system for closed-loop piano learning in augmented reality (XR). The system includes an optical music recognition (OMR) engine configured to receive note inputs, preprocess them through rotation correction and image enhancement, segment the preprocessed notes into staves and musical symbols using dual neural network architectures, analyze the position and relationship of the segmented musical symbols, and generate a structured MusicXML file based on the analyzed musical symbols. The system also includes a virtual reality (VR) headset configured to display a rendered virtual environment including a virtual piano, track a user's hand movements, and register the user's tracked hand movements.The system also includes a virtual reality piano interface module connected to the VR headset. The virtual reality piano interface module is configured to render a three-dimensional virtual piano model with individual interactive keys, dynamically highlight specific piano keys based on the sequential piano key instructions derived from the MusicXML file as generated by the OMR engine, detect collisions between virtual representations of the user's fingers and the virtual piano keys as registered via the VR headset, provide real-time auditory, visual, and haptic feedback based on the detected collisions, and adjust the difficulty of the piano lesson to the user's performance.The system further includes a computing device having a processor and a GPU connected to memory and configured to control operation of the virtual reality piano interface module and the VR headset, the computing device configured to: implement the virtual reality piano interface; process the MusicXML file to create sequential piano key instructions; transmit visual rendering data to the VR headset via the interface module; and process hand tracking data received from the VR headset.

[0008] One objective of the present disclosure is to provide an artificial intelligence-based system for closed-loop piano learning in augmented reality (XR).

[0009] Another objective of the present disclosure is to automate the interpretation of music notation by an advanced OMR system using dual U-Net architectures, enabling accurate processing of various music notation formats into machine-readable MusicXML.

[0010] Another objective of the present disclosure is to provide personalized, adaptive learning experiences through AI-driven analysis of the user's performance metrics and to identify errors in timing, finger placement, and dynamics in order to adjust the difficulty level accordingly.

[0011] Another objective of the present disclosure is to increase user engagement and accelerate skill acquisition through gamification elements and spatial audio, creating a comprehensive learning ecosystem that combines theoretical music knowledge with practical piano playing skills.

[0012] Another goal of the present disclosure is to overcome the limitations of traditional piano learning methods by creating an immersive XR environment that reduces cognitive load through intuitive visual guidance and real-time feedback.

[0013] To further clarify the advantages and features of the present disclosure, the invention will be explained in more detail with reference to specific embodiments illustrated in the accompanying drawings. These drawings illustrate only typical embodiments of the invention and are therefore not to be considered as limiting its scope. The invention will be described and explained in more detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE CHARACTERS

[0014] These and other features, aspects, and advantages of the present disclosure will become more fully understood when the following detailed description is read with reference to the accompanying drawings, in which like characters represent like parts throughout. Fig. 1 shows a block diagram of an artificial intelligence-based system for closed-loop piano learning in augmented reality (XR) according to an embodiment of the present disclosure; Fig. 2 shows a block diagram illustrating the architecture of the AI-assisted closed-loop piano learning system according to an embodiment of the present disclosure; and Fig. 3 illustrates a diagram showing the end-to-end pipeline of the proposed system according to an embodiment of the present disclosure.

[0015] Those skilled in the art will also appreciate that the elements in the drawings are shown for convenience and are not necessarily to scale. For example, the flowcharts illustrate the method by key steps to enhance understanding of aspects of the present disclosure. Furthermore, with respect to device construction, one or more components of the device may be represented in the drawings by conventional symbols. The drawings may show only the specific details relevant to understanding embodiments of the present disclosure in order not to clutter the drawings with details that would be readily apparent to those skilled in the art from the present description. DETAILED DESCRIPTION:

[0016] To facilitate an understanding of the principles of the invention, reference will now be made to the embodiment illustrated in the drawings and a clear description thereof. However, the scope of the invention is not limited thereby. Changes and further modifications to the illustrated system, as well as further applications of the principles of the invention, are possible, as would normally occur to one skilled in the art to which the invention pertains.

[0017] It will be understood by those skilled in the art that the foregoing general description and the following detailed description are exemplary and explanatory of the invention and are not intended to be limiting thereof.

[0018] References in this specification to "one aspect," "another aspect," or similar language mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Therefore, the language "in one embodiment," "in another embodiment," and similar language throughout this specification may or may not refer to the same embodiment.

[0019] The terms "comprises," "comprising," or other variations thereof are intended to cover non-exclusive inclusion, such that a process or method comprising a list of steps may include not only those steps, but also additional steps not expressly listed or inherent in that process or method. Likewise, the statement "comprises" for one or more devices, subsystems, elements, structures, or components does not exclude, without further limitation, the existence of other devices, subsystems, elements, structures, components, or additional devices, subsystems, elements, structures, or components.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention pertains. The systems, methods, and examples provided herein are for illustrative purposes only and should not be considered limiting.

[0021] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0022] The functional units described in this specification are referred to as devices. A device may be implemented in programmable hardware devices such as processors, digital signal processors, central processing units, field-programmable gate arrays, programmable array logic systems, programmable logic devices, cloud processing systems, or the like. The devices may also be implemented in software for execution by various types of processors. An identified device may contain executable code and may consist, for example, of one or more physical or logical blocks of computer instructions, which may be organized, for example, as an object, procedure, function, or other construct.However, the executable file of an identified device does not have to be physically stored in the same location, but may consist of different instructions stored in different locations which, logically linked, form the device and fulfill its purpose.

[0023] The executable code of a device or module may consist of one or more instructions and may even be distributed across multiple code segments, different applications, and multiple storage devices. Similarly, operational data may be identified and represented within the device and presented in any form and data structure. The operational data may be captured as a single data set or distributed across different storage devices and may be present, at least in part, as electronic signals in a system or network.

[0024] References in this specification to "a selected embodiment," "an embodiment," or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosed subject matter. Therefore, the phrases "a selected embodiment," "in an embodiment," or "in an embodiment" in various places in this specification do not necessarily refer to the same embodiment.

[0025] Furthermore, the described features, structures, or characteristics may be combined in any manner in one or more embodiments. The following description contains numerous specific details in order to provide a thorough understanding of embodiments of the disclosed subject matter. However, those skilled in the art will recognize that the disclosed subject matter may be practiced without one or more of the specific details, or with different methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail in order not to obscure aspects of the disclosed subject matter.

[0026] According to the exemplary embodiments, the disclosed computer programs or modules may be executed in a variety of ways, for example, as an application in the memory of a device or as a hosted application on a server that communicates with the device application or browser using various standard protocols such as TCP / IP, HTTP, XML, SOAP, REST, JSON, and other suitable protocols. The disclosed computer programs may be written in exemplary programming languages that execute from the memory of the device or from a hosted server, such as BASIC, COBOL, C, C++, Java, Pascal, or scripting languages such as JavaScript, Python, Ruby, PHP, Perl, or other suitable programming languages.

[0027] Some of the disclosed embodiments involve or otherwise involve the transmission of data over a network, for example, the delivery of various inputs or files over the network. The network may include, for example, the Internet, wide area networks (WANs), local area networks (LANs), analog or digital wired and wireless telephone networks (e.g., PSTN, Integrated Services Digital Network (ISDN), cellular networks, and Digital Subscriber Line (xDSL)), radio, television, cable, satellite, and / or other transmission or tunneling mechanisms for transmitting data. The network may include multiple networks or subnetworks, each including, for example, a wired or wireless data path. The network may include a circuit-switched voice network, a packet-switched data network, or another network for transmitting electronic communications.For example, the network may include Internet Protocol (IP) or Asynchronous Transfer Mode (ATM) networks supporting voice, such as VoIP, Voice over ATM, or other comparable protocols for voice data communication. In one implementation, the network includes a cellular network configured for the exchange of text or SMS messages.

[0028] Examples of the network include a Personal Area Network (PAN), a Storage Area Network (SAN), a Home Area Network (HAN), a Campus Area Network (CAN), a Local Area Network (LAN), a Wide Area Network (WAN), a Metropolitan Area Network (MAN), a Virtual Private Network (VPN), an Enterprise Private Network (EPN), the Internet, a Global Area Network (GAN), etc.

[0029] Fig. 1 shows a block diagram of an artificial intelligence-based system for closed-loop piano learning in augmented reality (XR) according to an embodiment of the present disclosure.

[0030] Referring to Fig. 1, the system (100) comprises: an optical music recognition (OMR) engine (102) configured to: receive note inputs; preprocess the note input through rotation correction and image enhancement; segment the preprocessed notes into staves and music symbols using dual neural network architectures; analyze the position and relationship of the segmented music symbols; and generate a structured MusicXML file based on the analyzed music symbols. The system (100) further comprises a virtual reality (VR) headset (104) configured to: display a rendered virtual environment including a virtual piano; track a user's hand movements; and register the tracked user's hand movements.The system (100) further comprises a virtual reality piano interface module (106) connected to the VR headset (104), the virtual reality piano interface module (106) configured to: render a three-dimensional virtual piano model with individual interactive keys; dynamically highlight specific piano keys based on the sequential piano key instructions from the MusicXML file generated by the OMR engine (102); detect collisions between virtual representations of the user's fingers and the virtual piano keys registered via the VR headset (104); provide real-time audible, visual, and haptic feedback based on the detected collisions; and adapt the difficulty level of piano lessons to the user's performance.The system (100) further includes a computing device (108) having a processor and a GPU connected to memory and controlling operation of the virtual reality piano interface module (106) and the VR headset (104). The computing device (108) is configured to: implement a virtual reality piano interface; process the MusicXML file to create sequential piano key instructions; transmit visual rendering data to the VR headset (104) via the interface module (106); and process hand tracking data received from the VR headset (104).

[0031] In one embodiment, the OMR engine (102) comprises: a preprocessing module (102a) for performing rotation corrections and image enhancements, wherein a Real-Enhanced Super-Resolution Generative Adversarial Network (Real-ESRGAN) is implemented to perform image enhancement by upscaling and sharpening the note input; and a segmentation module (102b) for isolating and identifying note symbols using staff and symbol recognition. The segmentation module (102b) of the OMR engine (102) comprises: a first U-Net for segmenting staff lines from the background into a binary mask; and a second U-Net for isolating note symbols such as clefs and rests in a separate mask; the dual architecture reduces interference between staff lines and symbols during recognition.

[0032] In one embodiment, the OMR engine (102) further comprises a post-processing module (102c) configured to perform symbol linking, layout analysis, and generation of a MusicXML data file, wherein recognized symbols are converted into a MusicXML data file that represents a standard format for musical notation, wherein the post-processing module (102c) sorts the recognized symbols by track and time, aligns them across staves, and inserts rests to synchronize rhythms during the generation of the MusicXML data file, and wherein the generated data file is transmitted to the computing device (108).

[0033] In one embodiment, the VR headset (104) comprises: finger tracking sensors (104a) configured to track the positions and movements of the user's individual fingers; and haptic feedback generators (104b) configured to provide tactile sensations that simulate key resistance when virtual keys are pressed, wherein the VR headset (104) uses ball colliders assigned to virtual fingertips and labeled as FingerTip for collision detection with virtual piano keys, wherein a key press is triggered when the fingertips cross the key colliders of the specific piano keys.

[0034] In one embodiment, the virtual reality piano interface module (106) is connected to the VR headset (104) to control the headset, wherein the interface module (106) is controlled by the computing device (108), wherein the virtual reality piano interface module (106) is configured to display a 3D model of a virtual piano via the VR headset, wherein the model of the virtual piano is created with individual colliders and animations for each key, and wherein the keys are labeled as "PianoKey" and linked to audio clips for sound playback controlled by the computing device (108).

[0035] In one embodiment, the virtual reality piano interface module (106) is configured to perform dynamic key highlighting. A piano highlight manager mechanism (106a) configures the interface module (106) to analyze the MusicXML file to generate a key sequence, synchronize the key highlighting to the tempo of a piece of music, and change the color or material of virtual piano keys to indicate which keys should be pressed at specific times. A feedback mechanism (106b) configures the interface module (106) to display a green color on a virtual piano key when the key is correctly pressed by the user, display a flashing red color on a virtual piano key when the key is incorrectly pressed by the user, and increment an error counter when an incorrect key is pressed.

[0036] In one embodiment, the VR piano interface module (106) further comprises: a spatial audio module (106c) configured to generate three-dimensional sound at a position of each virtual piano key when the key is pressed; to simulate a natural sound roll-off using audio attenuation settings; and to provide audible feedback corresponding to the pitch and velocity of each key press, wherein the function of the spatial audio module (106c) is controlled by the computing device (108) after receiving the real-time hand tracking and key press data from the VR headset (104), wherein the VR headset (104) includes speakers (104c) that enable the user to hear the output sound.

[0037] In one embodiment, the computing device (108) is further configured to implement adaptive learning, wherein the computing device (108) tracks the user's performance metrics, including timing accuracy, keystroke precision, and rhythm consistency; adjusts the tempo for complex sections based on error rates; repeats problematic actions until a predetermined mastery threshold is reached; and generates progress reports that document the user's improvement over time, and wherein the implementation of the adaptive learning occurs via a virtual reality piano interface module (106) connected to the VR headset (104).

[0038] In one embodiment, the system (100) further comprises: a gamification module (110) connected to the computing device (108) for integration with the virtual reality piano interface module (106), the gamification module (110) configured to: award points for correct keystrokes based on timing and accuracy; track high scores to motivate continued practice; implement achievement badges for completing songs or reaching skill milestones; and provide a virtual conductor avatar to provide encouragement and guidance during practice sessions.

[0039] In one embodiment, the OMR engine (102), the virtual reality headset (104), the virtual reality piano interface module (106), the computing device (108), and the gamification module (110) may be implemented in programmable hardware devices such as processors, digital signal processors, central processing units, field-programmable gate arrays, programmable array logic, programmable logic devices, cloud processing systems, or the like.

[0040] The present invention relates to an AI-assisted closed-loop piano learning system in XR that offers a novel approach to music education by integrating advanced technologies to address the challenges of piano learners. The system consists of four main components that work together to provide an immersive and responsive learning experience.

[0041] The first component is the Optical Music Recognition (OMR) engine, which serves as the input point for musical notation in the system. The OMR engine utilizes a sophisticated preprocessing pipeline to optimize sheet music for analysis. This includes rotation correction using the Hough transform to detect and align staff lines, and image enhancement using Real-ESRGAN (Enhanced Super-Resolution Generative Adversarial Network) to optimize and sharpen low-quality or degraded note input. The preprocessed images are then processed by dual U-Net neural network architectures: The first U-Net segments staff lines from the background into a binary mask, while the second U-Net isolates musical symbols such as notes, clefs, and rests. This dual approach minimizes interference between staff lines and symbols during recognition, significantly improving recognition accuracy.After segmentation, the system applies contour detection to group fragmented symbol pixels into coherent objects, calculates bounding boxes and centroids for each symbol cluster, and maps symbols to note positions to derive pitch and timing information. The final output of the OMR engine is a structured MusicXML file that encodes all musical information, including notes, duration, dynamics, and other musical elements.

[0042] The second component is the computer, which acts as the system's central processing unit. It runs the VR piano interface and manages the bidirectional data flow between the OMR engine and the XR environment. It processes the MusicXML files generated by the OMR engine to create sequential piano keystroke instructions that guide the user through the piece of music. Additionally, the computer processes the hand-tracking data received from the XR headset to detect and interpret user interactions with the virtual piano. The computer also handles the rendering calculations required to transmit visual data to the XR headset, ensuring smooth and responsive operation of the virtual environment.

[0043] The third component is the VR headset, which forms the immersive interface through which users experience the virtual piano environment. The VR headset renders the virtual environment with a detailed three-dimensional piano model and tracks the user's hand movements with high precision. Finger tracking sensors monitor the positions and movements of individual fingers, while ball colliders assigned to the virtual fingertips enable precise collision detection with virtual piano keys. The VR headset also features haptic feedback generators that simulate key resistance when virtual keys are pressed, thus making the piano playing experience more realistic. The captured hand movement data is continuously transmitted to the computing device for processing and integration into the learning application.

[0044] The fourth component is the VR piano interface, which integrates the results of the other components to create a harmonious learning environment. The interface creates a realistic, three-dimensional virtual piano model with individually interactive keys that respond to user input. A piano highlight manager analyzes the MusicXML data to generate a key sequence and synchronizes the key highlighting with the tempo of the piece of music. This visually guides users through the correct note sequence. When interacting with the virtual piano, the interface detects collisions between the virtual finger representations and the piano keys and triggers appropriate reactions. The interface features a comprehensive feedback mechanism that flashes green for correct keystrokes and red for incorrect keystrokes. An error counter is also maintained to track performance.A spatial audio module creates three-dimensional sound when each virtual piano key is pressed, simulating the natural decay of the sound and providing acoustic feedback that corresponds to the pitch and velocity of each keystroke. AI-driven analysis of the user interface identifies errors in timing, finger position, and dynamics. This information is used to adjust the difficulty of the lessons and provide targeted instruction. An adaptive learning module captures user performance metrics, including timing accuracy, keystroke precision, and rhythm consistency. It adjusts the tempo for complex sections based on error rates and repeats problematic measures until a predetermined mastery threshold is reached. The user interface also generates progress reports that document the user's progress over time, providing valuable insight into learning progress.

[0045] The system's integrated gamification module increases user engagement. It awards points for correct keystrokes based on timing and accuracy, tracks high scores to encourage continued practice, awards achievement badges for completing songs or reaching milestones, and features a virtual conductor avatar to provide encouragement and guidance during practice sessions. This comprehensive approach creates a closed-loop learning ecosystem where notes are automatically interpreted, presented in an intuitive virtual environment, practiced with real-time feedback, and continuously adjusted to individual performance—a revolution in the acquisition and refinement of piano skills.

[0046] Fig. 2 shows a block diagram illustrating the architecture of the AI-assisted closed-loop piano learning system according to an embodiment of the present disclosure.

[0047] Fig. As shown in Figure 2, the OMR engine serves as the entry point for musical notation in the system. Users can provide music in a variety of ways, including scanned sheet music, digital music files, or scores captured with a camera. The OMR engine processes this input and converts traditional music notation into structured data that can be interpreted by the VR component of the system. The VR piano interface allows the user to interact with the virtual piano through hand tracking via a VR headset. Users can naturally press virtual piano keys with their fingers and receive multimodal feedback that includes visual cues through highlighted keys, spatial audio, and performance evaluation. This interactive component creates an immersive learning environment. A key middleware bridges the gap between note recognition and virtual piano implementation.It enables bidirectional data flow, synchronizes the interpreted notation with the VR visualization, and integrates adaptive learning algorithms that adjust the difficulty level to user performance. This integration creates the "closed loop" that defines the system's responsiveness. MusicXML serves as a standardized data exchange format and enables seamless communication between the OMR engine and the VR piano interface. This structured XML format encodes all musical information, including notes, duration, dynamics, and other musical elements. This preserves the relationships between the notes and enables precise reconstruction of notes in the VR environment.

[0048] The present invention relates to a system that integrates optical music recognition (OMR) and virtual reality (VR) to create an immersive piano learning environment. It addresses the challenges beginners face in interpreting sheet music and playing musical instruments, particularly the piano. Traditional methods of music education often lack real-time feedback and intuitive guidance, which can lead to inefficiencies in skill acquisition. This invention bridges the gap between theoretical music interpretation and practical piano playing by combining advanced document analysis techniques with interactive VR technologies.

[0049] In one embodiment, optical music recognition automates the interpretation of sheet music into machine-readable formats such as MusicXML or MIDI. The OMR engine utilizes robust preprocessing techniques, including rotation correction and image optimization using Real-ESRGAN, followed by dual U-Net architectures for staff and symbol segmentation. These processes ensure high accuracy in the recognition of musical symbols, even in damaged or handwritten sheet music. The recognized notation is then converted into structured MusicXML files, which are used to generate piano key mappings and sequential instructions for learners.

[0050] In one embodiment, the VR component of the system leverages Unity-based virtual environments and a VR headset to create an interactive piano learning interface. Users interact with a virtual piano modeled as a realistic 3D object. Individual keys are dynamically highlighted based on instructions from the OMR output. Hand tracking allows users to play the piano with virtual hands, mimicking real-world interactions while receiving immediate visual and auditory feedback. This immersive experience reduces cognitive load and accelerates learning by guiding users through the highlighted keys in real time.

[0051] In one embodiment, the integration of OMR and VR creates a closed-loop learning system that adapts to individual user performance. AI algorithms analyze user input during practice sessions and identify errors in timing, finger position, and dynamics. Corrective feedback is provided in real time, allowing learners to iteratively refine their skills. This adaptive mechanism ensures continuous improvement while maintaining user engagement through gamified elements and interactive lessons.

[0052] The experimental results demonstrate the effectiveness of the proposed system in improving learning outcomes for beginners. Tests with popular songs such as "Happy Birthday" and "Jingle Bells" show a significant reduction in cognitive load and faster mastery of musical notation. The dynamic highlighting feature synchronizes with the timing of each song, allowing users to intuitively follow along without prior training. This innovation underscores the potential of combining OMR and VR technologies to revolutionize music education.

[0053] The present invention integrates optical music recognition (OMR) and virtual reality (VR) to create an immersive piano learning system. The workflow, as described in Fig. The OMR engine, as shown in Section 2 of the document, consists of two main components: (1) the OMR engine, which processes sheet music into structured data, and (2) the VR piano interface, which translates this data into an interactive learning environment. The OMR engine converts scanned or digital sheet music into machine-readable formats (e.g., MusicXML) via a multi-stage pipeline.

[0054] During the preprocessing phase, the system handles rotation correction and image enhancement. Scanned sheets often exhibit slight rotational skew, which impairs staff recognition. To address this problem, the system uses the Hough transform to detect staff lines and calculate the optimal rotation angle. This results in correctly aligned staff lines and ensures accurate symbol segmentation. For image enhancement, low-quality scans suffering from blurriness, noise, or compression artifacts are processed with Real-ESRGAN (Enhanced Super-Resolution GAN). This upscales and sharpens the image, improving the clarity of symbols and staff lines and improving segmentation accuracy. The segmentation phase involves two main processes: staff detection and symbol recognition. A U-Net architecture (U-Net-1) segments staff lines from the background and generates a binary mask.The loss function used combines pixel-wise cross-entropy and the Dice coefficient to compensate for class imbalances. A second U-Net (U-Net-2) isolates musical symbols such as notes, clefs, and rests in a mask. Separating staff lines and symbols reduces interference during recognition. In the post-processing phase, symbol linking and layout analysis are performed. Contour detection is used to group fragmented symbol pixels into contiguous objects, and bounding boxes and centroids are calculated for each symbol cluster. Symbols are then mapped to staff positions to determine pitch based on vertical position and timing based on horizontal position. Heuristics are applied to resolve overlaps, for example, to ensure that no two noteheads are at the same staff position.Finally, the recognized symbols are converted to MusicXML, a standard format for music notation. Post-processing by the OMR engine sorts the symbols by track and time, aligns them across the staves, and inserts rests for rhythm synchronization.

[0055] The VR component, created in Unity 3D and deployed on a VR headset, transforms MusicXML data into an interactive piano learning environment. A realistic 3D piano model is created with individual colliders and animations for each key. The keys are labeled "PianoKey" and linked to audio clips for sound playback. Meta Quest 3 tracks finger positions using sphere colliders on the fingertips (labeled "FingerTip"), and collision detection triggers keystrokes when the fingertips cross the key colliders. For dynamic key highlighting, the Piano Highlight Manager mechanism analyzes MusicXML data to generate a sequence of keystrokes. The keys are highlighted in real time using color or material changes, synchronized to the tempo of the song.The feedback mechanism includes visual and auditory responses: When a correct key is pressed, it turns green and plays the note; when an incorrect key is pressed, it flashes red, and an error counter increments. Spatial sound is generated by AudioSource.PlayClipAtPoint, which creates 3D sound at the key position, simulating a physical piano. Sound decay is modeled using Unity's audio damping settings, allowing the notes to fade out naturally. The integration of OMR and VR forms a closed workflow that connects note interpretation and piano playing. The data flow proceeds from the notes through OMR preprocessing and MusicXML generation, followed by VR key highlighting, user interaction, and real-time feedback. The system supports adaptive learning by tracking user performance (timing and accuracy) and dynamically adjusting the difficulty level.For example, it slows down the tempo for complex sections or repeats problematic bars. Gamification is integrated by awarding points for correct play and tracking high scores to motivate learners.

[0056] Fig. 3 illustrates a diagram showing the end-to-end pipeline of the proposed system according to an embodiment of the present disclosure.

[0057] Fig.Figure 3 shows the capture of music notes for processing. The OMR (Optical Music Recognition) subsystem translates the physical notation of the captured sheet music into digital data. The immersive VR environment allows learners to interact with the virtual piano. This environment contains interactive components that respond to both the processed music data and user input. The central processing engine, the OMR engine, uses computer vision and machine learning to analyze the music sheets. This component implements staff and symbol recognition to interpret the musical notation from the enhanced image inputs. The output of the OMR algorithm is a segmented image that shows the recognized musical elements highlighted in color. This visualization illustrates how the system identifies individual notes, rests, clefs, and other musical symbols in the sheet music.A processing module called the "Symbol-to-XML Converter" then converts the recognized musical symbols into structured data. This component maps spatial relationships between recognized symbols into the timing and pitch information required for playback. The standardized exchange format used is MusicXML, which contains the structured representation of the sheet music. This XML-based format encodes all musical information such as notes, duration, and dynamics in a machine-readable format that preserves musical relationships. Within the VR environment, a detailed 3D model of a piano, the so-called virtual piano, serves as the primary interface for user interaction. This component includes individual key models with associated colliders and animations. The Piano Key Identifier acts as a mapping component that translates MusicXML data into specific piano key instructions.This module determines which keys should be highlighted and when, based on the temporal information in the notation. The highlighted piano key serves as a visual feedback mechanism, displaying illuminated keys that guide the user through piano playing. The yellow highlighted key indicates the next note to be played according to the music sheet. Finally, user interaction represents the phase in which learners interact with the virtual piano using hand-tracking technology. This component forms the closed feedback loop in which user performance is captured and evaluated against the original notation.

[0058] The drawings and the foregoing description illustrate examples of embodiments. Those skilled in the art will recognize that one or more of the described elements may well be combined into a single functional element. Alternatively, certain elements may be separated into multiple functional elements. Elements of one embodiment may be added to another embodiment. For example, the order of the processes described herein may be changed and is not limited to the manner described herein. Furthermore, the actions of a flowchart need not be performed in the order shown; nor do all actions need to be performed. Also, actions that are not dependent on other actions may be performed in parallel with the other actions. The scope of the embodiments is in no way limited by these specific examples.Numerous variations, whether explicitly stated in the specification or not, such as differences in structure, dimensions, and use of materials, are possible. The scope of the embodiments is at least as broad as indicated in the following claims.

[0059] Advantages, further benefits, and solutions to problems have been described above with reference to specific embodiments. However, the advantages, advantages, solutions to problems, and any components that may result in or enhance an advantage, advantage, or solution are not to be construed as critical, required, or essential features or components of any or all of the claims. REFERENCES 100 The system includes: an optical music recognition (OMR) engine. 102 Optical Music Recognition Engine (OMR) 102a Preprocessing module 102b Segmentation module 102c Post-processing module 104 Virtual Reality (VR) headset 104a Finger tracking sensors 104b haptic feedback generators 104c speakers 106 Virtual Reality Piano Interface Module 106a Piano Highlight Manager Mechanism 106b Feedback mechanism 106c surround sound module 108 Computer device 110 Gamification Module 202 Integration of OMR and VR 204 OMR engine 204a Preprocessing 204b Segmentation 204c Post-processing 206 Virtual Piano Setup 206a VR piano interface 206b Dynamic key highlighting 206c Spatial Audio 208 Musicxml 302 Real World 304 Staff notation 306 Implemented Omr algorithm 308 Segmented image 310 Symbol to XML Converter 312 Virtual World 314 Piano key identification 316 Virtual Piano 318 Highlighted piano key 320 User Interaction

Claims

[1] A system for closed-loop piano learning in augmented reality (XR), consisting of: an optical music recognition (OMR) engine configured to receive note inputs, preprocess the note inputs through rotation correction and image enhancement, segment the preprocessed notes into staves and musical symbols using dual neural network architectures, analyze the position and relationship of the segmented musical symbols, and generate a structured MusicXML file based on the analyzed musical symbols; a virtual reality (VR) headset configured to display a rendered virtual environment including a virtual piano, track a user's hand movements, and register the user's tracked hand movements; a virtual reality piano interface module connected to the VR headset, the virtual reality piano interface module configured to: render a three-dimensional virtual piano model with individual interactive keys; dynamically highlight specific piano keys based on the sequential piano key instructions derived from the MusicXML file generated by the OMR engine; detect collisions between virtual representations of the user's fingers and the virtual piano keys as registered via the VR headset; provide real-time auditory, visual, and haptic feedback based on the detected collisions; and adjust the difficulty level of the piano lesson based on the user's performance; and a computing device comprising a processor and a GPU, connected to a memory, and configured to control the operation of a virtual reality piano interface module and a VR headset, wherein the computing device is configured to implement a virtual reality piano interface, process the MusicXML file to create sequential piano key instructions, transmit visual rendering data to the VR headset via the interface module, and process hand tracking data received from the VR headset. [2] The system of claim 1, wherein the OMR engine comprises: a preprocessing module configured to perform rotation correction and image enhancement, wherein a Real-Enhanced Super-Resolution Generative Adversarial Network (Real-ESRGAN) is implemented to perform image enhancement by upscaling and sharpening the note input; and a segmentation module configured to isolate and identify note symbols, wherein staff detection and symbol detection are used. [3] The system of claim 2, wherein the segmentation module of the OMR engine comprises: a first U-Net configured to segment staff lines from the background into a binary mask; and a second U-Net configured to isolate musical symbols such as notes, clefs, and rests in a separate mask; wherein the dual architecture reduces interference between staff lines and symbols during recognition. [4] The system of claim 1, wherein the OMR engine further comprises a post-processing module configured to perform symbol linking, layout analysis, and generation of a MusicXML data file, wherein recognized symbols are converted into a MusicXML data file that represents a standard format for musical notation, wherein the post-processing module, during generation of the MusicXML data file, sorts the recognized symbols by track and time, aligns them across staves, and inserts rests to synchronize rhythms, and wherein the generated data file is transmitted to the computing device. [5] The system of claim 1, wherein the VR headset comprises: finger tracking sensors configured to track positions and movements of the user's individual fingers; and haptic feedback generators configured to provide tactile sensations that simulate key resistance when virtual keys are pressed, wherein the VR headset uses ball colliders assigned to virtual fingertips and labeled as FingerTip for collision detection with virtual piano keys, wherein a key press is triggered when the fingertips cross the key colliders of the specific piano keys. [6] The system of claim 1, wherein the virtual reality piano interface module is connected to the VR headset for control, the interface module being controlled by the computing device, the virtual reality piano interface module being configured to display a 3D model of a virtual piano via the VR headset, the virtual piano model being created with individual colliders and animations for each key, and the keys being labeled as "PianoKey" and associated with audio clips for sound playback controlled by the computing device. [7] The system of claim 1, wherein the virtual reality piano interface module is configured to perform dynamic key highlighting; wherein a piano highlight manager mechanism configures the interface module to analyze the MusicXML file to generate a key sequence, synchronize the key highlighting to the tempo of a piece of music, and change the color or material of virtual piano keys to indicate which keys should be pressed at specific times; and wherein a feedback mechanism configures the interface module to display a green color on a virtual piano key when the key is correctly pressed by the user, display a red flashing color on a virtual piano key when the key is incorrectly pressed by the user, and increment an error counter when an incorrect key is pressed. [8] The system of claim 1, wherein the VR piano interface module further comprises: a spatial audio module configured to: generate three-dimensional sound at a position of each virtual piano key when the key is pressed; simulate natural sound decay using audio attenuation settings; and provide audible feedback corresponding to the pitch and velocity of each key press, wherein the function of the spatial audio module is controlled by the computing device after receiving the real-time hand tracking and key press data from the VR headset, the VR headset comprising speakers that enable the user to hear the output sound. [9] The system of claim 1, wherein the computing device is further configured to implement adaptive learning, wherein the computing device: tracks the user's performance metrics, including timing accuracy, keystroke precision, and rhythm consistency; adjusts tempo for complex sections based on error rates; repeats problematic actions until a predetermined mastery threshold is reached; and generates progress reports documenting the user's improvement over time, and wherein the implementation of the adaptive learning occurs via a virtual reality piano interface module connected to the VR headset. [10] The system of claim 1, further comprising: a gamification module connected to the computing device for integration with the virtual reality piano interface module, the gamification module configured to award points for correct keystrokes based on timing and accuracy, track high scores to motivate continued practice, award achievement badges for completing songs or reaching skill milestones, and provide a virtual conductor avatar to provide encouragement and guidance during practice sessions.

Images