Smart musical instrument aid and method

Abstract

The application discloses an intelligent musical instrument auxiliary device and method, and relates to the technical field of musical instruments. A specific implementation of the device comprises: an audio collector for collecting sound; a processor for equally dividing a waveform graph of the sound by beats to form a beat axis, and superimposing vertical line marks at beat points of the beat axis; real-time detection of the volume and pitch of the sound to determine pitch deviation; a display for displaying the waveform graph of the sound in the form of a beat axis, the horizontal axis of the waveform graph being equally divided by beats, and the vertical axis representing sound amplitude; display of the volume in each beat section; and display of the pitch, the pitch deviation, and a range to which the pitch deviation belongs in a preset color. The implementation provides visual feedback and targeted performance guidance.

Description

Technical Field

[0001] This invention relates to the field of musical instrument technology, and in particular to an intelligent musical instrument auxiliary device and method. Background Technology

[0002] For traditional stringed instruments such as the erhu and pipa, bowing, playing, pitch training, rhythm control, and fingering are the core challenges in the learning process. Traditional practice tools, such as pitch meters, only have basic pitch measurement functions.

[0003] In the process of realizing this invention, the inventors discovered that the prior art has at least the following problems: pitch meters cannot provide visual feedback and targeted performance guidance. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an intelligent musical instrument assistive device, method, apparatus, and computer-readable medium capable of providing visual feedback and targeted performance guidance.

[0005] To achieve the above objectives, according to one aspect of the present invention, a smart musical instrument auxiliary device is provided, comprising: An audio capture device is used to capture sound. The processor is configured to divide the waveform of the sound into equal parts according to the beat to form a beat axis, and to superimpose vertical line marks on the beat points of the beat axis; and to detect the volume and pitch of the sound in real time to determine the pitch deviation. A display for displaying a waveform of the sound in the form of a beat axis, wherein the horizontal axis of the waveform is divided equally by beats and the vertical axis represents the sound amplitude; displaying the volume within each beat segment; and displaying the pitch, the pitch deviation, and the range to which the pitch deviation belongs in a preset color.

[0006] The processor is also used to determine the fingering position of the virtual fingerboard based on the pitch; The display is used to highlight the finger positions on the virtual fingerboard; And / or, The pitch range includes the normal pitch range marked in green, the slightly deviated pitch range marked in yellow, and the severely deviated pitch range marked in red.

[0007] The processor is also configured to generate a directional fine-tuning indication when a pitch deviation is detected to exceed a preset threshold and the duration reaches a preset duration. The directional fine-tuning indication includes a highlighted suggested string position shifted in the string correction direction. And / or, The directional fine-tuning indication is displayed on the monitor, and / or the corrective guidance information for the finger position is played through a player.

[0008] The processor receives an ear training interaction mode instruction and responds to the user's touch operation on the virtual fingerboard. When the touch duration is less than a preset threshold, it plays a short pitch sample of the corresponding position; when the touch duration reaches the preset threshold, it continues to play a long pitch sample of the corresponding position until the touch ends. The ear training interactive mode includes one or more of the following: ear training mode, interval imitation, chord imitation, and melody imitation. In the ear training interaction mode, when the ear training mode is included, the display is used to highlight the corresponding solfège name on the virtual fingerboard in response to the pitch; the player plays the pitch sample corresponding to the pitch.

[0009] The processor is used to determine the current pitch distribution range by using a sliding window algorithm for continuously acquired pitch sequences. By comparing the pitch distribution range with the position database, the display mode of the key and position corresponding to the current performance is switched; wherein, when switching positions, the display of the virtual fingerboard achieves a smooth transition from the current position to the target position through gradual transparency.

[0010] The processor recognizes and executes the voice control commands collected by the audio collector, which include one or more of the following: turning on the metronome, switching instrument modes, selecting a key, and starting / stopping the ear training mode; And / or, It also includes physical buttons, the mapping relationship of which is determined according to user settings.

[0011] In response to two tapping signals within a preset tapping time period, the processor determines the number of tapping beats based on the time interval between the two tapping signals; The display shows the number of taps and the target number of taps entered by the user; And / or, In response to multiple tapping signals within a preset tapping time period, the processor determines the real-time tapping beat count based on the time interval between two consecutive tapping signals. The display shows the real-time tapping beat count.

[0012] Also includes: A metronome is used to respond to beat operation commands, set the beat speed, set the accent, or set a beat flashing synchronization prompt; A timer is used in response to timing operation commands to set practice reminders and set segmented timings; A tuner is used to set the tuning parameters of an instrument in response to tuning commands.

[0013] The processor is also used to adjust the time length of each beat segment in the beat axis in real time according to the set beat speed and beat type, so that the waveform is synchronized with the beat generated by the metronome. And / or, The processor's position database includes position data for stringed instruments, and the position data includes the correspondence between pitch and finger position for each key. The processor is also used to load the corresponding finger position data according to the instrument type and key, and display the corresponding finger positions on the virtual fingerboard.

[0014] According to a second aspect of the present invention, a smart musical instrument assistance method is provided, comprising: Sound acquisition; The waveform of the sound is divided into equal parts according to the beat to form a beat axis, and vertical lines are superimposed on the beat points of the beat axis to display the waveform of the sound in the form of a beat axis. The horizontal axis of the waveform is divided into equal parts according to the beat, and the vertical axis represents the sound amplitude. The volume and pitch of the sound are detected in real time to determine the pitch deviation. The volume is displayed in each beat segment, and the pitch, the pitch deviation, and the range to which the pitch deviation belongs are displayed in a preset color.

[0015] According to a third aspect of the present invention, a smart musical instrument auxiliary electronic device is provided, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0016] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described above.

[0017] One embodiment of the above invention has the following advantages or beneficial effects: an audio acquisition device for acquiring sound; a processor for dividing the waveform of the sound into equal parts according to beats to form a beat axis, and superimposing vertical lines on the beat points of the beat axis; real-time detection of the volume and pitch of the sound to determine the pitch deviation; a display for displaying the waveform of the sound in the form of a beat axis, wherein the horizontal axis of the waveform is divided into equal parts according to beats, and the vertical axis represents the sound amplitude; displaying the volume within each beat segment; and displaying the pitch, the pitch deviation, and the range to which the pitch deviation belongs in a preset color. Based on the content displayed on the display, visual feedback and targeted performance guidance can be provided.

[0018] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0019] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a schematic diagram of the structure of an intelligent musical instrument auxiliary device according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the main interface according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the pitch interface according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the rhythm training interface according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the finger technique learning interface according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the tuning interface according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the settings interface according to an embodiment of the present invention; Figure 8 This is a hardware connection diagram of an intelligent musical instrument auxiliary device according to an embodiment of the present invention; Figure 9 This is a schematic diagram of the software architecture in an intelligent musical instrument auxiliary device according to an embodiment of the present invention; Figure 10 This is an interactive schematic diagram of an intelligent musical instrument auxiliary device according to an embodiment of the present invention; Figure 11 This is a schematic diagram illustrating the use of an intelligent musical instrument auxiliary device applied to the erhu according to an embodiment of the present invention; Figure 12 This is a schematic diagram of the position switching according to an embodiment of the present invention; Figure 13 This is a schematic diagram of the ear training mode according to an embodiment of the present invention; Figure 14 This is a flowchart illustrating the intelligent musical instrument assistance method according to an embodiment of the present invention; Figure 15 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied; Figure 16 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0020] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] Traditional practice tools have the following obvious drawbacks: First, the visualization is low. Traditional waveform displays use a time axis, making it difficult for performers to establish an intuitive correspondence between the sound waveform and the rhythm and beat of their performance. Performers cannot intuitively perceive the sonic characteristics within each beat, and it is even more difficult to judge the rhythmic stability of their performance.

[0022] Secondly, it lacks interactivity. It can only display pitch deviation values ​​and cannot provide concrete fingering and position guidance based on the characteristics of the instrument, making it difficult for players to quickly master the correct string positions.

[0023] Third, its functionality is limited. It only has pitch measurement capabilities and lacks a metronome and timer. While some pitch meters on the market integrate a metronome and timer, the metronome cannot work simultaneously with pitch monitoring, and practical auxiliary functions such as voice control are unavailable. Users would need to carry multiple devices, limiting its usability.

[0024] Fourth, the user experience is poor. Existing devices are complicated to operate and have unfriendly interfaces, making them particularly unsuitable for middle-aged and elderly musical instrument enthusiasts.

[0025] It is evident that pitch meters cannot provide visual feedback or targeted performance guidance.

[0026] To provide visual feedback and targeted performance guidance, the following technical solutions from the embodiments of the present invention can be adopted.

[0027] See Figure 1 , Figure 1 This is a schematic diagram of the structure of an intelligent musical instrument auxiliary device according to an embodiment of the present invention. Specifically, it includes: Audio acquisition device 101 is used to acquire sound. As an example, audio acquisition device 101 uses an I2S interface digital microphone with a sampling rate of 48kHz and a sampling precision of 16bit, and is able to acquire the sound signal produced by musical instrument performance in real time and convert it into digital audio data.

[0028] The processor 102 is the core control and computing unit of the intelligent musical instrument auxiliary device. The processor 102 adopts a 32-bit microcontroller with a hardware floating-point unit, a main frequency of not less than 200MHz, and at least 256KB of SRAM and 2MB of Flash storage space, which has sufficient computing power to support tasks such as real-time pitch detection, beat analysis and waveform rendering.

[0029] Processor 102 is used to divide the sound waveform into equal parts according to the beat to form a beat axis. Specifically, processor 102 receives sound from audio acquisition unit 101 and analyzes and processes the sound signal using built-in real-time pitch detection algorithm and beat detection algorithm. Regarding waveform display, processor 102 replaces the time axis of the traditional sound waveform with a beat axis, aligning the sound signal with the music beat in real time. That is, it divides the waveform into several beat segments along the horizontal axis according to the currently set beat, so that the waveform within each beat segment corresponds to the time position of that beat.

[0030] The processor 102 also superimposes vertical line markers on the beat points of the beat axis. For example, the red vertical line in the beat axis waveform represents the first beat, and this red vertical line can be displayed by flashing. The red vertical line is used to visually distinguish the start time of the beat, and users can easily align the playing waveform with the beat by observing the red vertical line, thus achieving visual cues. Moreover, the beat sound can be output at the first beat to achieve auditory cues.

[0031] The processor 102 is based on real-time detection of sound volume and pitch. Specifically, the processor 102 performs real-time volume analysis on the sound within each beat segment to determine the sound volume. At the same time, it performs pitch analysis on the sound within each beat segment, calculating the pitch deviation between the actual played pitch and the standard pitch.

[0032] Display 103 is electrically connected to processor 102. As an example, display 103 is a 3.5-inch capacitive touchscreen, used to display various visual information and to receive user touch input.

[0033] The display 103 is used to display the waveform of sound in the form of a beat axis. Specifically, after receiving the waveform data processed by the processor 102, the display 103 draws a beat axis waveform on the display area, where the vertical axis of the waveform represents the sound amplitude, and the horizontal axis is divided equally according to the beat, forming a visual waveform display synchronized with the music beat. Users can intuitively view the sound waveform characteristics within each beat through the interface. For example, the waveform of the generated sound is drawn and rendered in real time based on the sound amplitude, i.e., the volume.

[0034] The display 103 is also used to display pitch, pitch deviation, and the range to which the pitch deviation belongs in a preset color.

[0035] Through the coordinated operation of the audio acquisition unit 101, processor 102, and display 103, synchronized visualization of the beat is achieved. During performance, users can simultaneously observe the beat axis waveform to judge the accuracy of the rhythm, the evenness of the sound, and the pitch, thereby significantly improving practice efficiency and performance quality.

[0036] See Figure 2 , Figure 2 This is a schematic diagram of the main interface according to an embodiment of the present invention. The top of the main interface displays status information such as battery level, volume, and current mode. Above the main interface is the pitch deviation, below is a waveform graph, and at the bottom are function switching buttons. The pitch deviation arrow points to different colored areas in real time according to the pitch deviation. Figure 2 The upper part of the left-hand panel displays the real-time pitch deviation bar, the middle part displays the beat axis waveform, and the lower part displays the beat and timing, as well as mode selection, etc. Figure 2 The image on the right is a schematic diagram of the beat axis waveform. It displays the current sound waveform. The red vertical line in the beat axis waveform represents the first beat, for easy alignment.

[0037] In one embodiment of the present invention, the processor 102 further indicates pitch deviation by pointing to an arrow. This allows the user to determine the degree of pitch deviation through the color indicated by the arrow, i.e., the range of pitch deviation. As an example, the pitch range includes a normal pitch range indicated by green, a slightly deviated pitch range indicated by yellow, and a severely deviated pitch range indicated by red. Green indicates accurate pitch, yellow indicates a small pitch deviation, and red indicates a large pitch deviation, allowing the user to quickly determine the range of pitch deviation by color, thus providing a multi-dimensional and intuitive presentation of pitch accuracy.

[0038] By combining the aforementioned color types, multi-level visual feedback on pitch deviations is achieved. Users can quickly identify pitch deviations during performance, allowing them to focus more on the performance itself and effectively improve practice efficiency and pitch control.

[0039] See Figure 3 , Figure 3 This is a schematic diagram of the pitch interface according to an embodiment of the present invention. The information bar at the top of the pitch interface displays the selected instrument type. The pitch interface also displays the current interactive pitch, key selection and position interaction, ear training, virtual fingerboard, etc.

[0040] In one embodiment of the present invention, the processor 102 also has an indication function, capable of actively providing correction guidance when the player's pitch continuously deviates, helping the user to quickly correct the fingering position. The processor 102 is also used to determine the fingering position on the virtual fingerboard based on pitch. The display 103 is used to highlight the fingering position on the virtual fingerboard. The processor 102 generates a directional fine-tuning instruction when a pitch deviation exceeding a preset threshold and lasting for a preset duration is detected. During real-time pitch analysis, the processor 102 continuously tracks and times the pitch deviation. When the duration of a continuous pitch deviation exceeds the preset threshold and the duration reaches the preset duration, the processor 102 determines that the user is currently in a state of continuous pitch deviation, thereby generating a directional fine-tuning instruction. This dual judgment based on continuity and duration effectively avoids frequent triggering of unnecessary prompts due to instantaneous deviations of individual notes, ensuring the timeliness of guidance.

[0041] The directional fine-tuning indication includes shifting the highlighted suggested fingering position in the direction of fingering correction. When the processor 102 detects that the playing pitch is consistently too high, it determines that the fingering position may have shifted. At this time, the processor 102 controls the highlighted display position on the display 103 to shift in the direction of fingering correction, thus visually prompting the user to move in that direction. The user does not need to understand abstract pitch deviation values; they can achieve pitch correction simply by following the visual guidance.

[0042] And / or, directional fine-tuning indication includes displaying directional fine-tuning indication via display 103, and / or playing correction guidance information for the strumming position via player 104.

[0043] Directional fine-tuning instructions include displaying text prompts on the display 103. For example, when the processor 102 determines that the pitch is too low and the finger needs to be moved upward, it displays "move finger upward" on the display 103, attracting the user's attention with flashing text and indicating the specific direction of the correction action. Furthermore, when the pitch is out of range, a voice prompt is activated, and the player 104 plays a guiding voice message: "Please move your finger upward," achieving dual correction guidance through both visual and auditory means.

[0044] Through the above embodiments, the abstract string position correction is transformed into a highlighted suggested string position and directional fine-tuning indication, enabling users to obtain intuitive and easy-to-understand real-time guidance during performance and improving practice efficiency.

[0045] The processor 102 is also equipped with an ear training interactive mode, which helps users build pitch auditory memory and improve pitch control through dual visual and auditory interaction.

[0046] The processor 102 receives an instruction for the ear training interactive mode. The ear training interactive mode instruction can be given by the user via a mode selection button on the touch display 103 or via voice command. The ear training mode is designed for instrumental music users, guiding them to imitate and practice using preset standard pitch samples to help them build auditory pitch memory and improve their pitch control. It is fully suitable for learning traditional Chinese instruments such as the erhu and pipa. Upon entering the ear training interactive mode, the display 103 shows the virtual fingerboard of the currently selected instrument for the user to touch and operate.

[0047] In the ear training interactive mode, the processor 102 responds to the user's touch operation on the virtual fretboard, distinguishing between short and long notes based on the touch duration. Specifically, when the user touches a string position on the virtual fretboard, the processor 102 detects the duration of the touch and compares it with a preset threshold of 2 seconds. When the detected touch duration is less than 2 seconds, a short pitch sample for the corresponding position is played; when the detected touch duration is ≥ 2 seconds, a long pitch sample for the corresponding position is continuously played until the touch ends. The processor 102 obtains corresponding samples from a built-in standard pitch database for each key and position of the target instrument. The frequency error of this database is ≤ 0.1 cents, ensuring the accuracy of the ear training baseline. Generating corresponding standard pitches by touching the virtual fretboard is essentially an instant pitch feedback sound, similar to the principle of producing a pitch sound by pressing a note with a finger. By detecting the corresponding pitch, illuminating the note, and simultaneously emitting the pitch sound, the user can listen and imitate, forming a memory association between auditory and motor perception. In another ear training mode, when the performer produces a pitch sound by operating the instrument, the corresponding standard pitch is automatically played as feedback. For example, when the performer plays the erhu, if the corresponding pitch on the fingerboard is detected while bowing, the corresponding standard pitch is automatically played, thus synchronizing bowing practice with ear training.

[0048] Furthermore, the ear training interactive mode includes one or more of the following: ear training mode, interval imitation, chord imitation, and melody imitation. It supports graded ear training modes for interval, chord, and melody imitation, providing a progressive auditory training path for users at different learning stages. Through these graded training modes, users can gradually improve their comprehensive musical abilities in listening, distinguishing, and imitating sounds, progressing from single intervals to complex melodies. As an example, in ear training mode, display 103, in response to pitch, highlights the corresponding solfège syllable on the virtual fretboard. Simultaneously, player 104 plays pitch samples corresponding to the pitches, thus achieving ear training.

[0049] In one embodiment of the present invention, the processor 102 can identify the current key range of the performance and switch the position display according to the pitch sequence actually played by the user, eliminating the tedious operation of manually switching keys or positions during the performance and improving the continuity of practice.

[0050] Processor 102 is used to determine the current pitch distribution range by employing a sliding window algorithm for continuously acquired pitch sequences. Specifically, it continuously acquires pitch sequences and analyzes the current pitch distribution range using a sliding window algorithm. Processor 102 acquires real-time pitch detection results at a fixed sampling period and stores multiple consecutive pitch data points into a sliding window. This sliding window maintains pitch data from the most recent period and is continuously updated over time: each time a new pitch data point is acquired, the window slides forward one position, removing the oldest data point. Processor 102 performs statistical analysis on the pitch data within the window, such as calculating the maximum and minimum values ​​of the pitch distribution and the main concentrated areas, thereby determining the current pitch distribution range.

[0051] The processor 102 can also dynamically adjust the number of notes or the duration covered by the sliding window according to user settings. This ensures that the pitch data collected within the window can fully reflect the pitch distribution, avoiding unstable recognition due to an excessively small window or delayed switching response due to an excessively large window, thereby improving the accuracy and timeliness of position matching.

[0052] After determining the current pitch distribution range, the processor 102 compares the pitch distribution range with the position database and switches to the mode and position display mode corresponding to the current performance. The processor 102 has a built-in complete position database, which stores the pitch range and finger position correspondence of different instruments in different modes for each position.

[0053] When switching shifters, the virtual fretboard display achieves a smooth transition from the current shifter to the target shifter through a gradual change in transparency. Upon receiving the shifter switching command, the processor 102 does not directly replace the virtual fretboard image. Instead, it generates a series of intermediate transition frames. In each frame, the transparency of the current shifter image gradually decreases, while the transparency of the target shifter image gradually increases, thus achieving a visually smooth transition. This gradual transparency transition effectively avoids visual discomfort and distraction caused by abrupt changes in screen display.

[0054] Through the above analysis of the sliding window, seamless automatic switching of key and position is achieved during the performance, allowing users to continue to focus on the performance itself.

[0055] In one embodiment of the present invention, the intelligent musical instrument auxiliary device supports multiple interactions, and users can operate it through both voice control and physical buttons.

[0056] In terms of voice control, the processor 102 recognizes and executes voice control commands collected by the audio acquisition unit 101. During performance, the audio acquisition unit 101 continuously collects ambient sounds. When the user issues a voice command, it recognizes keywords in the collected sounds and transmits the recognition results to the processor 102. The processor 102 executes corresponding operations based on the recognition results. Voice control commands include one or more of the following: turning on the metronome, switching instrument modes, selecting a key, and starting / stopping the ear training mode. As an example, full-function voice control is achieved through the Weichuang voice recognition chip, supporting various voice commands such as turning on the metronome, increasing the volume, switching erhu modes, and selecting a key. It also supports voice control to turn the ear training mode on or off, further enhancing ease of use.

[0057] Regarding physical buttons, the intelligent musical instrument assistive device also features physical buttons. These physical buttons are mapped to frequently used function shortcuts based on user settings. As an example, it is equipped with two customizable physical buttons that can be set as shortcuts for frequently used functions, such as metronome start / stop, tap tempo setting, and recording on / off. Users can access the system settings interface to configure the physical buttons according to their personal usage habits. The settings parameters for the physical buttons are stored in the intelligent musical instrument assistive device's storage unit, and the processor 102 loads the user-preset button mapping configuration each time the device is powered on.

[0058] The combination of voice control and physical buttons provides multiple interaction methods, effectively solving the problem of complex operation of traditional pitch meters.

[0059] In one embodiment of the present invention, the intelligent musical instrument assistive device is further provided with a metronome, a timer and a tuner, which work in conjunction with the beat axis waveform display and fingering guidance to provide users with comprehensive performance assistance.

[0060] The metronome is used to respond to beat operation commands, setting the beat tempo, setting the accent, or setting a synchronized beat flashing indicator. Specifically, users can issue beat operation commands to the processor 102 via the touch screen display 103's metronome setting interface or via voice commands. Regarding beat tempo settings, the metronome supports a tempo adjustment of 40-240 BPM with an accuracy of ±1 BPM. For accent settings, the metronome supports accent settings, allowing users to customize the accent position for each measure. Aurally, the metronome offers various beat timbres, such as woodblock, drum beats, electronic sounds, and vocals. Visually, the metronome provides synchronized beat flashing indicators, providing dual visual and auditory feedback; a designated area or indicator light on the display 103 flashes synchronously with the beat rhythm.

[0061] See Figure 4 , Figure 4This is a schematic diagram of the rhythm training interface according to an embodiment of the present invention. The pitch deviation indicator area is located below the top status bar, prominently displaying the real-time pitch deviation status of the current performance. The current pitch deviation value is displayed numerically in cents, with positive values ​​indicating too high and negative values ​​indicating too low. A horizontal arc-shaped colored bar displays the pitch deviation range, covering a continuous interval from negative to positive values. The deviation bar uses a gradient color scheme, with the central area in green indicating the accurate range, and the ends gradually changing to yellow and red to indicate the deviation range. The cursor position moves in real-time according to the deviation.

[0062] See Figure 5 , Figure 5 This is a schematic diagram of the fingering learning interface according to an embodiment of the present invention. In fingering demonstration mode, the highlighted position is preset and driven by the training program for teaching demonstration; in real-time follow-up mode, the highlighted position is driven in real time by the pitch detection module for practice feedback. The two modes can be switched at any time to meet the needs of different learning stages.

[0063] The timer is used to respond to timing operation commands, set practice reminders, and set segmented timing. The timer supports both countdown and forward modes, with a maximum duration of 99 hours and 59 minutes. It features a practice reminder function, allowing users to set the practice duration and receive an automatic voice prompt for a break when the time is up. The timer also has a segmented timing function, recording the duration of multiple performance segments for analysis of practice effectiveness.

[0064] The tuner is used to set the tuning parameters of instruments in response to tuning operation commands. As an example, it adopts the internationally recognized twelve-tone equal temperament tuning rule, fixing the A4 tone at 440Hz as the reference pitch. All pitch deviation calculations are based on this reference, eliminating the need to recalibrate the reference for different instruments and ensuring consistent tuning accuracy across all scenarios. Users can switch tuning modes through the instrument selection menu on the display 103. The processor 102 loads the corresponding standard pitch parameter table according to the selected instrument type, enabling the tuner to accurately identify and indicate the tuning standard for different instruments.

[0065] See Figure 6 , Figure 6 This is a schematic diagram of the tuning interface according to an embodiment of the present invention. It covers the entire range of the twelve-tone equal temperament. Based on the tuning rules of the twelve-tone equal temperament, it covers the complete 88-key piano range from C1 to C8, and is compatible with the tuning needs of various stringed instruments such as the erhu, pipa, violin, and guitar, wind instruments such as the flute and clarinet, as well as keyboard instruments, achieving multi-functionality.

[0066] By integrating a metronome, timer, and tuner, the device incorporates multiple functions required for instrument practice into a smart instrument aid, effectively solving the problems of traditional pitch meters having limited functionality and application scenarios.

[0067] In one embodiment of the present invention, the processor 102 is also used to realize the linkage between the beat axis waveform display and the metronome, so as to ensure that the waveform diagram is synchronized with the beat generated by the metronome.

[0068] The processor 102 is also used to adjust the duration of each beat segment in the beat axis in real time according to the set beat speed and beat type, so that the waveform is synchronized with the beat generated by the metronome. Specifically, it supports custom beat speed and beat type (2 / 4, 3 / 4, 4 / 4, 6 / 8, etc.), which users can flexibly set according to the requirements of the practice pieces.

[0069] Through the aforementioned real-time adjustment mechanism, the processor 102 aligns the waveform data with the beat and draws it onto the beat axis waveform display area. Regardless of the speed at which the user plays, they can observe the intuitive effect of the waveform changing synchronously with the beat on the screen, providing an accurate visual reference for rhythm training.

[0070] In one embodiment of the present invention, considering that general tapping speed measurement requires collecting multiple tapping data and taking the average value, it is mainly used to obtain the objective playing speed of music, adapt to the arrangement, and align the band with the accompaniment. If the average value is used in the basic skills training scenario, it will mask the defects of rhythm fluctuation. That is, the mean processing cannot reflect the rhythm fluctuation, which can easily create the illusion that the sense of rhythm has been met, and is not conducive to the long-term training of beat perception.

[0071] In this embodiment of the invention, the rhythm level is realistically fed back based on the tapping signal, making it suitable for rhythm training of various string, wind, and keyboard instruments.

[0072] Specifically, it includes a beat training mode and a beat assist mode.

[0073] In beat training mode, in response to two tapping signals within a preset tapping time period, processor 102 determines the number of tapping beats based on the time interval between the two tapping signals; display 103 displays the number of tapping beats and the target number of taps input by the user.

[0074] After activating the beat training mode, the processor 102 monitors the tapping signals and records the time points of two consecutive tapping signals within a preset tapping time period. The number of tapping beats is directly determined based on the time interval between the two tapping signals. In beat training mode, no further tapping signals are collected within the preset tapping time period, eliminating the need for averaging multiple sets of data or filtering and smoothing. The display 103 shows the number of tapping beats and the target number of taps input by the user. This provides a clear visual comparison of the target number of taps and the actual number of tapping beats, facilitating beat deviation comparison for the user.

[0075] In beat-assisted mode, in response to multiple tapping signals within a preset tapping time period, processor 102 determines the real-time tapping beat count based on the time interval between two consecutive tapping signals; display 103 displays the real-time tapping beat count.

[0076] For example, if there are 5 tapping signals within a preset tapping time period, calculate the real-time tapping beats for the first and second tapping signals; the real-time tapping beats for the second and third tapping signals; the real-time tapping beats for the third and fourth tapping signals; and the real-time tapping beats for the fourth and fifth tapping signals. The number of beats tapped in real time changes with the number of beats tapped in the signal, allowing users to adjust the beat independently, thereby solving the problems of rushing or dragging beats during the performance and forming a complete rhythm training system.

[0077] In one embodiment of the present invention, the position database of the processor 102 includes position data for stringed instruments, such as the erhu and / or pipa. The position database includes the correspondence between pitch and finger position for each key.

[0078] The processor 102 is also used to load corresponding position data according to the instrument type and key, and display the corresponding finger positions on the virtual fingerboard. Specifically, the user first specifies the instrument type and key using the instrument through the instrument selection interface on the display 103 or voice control commands. After receiving the command, the processor 102 retrieves and loads position data matching the instrument type and key from the position database. Subsequently, the processor 102 controls the display 103 to display the virtual fingerboard graphic of the instrument in the selected key, and marks the finger positions for each pitch at the corresponding positions on the virtual fingerboard according to the loaded position data. When entering learning mode, the screen displays the correct position position corresponding to the currently played pitch in real time, highlighting suggested finger positions, thereby achieving dynamic mapping of the played pitch to the corresponding instrument's virtual fingerboard through real-time pitch analysis, and achieving accuracy of the virtual fingerboard with the help of real-time pitch accuracy. By observing the highlighted marks on the virtual fingerboard, the user can intuitively understand the correct position to press the string, significantly reducing the learning threshold for the user.

[0079] In one embodiment of the present invention, the intelligent musical instrument auxiliary device is also equipped with a low-power power management system to ensure that the intelligent musical instrument auxiliary device operates reliably for a long time.

[0080] The intelligent musical instrument auxiliary device features a built-in 1500-1800mAh high-capacity lithium battery, providing over 10 hours of continuous use and over 30 days of standby time. It supports both Type-C wired fast charging and Qi standard wireless charging. Intelligent power management automatically adjusts screen brightness and processor frequency to extend battery life. It also features automatic low battery reminders, with a voice prompt to charge when the battery level reaches 10%.

[0081] See Figure 7 , Figure 7This is a schematic diagram of the settings interface according to an embodiment of the present invention. The settings interface allows adjustment of volume, metronome sound, metronome flashing display, and ear training mode sound, etc.

[0082] See Figure 8 , Figure 8 This is a hardware connection diagram of an intelligent musical instrument auxiliary device according to an embodiment of the present invention. Figure 8 See Table 1 for device settings.

[0083] The main controller is the processor 102 in the intelligent musical instrument auxiliary device. As the core data processing unit, the main controller is bidirectionally connected to other modules through different interfaces.

[0084] The input of the power management module is connected to a 1500-1800 mAh lithium battery, which provides power. The output of the power management module is electrically connected to the power supply pins of the main controller, digital microphone, capacitive touch screen, and voice recognition chip, providing a stable operating voltage for the entire device.

[0085] The digital microphone, also known as the audio acquisition unit 101, is used to acquire sound in real time and transmit the digital audio data to the main controller via the I2S bus for pitch analysis and beat detection.

[0086] The touch display unit, or monitor 103, uses a 3.5-inch capacitive touchscreen. This 3.5-inch capacitive touchscreen connects to the main controller via an 8080 parallel interface, has a resolution of 320×480, and supports multi-touch. The main controller sends RGB data display to the touchscreen via the 8080 parallel interface, showing the performance waveform, position map, and interactive interface. Simultaneously, it receives touch signals from the touchscreen, enabling user gestures such as swiping and clicking.

[0087] The system uses a VICHUAN voice recognition chip to recognize voice control commands. This chip integrates a microphone and speaker unit and supports offline voice recognition for receiving user voice control commands.

[0088] The function buttons are accessed via a button-triggered method to implement custom shortcut key functions and provide physical button input signals to the main controller. The lithium battery also provides power to all the above modules.

[0089] Table 1. Configuration Information of Main Hardware for Intelligent Musical Instrument Assistive Devices

[0090] See Figure 9 , Figure 9 This is a schematic diagram of the software architecture in an intelligent musical instrument auxiliary device according to an embodiment of the present invention.

[0091] The hardware abstraction layer (HAL) is located at the bottom of the software architecture, providing a unified hardware access interface for upper-layer software modules and shielding them from the differences in underlying hardware. This layer includes: audio driver, voice chip driver, and power management driver.

[0092] The basic service layer sits above the hardware abstraction layer, encapsulating various core data processing and basic functional services, and providing reusable functional interfaces for upper-layer business applications. The basic service layer includes: Audio preprocessing module: Performs preprocessing operations such as noise reduction, filtering, and frame segmentation on the raw audio data acquired by the hardware abstraction layer.

[0093] Pitch detection module: Extracts real-time fundamental frequency based on preprocessed audio data.

[0094] Touch event processing module: parses the touch signals reported by the hardware abstraction layer, converts them into standardized interactive events such as click, swipe, long press, etc., and distributes them to the corresponding modules in the upper layer.

[0095] Position Database Service: Manages complete position data for multiple instruments and keys, and provides a retrieval interface for pitch-to-finger position mapping relationships.

[0096] Offline speech recognition module: Receives speech chip recognition results forwarded by the hardware abstraction layer, performs instruction matching and semantic parsing, and outputs standardized control commands; Power Management Service: Obtains battery status information through the power management driver.

[0097] The core business layer includes: Beat axis rendering module: Divides the waveform of the sound into beats to form a beat axis, and overlays vertical line marks at the beat points of the beat axis.

[0098] Waveform color dynamic encoding module: Real-time volume detection.

[0099] The position interaction component includes a position correction prompt module, a fingering guidance module, an automatic position switching module, an ear training interaction service, and a position mapping and matching module.

[0100] Position correction prompt module: Compares the detected real-time pitch with the target key in the position database, and highlights the correct fingering position on the virtual fretboard. When the pitch deviates from the target pitch by more than a preset threshold, it generates text prompts, shifts the highlighted area on the screen, and provides voice prompts.

[0101] Automatic position switching module: It analyzes the pitch distribution range of continuous playing through a sliding window algorithm, automatically compares it with the position database, and automatically switches to the display mode of the corresponding key and position during the performance.

[0102] Ear training interactive service: Built-in standard pitch database for various modes, triggers the corresponding pitch sound based on the duration of the user's touch on the corresponding area of ​​the bitmap.

[0103] Position mapping matching module: maps the real-time detected pitches onto the virtual fingerboard of the instrument.

[0104] The application interaction layer includes: interface rendering module, voice broadcasting module, and training mode management module.

[0105] Interface rendering module: responsible for the overall layout drawing of the interface, data display and refresh, and interface navigation control.

[0106] Voice broadcast module: broadcasts voice messages to provide feedback on practice results, low battery reminders, and reminders when practice time is up.

[0107] Training mode management module: Manages various application scenarios such as sound equalization training, rhythm training mode, pitch training mode, free practice mode, fingering learning mode, and ear training mode.

[0108] See Figure 10 , Figure 10 This is an interactive schematic diagram of an intelligent musical instrument auxiliary device according to an embodiment of the present invention.

[0109] The user triggers the power-on function via touch or button. The main controller responds to the power-on interrupt, performing peripheral initialization and parameter loading. Simultaneously, the display driver module loads the main interface, and the voice module announces "Startup complete".

[0110] The audio acquisition module samples audio at a 48kHz sampling rate, inputting via an I2S interface. The main controller processes the frame data every 10ms. It then calls a pitch detection algorithm, employing the YIN algorithm for fundamental frequency detection, and outputs a standard pitch value after octave correction and pitch conversion.

[0111] The position mapping module performs string position matching and query, and calculates the pitch deviation value.

[0112] First feedback mode: Deviation > 10 cents: It is judged as pitch error, the arrow corresponding to the pitch deviation belongs to the yellow or red area and the corresponding string position is highlighted; the voice module prompts "too high" or "too low".

[0113] Second feedback mode: Deviation ≤ 10 cents: The pitch is considered acceptable, and the arrow corresponding to the pitch deviation is in the green area.

[0114] The main controller receives offline voice recognition commands from the voice module, such as "Start the metronome." After recognition, it parses the command code and loads the metronome parameters, such as tempo 60. The display driver module shows the beat animation, and the voice module plays a confirmation prompt. Subsequently, at each beat, the display module overlays a vertical line mark on the beat point of the beat axis and flashes it as a prompt, while the voice module plays a beat cue tone.

[0115] The user selects the instrument type, such as guitar or violin. The main controller loads the corresponding fret database based on the instrument parameters and displays the fret diagram on the display driver. The main controller matches the current fingering in real time and highlights the next recommended fret on the display module.

[0116] See Figure 11 , Figure 11 This is a schematic diagram of the use of an intelligent musical instrument auxiliary device applied to the erhu according to an embodiment of the present invention. Figure 11 The intelligent musical instrument pitch learning system in this invention uses the intelligent musical instrument auxiliary device in this embodiment to learn the erhu.

[0117] It provides learners with multi-dimensional auditory training functions, including single-note recognition training, melody imitation training, and interval and chord training. By playing standard sound sources and collecting user feedback audio, it automatically compares deviations, thereby training learners' absolute pitch and relative interval perception abilities.

[0118] When a user plays an instrument, real-time audio is captured to calculate pitch deviation. The pitch range corresponding to the pitch deviation is then visually presented to the learner in color, enabling instant visual error correction.

[0119] It captures the audio signal generated by the user's performance in real time, compares the real-time pitch with the expected pitch of the melody of the preset instrument, and automatically calculates and matches the position required for the current section.

[0120] After powering on, the user selects the Erhu mode and D major, loading the Erhu D major position database. The user sets the metronome to 4 / 4 time and the tempo to 60 BPM, activating the metronome function.

[0121] When a user plays the erhu, a digital microphone captures the sound signal in real time and sends it to a pitch detection algorithm. The analysis determines the current pitch is B4, within 10 cents of the correct range, which is displayed in green. Simultaneously, the position display area highlights the position of the first finger on the outer string in the first position of the D major scale, indicating that the user is pressing the string correctly.

[0122] During the performance, the user outputs: "Speed ​​up." After the voice recognition chip recognizes the command, it automatically adjusts the metronome speed to 70 BPM.

[0123] After 30 minutes of practice, the timer reaches the preset duration, and a voice prompt will say: "Practice time is up. We recommend a 5-minute break." After the practice session, users can view statistics such as pitch accuracy, rhythm, and bowing stability.

[0124] See Figure 12 , Figure 12 This is a schematic diagram of the position switching according to an embodiment of the present invention.

[0125] When the learner begins to play the instrument, the audio acquisition unit is triggered to start working. The audio acquisition unit performs high-fidelity acquisition at a high sampling rate of 48kHz, generates PCM audio data, and transmits the PCM audio data to the main control chip in real time through the communication interface.

[0126] After receiving PCM audio data, the main control chip executes an audio preprocessing algorithm. Specifically, the main control chip first uses sliding window framing technology to segment the continuous audio stream into frames and extracts the real-time pitch sequence of each frame. Then, the main control chip uses this real-time pitch sequence as a query condition to send a query request to the position matching database to retrieve position information matching the current pitch distribution range. The position matching database performs a search and analysis based on a preset instrument fingering mapping table, and finally returns the target position ID with the highest matching degree to the main control chip. After obtaining the target position ID, the main control chip compares it with the position currently displayed on the display unit to perform a consistency judgment to determine whether the display guidance needs to be updated.

[0127] When the main control chip determines that the target position is inconsistent with the current display, indicating that a position switch is necessary, it sends a position switch command to the display unit. The display unit responds to the command by providing a visual prompt on the user interface or by driving the speaker to output an audio prompt, until a consistent effect is achieved after the switch. After the switch is complete, the display unit returns a switch completion response signal to the main control chip to ensure system synchronization. The system may also broadcast a voice prompt stating "Switched to {X} position" to the user, and / or display the broadcast content in the information bar.

[0128] See Figure 13 , Figure 13 This is a schematic diagram of the ear training mode according to an embodiment of the present invention. Seamless switching of multimodal training is achieved through a state machine management mode. In standby mode, the main menu is displayed, waiting for the user to select a training type. In any training state, if a user's instruction to return to the main menu is received, the state machine will immediately jump back to the initial standby state.

[0129] Single-tone training

[0130] In response to the user selecting single-tone auditory training from standby mode, the system enters single-tone training mode. In this mode, the system defines the following differentiated feedback logic based on touch duration: When the main control chip detects that the user touches the corresponding position on the instrument for less than 2 seconds, it triggers a standard single-note playback operation. After playback, the blind listening mode is activated. After the user completes the blind listening selection, pitch matching verification is performed. For example, if the erhu is played to produce a D5 pitch, the intelligent instrument assistance device detects this, triggers pitch matching, and plays the standard pitch. If a pitch deviation occurs, and the pitch of the erhu is not audibly the same as the standard pitch, it can be determined that the player's pitch is off, and the string pressing is incorrect. Conversely, when the pitches of the two sounds are exactly the same, they are audibly at the same pitch, indicating that the string pressing is correct. Then, it automatically returns to the initial state of single-note training, waiting for the user to continue to the next round of practice.

[0131] When a user touches the control for a duration of 2 seconds or more, a single-tone loop playback operation is triggered to strengthen the user's auditory memory. The loop playback will continue until a user touch end command is detected, at which point the single-tone playback will stop and the system will return to the initial state of single-tone training.

[0132] Interval training

[0133] In response to the user's selection of interval and chord training, the system enters interval training mode.

[0134] After playing standard intervals or chords, the system collects the user's playing data and executes a pitch deviation analysis algorithm to provide the comparison results to the user.

[0135] Users can choose the target interval or chord, and the corresponding chord will be played according to the user's selection.

[0136] Melody Training

[0137] In response to the user's selection of melody imitation training, the system enters the melody training state.

[0138] Load the practice melody and play segmented demonstrations. After the demonstrations are complete, wait for the user to imitate and play the melody. Compare the user's pitch and rhythm with the standard melody, score the performance, and output the complete melody score.

[0139] During melody training, the practice melody can be automatically reloaded or triggered by the user according to the settings, and the segmented demonstration playback can be resumed.

[0140] See Figure 14 , Figure 14 This is a flowchart illustrating an intelligent musical instrument assistance method according to an embodiment of the present invention. The intelligent musical instrument assistance device can implement the intelligent musical instrument assistance method. Specifically, it includes: S1401. Acquire sound; divide the waveform of the sound into equal parts according to the beat to form a beat axis, and superimpose vertical line marks on the beat points of the beat axis to display the waveform of the sound in the form of a beat axis. The horizontal axis of the waveform is divided into equal parts according to the beat, and the vertical axis represents the sound amplitude.

[0141] S1402. Real-time detection of the volume and pitch of the sound to determine the pitch deviation, displaying the volume in each beat segment, and displaying the pitch, the pitch deviation, and the range to which the pitch deviation belongs in a preset color.

[0142] In one embodiment of the present invention, the fingering position of the virtual fingerboard is determined based on the pitch, and the fingering position is highlighted on the virtual fingerboard; The pitch range includes the normal pitch range marked in green, the slightly deviated pitch range marked in yellow, and the severely deviated pitch range marked in red.

[0143] In one embodiment of the present invention, when a pitch deviation is detected to exceed a preset threshold and the duration reaches a preset duration, a directional fine-tuning indication is generated, the directional fine-tuning indication including a highlighted suggested string pressing position shifted in the string pressing correction direction; And / or, The directional fine-tuning indication is displayed on the monitor, and / or the corrective guidance information for the finger position is played through a player.

[0144] In one embodiment of the present invention, an ear training interaction mode instruction is received, and in response to a user's touch operation on the virtual fingerboard, when the touch duration is less than a preset threshold, a short pitch sample of the corresponding position is played; when the touch duration reaches the preset threshold, a long pitch sample of the corresponding position is continuously played until the touch ends. The ear training interactive mode includes one or more of the following: ear training mode, interval imitation, chord imitation, and melody imitation. In the ear training interaction mode, when the ear training mode is included, the display is used to highlight the corresponding solfège name on the virtual fingerboard in response to the pitch; the player plays the pitch sample corresponding to the pitch.

[0145] In one embodiment of the present invention, the method further includes determining the current pitch distribution range by using a sliding window algorithm for continuously acquired performance pitch sequences; By comparing the pitch distribution range with the position database, the system switches to the key and position display mode corresponding to the current performance. When switching positions, the virtual fingerboard display uses a gradual change in transparency to achieve a smooth transition from the current position to the target position. Furthermore, the virtual fingerboard can also be displayed by retrieving stored position images.

[0146] In one embodiment of the present invention, it is also used to identify and execute voice control commands collected by the audio collector, the voice control commands including one or more of the following: turn on the metronome, switch instrument mode, select key and start / stop ear training mode; It also includes physical buttons, which are mapped to frequently used function shortcuts according to user settings.

[0147] In one embodiment of the present invention, it further includes: a metronome, used to set the beat speed, set the accent, or set a beat flashing synchronization prompt in response to a beat operation command; A timer is used in response to timing operation commands to set practice reminders and set segmented timings; A tuner is used to set the tuning parameters of an instrument in response to tuning commands.

[0148] In one embodiment of the present invention, in response to two tapping signals within a preset tapping time period, the number of tapping beats is determined based on the time interval between the two tapping signals; the number of tapping beats and the target number of tapping beats input by the user are displayed; And / or, In response to multiple tapping signals within a preset tapping time period, the real-time tapping beat count is determined based on the time interval between two consecutive tapping signals; the real-time tapping beat count is then displayed.

[0149] In one embodiment of the present invention, the time length of each beat segment in the beat axis is adjusted in real time according to the set beat speed and beat type, so that the waveform is synchronized with the beat generated by the metronome.

[0150] In one embodiment of the present invention, the finger position database includes finger position data for the erhu and / or pipa, wherein the finger position data includes the correspondence between pitch and finger position for each key. Based on the instrument type and key, the corresponding finger position data is loaded and displayed on the virtual fingerboard.

[0151] Figure 15 An exemplary system architecture 1500 is shown that can be applied to the intelligent musical instrument assistance method or intelligent musical instrument assistance device according to embodiments of the present invention.

[0152] like Figure 15 As shown, system architecture 1500 may include terminal devices 1501, 1502, and 1503, network 1504, and server 1505. Network 1504 is used as a medium to provide a communication link between terminal devices 1501, 1502, and 1503 and server 1505. Network 1504 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0153] Users can use terminal devices 1501, 1502, and 1503 to interact with server 1505 via network 1504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 1501, 1502, and 1503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0154] Terminal devices 1501, 1502, and 1503 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0155] Server 1505 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 1501, 1502, and 1503 (for example only). The backend management server can analyze and process data such as received product information query requests, and feed back the processing results (such as target push information, product information - for example only) to the terminal devices.

[0156] It should be noted that the intelligent musical instrument assistance method provided in this embodiment of the invention is generally executed by server 1505, and correspondingly, the intelligent musical instrument assistance device is generally set in server 1505.

[0157] It should be understood that Figure 15 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0158] The following is for reference. Figure 16 It shows a schematic diagram of the structure of a computer system 1600 suitable for implementing a terminal device of the present invention. Figure 16 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0159] like Figure 16 As shown, the computer system 1600 includes a central processing unit (CPU) 1601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 1602 or programs loaded from storage portion 1608 into random access memory (RAM) 1603. The RAM 1603 also stores various programs and data required for the operation of the system 1600. The CPU 1601, ROM 1602, and RAM 1603 are interconnected via a bus 1604. An input / output (I / O) interface 1605 is also connected to the bus 1604.

[0160] The following components are connected to I / O interface 1605: an input section 1606 including a keyboard, mouse, etc.; an output section 1607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1608 including a hard disk, etc.; and a communication section 1609 including a network interface card such as a LAN card, modem, etc. The communication section 1609 performs communication processing via a network such as the Internet. A drive 1610 is also connected to I / O interface 1605 as needed. Removable media 1611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1610 as needed so that computer programs read from them can be installed into storage section 1608 as needed.

[0161] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1609, and / or installed from removable medium 1611. When the computer program is executed by central processing unit (CPU) 1601, it performs the functions defined above in the system of this invention.

[0162] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0163] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0164] The modules described in the embodiments of the present invention can be implemented in software or in hardware.

[0165] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: An audio capture device is used to capture sound. The processor is configured to divide the waveform of the sound into equal parts according to the beat to form a beat axis, and to superimpose vertical line marks on the beat points of the beat axis; and to detect the volume and pitch of the sound in real time to determine the pitch deviation. A display for displaying a waveform of the sound in the form of a beat axis, wherein the horizontal axis of the waveform is divided equally by beats and the vertical axis represents the sound amplitude; displaying the volume within each beat segment; and displaying the pitch, the pitch deviation, and the range to which the pitch deviation belongs in a preset color.

[0166] An audio acquisition device is used to acquire sound; a processor is used to divide the waveform of the sound into equal parts according to the beat to form a beat axis, and to superimpose vertical lines on the beat points of the beat axis; to detect the volume and (same as above) pitch of the sound in real time to determine the pitch deviation; a display is used to display the waveform of the sound in the form of a beat axis, wherein the horizontal axis of the waveform is divided into equal parts according to the beat, and the vertical axis represents the sound amplitude; to display the volume within each beat segment; and to display the pitch, the pitch deviation, and the range to which the pitch deviation belongs in a preset color. Based on the content displayed on the display, visual feedback and targeted performance guidance can be provided.

[0167] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention. It should be noted that the acquisition, storage, and application of user personal information involved in the technical solutions of this disclosure comply with relevant laws and regulations and do not violate public order and good morals.

Claims

1. An intelligent musical instrument auxiliary device, characterized in that, include: An audio capture device is used to capture sound. A processor is used to divide the waveform of the sound into equal parts according to the beat to form a beat axis, and to superimpose vertical line marks on the beat points of the beat axis; The volume and pitch of the sound are detected in real time to determine the pitch deviation; A display for showing the waveform of the sound in the form of a beat axis, wherein the horizontal axis of the waveform is divided equally by beats and the vertical axis represents the sound amplitude; The volume is displayed within each beat segment; and the pitch, the pitch deviation, and the range to which the pitch deviation belongs are displayed in a preset color.

2. The intelligent musical instrument auxiliary device according to claim 1, characterized in that, The processor is also used to determine the fingering position of the virtual fingerboard based on the pitch; The display is used to highlight the finger positions on the virtual fingerboard; And / or, The pitch range includes the normal pitch range marked in green, the slightly deviated pitch range marked in yellow, and the severely deviated pitch range marked in red.

3. The intelligent musical instrument auxiliary device according to claim 1, characterized in that, The processor is also configured to generate a directional fine-tuning indication when a pitch deviation is detected to exceed a preset threshold and the duration reaches a preset duration. The directional fine-tuning indication includes a highlighted suggested string position shifted in the string correction direction. And / or, The directional fine-tuning indication is displayed on the monitor, and / or the corrective guidance information for the finger position is played through a player.

4. The intelligent musical instrument auxiliary device according to claim 1, characterized in that, The processor receives an ear training interaction mode instruction and responds to the user's touch operation on the virtual fingerboard. When the touch duration is less than a preset threshold, it plays a short pitch sample of the corresponding position; when the touch duration reaches the preset threshold, it continues to play a long pitch sample of the corresponding position until the touch ends. The ear training interactive mode includes one or more of the following: ear training mode, interval imitation, chord imitation, and melody imitation; In the case of the ear training interaction mode, the display is used to highlight the corresponding solfège syllable on the virtual fingerboard in response to the pitch. The player plays the pitch sample corresponding to the stated pitch.

5. The intelligent musical instrument auxiliary device according to claim 1, characterized in that, The processor is used to determine the current pitch distribution range by using a sliding window algorithm for continuously acquired pitch sequences. By comparing the pitch distribution range with the position database, the display mode of the key and position corresponding to the current performance is switched; wherein, when switching positions, the display of the virtual fingerboard achieves a smooth transition from the current position to the target position through gradual transparency.

6. The intelligent musical instrument auxiliary device according to claim 1, characterized in that, The processor recognizes and executes the voice control commands collected by the audio collector, which include one or more of the following: turning on the metronome, switching instrument modes, selecting a key, and starting / stopping the ear training mode; And / or, It also includes physical buttons, the mapping relationship of which is determined according to user settings.

7. The intelligent musical instrument auxiliary device according to claim 1, characterized in that, Also includes: A metronome is used to respond to beat operation commands, set the beat speed, set the accent, or set a beat flashing synchronization prompt; A timer is used to respond to timing operation commands, set practice reminders, and set segmented timing. A tuner is used to set the tuning parameters of an instrument in response to tuning operation commands.

8. The intelligent musical instrument auxiliary device according to claim 1, characterized in that, In response to two tapping signals within a preset tapping time period, the processor determines the number of tapping beats based on the time interval between the two tapping signals; The display shows the number of taps and the target number of taps entered by the user; And / or, In response to multiple tapping signals within a preset tapping time period, the processor determines the real-time tapping beat count based on the time interval between two consecutive tapping signals. The display shows the real-time tapping beat count.

9. The intelligent musical instrument auxiliary device according to claim 1, characterized in that, The processor is also used to adjust the time length of each beat segment in the beat axis in real time according to the set beat speed and beat type, so that the waveform is synchronized with the beat generated by the metronome; And / or, The processor's position database includes position data for stringed instruments, and the position data includes the correspondence between pitch and finger position for each key. The processor is also used to load the corresponding finger position data according to the instrument type and key, and display the corresponding finger positions on the virtual fingerboard.

10. A method for assisting a smart musical instrument, characterized in that, include: Sound acquisition; The waveform of the sound is divided into equal parts according to the beat to form a beat axis, and vertical lines are superimposed on the beat points of the beat axis to display the waveform of the sound in the form of a beat axis. The horizontal axis of the waveform is divided into equal parts according to the beat, and the vertical axis represents the sound amplitude. The volume and pitch of the sound are detected in real time to determine the pitch deviation. The volume is displayed in each beat segment, and the pitch, the pitch deviation, and the range to which the pitch deviation belongs are displayed in a preset color.

Images