A digital timbre triggering and music interaction system based on a thumb piano playing structure

Abstract

This invention discloses a digital timbre triggering and music interaction system based on the thukadeva playing structure, belonging to the field of digital timbre triggering and music interaction technology. It includes: a main control processing module, a digital timbre storage and triggering module, an audio output module, a display and music interaction module, and a power control module. The main control processing module receives trigger signals, looks up mapping tables, and synchronizes information; the digital timbre module retrieves and processes timbre data to generate PCM audio frames; the audio output module buffers and outputs audio with low latency; the display and interaction module implements track loading, note matching, and difficulty adaptation; and the power module is responsible for power supply and basic system control. This improves the stability of digital timbre triggering and music interaction, solving the problem in existing technologies that cannot simultaneously meet users' multiple needs for the traditional thukadeva playing feel, stable digital timbre output, and engaging music interaction.

Description

Technical Field

[0001] This invention relates to the field of digital timbre triggering and music interaction technology, and in particular to a digital timbre triggering and music interaction system based on the thumb piano playing structure. Background Technology

[0002] The digitization and intelligentization of traditional musical instruments is an important development trend in the field of musical instruments. The core goal of this development direction is to lower the learning threshold for musical performance, enabling ordinary people to easily learn to play musical instruments and thus obtain a good musical experience. Therefore, musical equipment that combines the feel of traditional playing with the advantages of digitalization has become the market demand direction.

[0003] Currently, in the field of digital technology related to the thumb piano, existing technologies still have many defects and shortcomings, and a mature technical solution that can take into account the unique playing feel of the thumb piano, stable digital sound production, and interactive music gamification has not yet been formed. Traditional thumb pianos rely on the physical vibration of metal plates to produce sound. This method not only suffers from cumbersome tuning and inconsistent tone, but its timbre and overall effect are also easily affected by manufacturing precision. Furthermore, the purely physical design of traditional thumb pianos makes them difficult to integrate effectively with various digital content and music games, failing to meet users' digital interactive needs. Existing electronic musical instruments and music game devices mostly use keyboard or touch controls, completely lacking the unique tactile feedback of the thumb piano, resulting in a significantly different playing experience. These devices also generally have a high learning curve and weak entertainment value, making them unattractive to non-professionals and beginners. In conclusion, current technology lacks a solution that can fully digitize the sound production process of the thumb piano while retaining its tactile feedback, and deeply integrate it with interactive music games. It cannot simultaneously satisfy users' multiple needs for the tactile feedback of the traditional thumb piano, stable digital sound output, and engaging musical interaction. Summary of the Invention

[0004] To address the technical problem that existing technologies cannot simultaneously meet users' multiple needs for traditional thumb piano playing feel, stable digital sound output, and engaging music interaction, this invention provides a digital sound triggering and music interaction system based on the thumb piano playing structure. The technical solution is as follows: On one hand, a digital timbre triggering and music interaction system based on the playing structure of a thumping piano is provided. This system includes: a main control processing module, a digital timbre storage and triggering module, an audio output module, a display and music interaction module, and a power control module. The main control processing module receives digital trigger signals from the sensor detection module and writes timestamps, reads the current playing mode parameters and queries a predefined timbre mapping table, sends timbre triggering commands to the digital timbre storage and triggering module, synchronizes triggering information to the display and music interaction module, and can also merge repeated triggers of the same peg within the de-erase time window. The digital timbre storage and triggering module retrieves corresponding digital timbre data from the memory according to the timbre triggering command, and processes the digital timbre data... The audio is preprocessed to generate compliant PCM audio frames and transmitted to the audio output module. The audio output module receives the PCM audio frames and writes them into a circular buffer, then outputs them to the amplifier, speakers, and headphones to ensure that the end-to-end audio output latency is less than a perceptible threshold. The display and music interaction module includes a display component and an interaction control unit, which loads the rhythm and note sequence corresponding to the track, maps the target notes to the positions of the metal plectrums, renders a waterfall-style note track, completes the plucking action and target note matching judgment based on trigger information and time reference, and automatically adjusts the difficulty parameters based on the user's playing hit rate. The power control module provides stable power supply and implements basic control functions such as power on / off, mode switching, and volume adjustment.

[0005] Beneficial effects The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: Through the collaborative design and refined functional implementation of each module, several significant technical effects have been achieved: First, by integrating the plucking input, mechanical touch, and sensor detection units into the main control processing module, the traditional thumb piano's plucking feel and damping experience are preserved. Simultaneously, through multi-type sensor detection methods and de-jittering and effective trigger judgment functions, the plucking action is accurately captured and converted into a digital signal, effectively eliminating disturbances and mis-touches, ensuring the accuracy and stability of the trigger signal. Second, the main control processing module's merging processing of repeated triggers, precise reading of performance mode parameters, and intelligent querying of the timbre mapping table enable efficient processing of trigger signals and precise generation of timbre trigger commands, laying the foundation for rapid and matched timbre triggering. Third, the digital timbre storage and trigger module's precise retrieval of timbre data, multi-dimensional preprocessing, and multi-channel mixing and limiting processing achieve highly adaptable digital timbre generation, effectively avoiding audio distortion and ensuring high-quality and consistent timbre output. Fourth, the audio output module, through continuous buffering in a circular buffer, high-speed digital-to-analog conversion, and power amplification, achieves... The system integrates full-process timing control, achieving low-latency audio output with end-to-end latency below the user's perceptible threshold, ensuring real-time performance and immersive experience. Fifth, the display and music interaction module, through unified track loading and time reference, achieves precise mapping of notes and keys, visual rendering and highlighting of waterfall tracks. Combined with accurate time deviation calculation and matching judgment, it provides clear performance guidance. Simultaneously, through real-time recording of performance data, accurate hit rate statistics, and adaptive adjustment of difficulty parameters, it achieves dynamic matching of performance difficulty with user skill level, significantly lowering the learning threshold for music. Sixth, each module has a clear division of labor and seamless integration. The power control module provides stable power supply to the entire system and enables convenient basic control, ensuring the stability, smoothness, and ease of use of the entire system. Overall, it achieves a deep integration of traditional thumb piano touch with digital sound output and gamified music interaction, enhancing the realism and auditory quality of the performance experience, while also increasing the product's fun and interactivity, and taking into account the system's mass production feasibility and practicality. Attached Figure Description

[0006] Figure 1 A schematic diagram of a digital timbre triggering and music interaction system based on a thumb piano playing structure is provided in an embodiment of this application; Figure 2 This application provides a flowchart of the merging process of a digital timbre triggering and music interaction system based on a thumb piano playing structure. Detailed Implementation

[0007] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0008] like Figure 1The diagram shown is a structural schematic of a digital timbre triggering and music interaction system based on a thumb piano playing structure provided in an embodiment of this application. The system includes: a main control processing module, a digital timbre storage and triggering module, an audio output module, a display and music interaction module, and a power control module.

[0009] As the first module of this system, the main control processing module is used to receive digital trigger signals from the sensor detection module and write timestamps, read the current playing mode parameters and query the predefined timbre mapping table, send timbre trigger commands to the digital timbre storage and trigger module, synchronize trigger information to the display and music interaction module, and also merge repeated triggers of the same piano within the de-shake time window.

[0010] It should be understood that the main control processing module includes a toggle input unit, a mechanical tactile structure unit, and a sensor detection unit; The plucked input unit is a semi-mechanical thumb piano plucked input structure, which includes multiple metal piano components. Each metal piano is arranged in a fan-shaped or linear manner. One end of each metal piano is the plucking end for the user's thumb to pluck. The metal piano can produce limited elastic deformation and does not serve as the main sound source. The mechanical tactile structure unit works in conjunction with the plucking input unit to provide plucking damping for the metal plucking piece and suppress audible physical vibrations. The mechanical tactile structure unit includes a limiting structure and a damping structure that match the metal plucking piece. The limiting structure limits the maximum displacement of the metal plucking piece, and the damping structure, in conjunction with the stiffness of the metal plucking piece, controls the plucking trigger force within a preset range. The thickness, length, and elastic parameters of the metal plucking piece are all designed according to the standard of simulating the plucking damping of a traditional thumb piano.

[0011] The sensing and detection unit is set up corresponding to each metal peg and installed in a fixed area at the base and below the metal peg. It is used to detect the physical signal generated by the user plucking the metal peg and convert it into a digital trigger signal, which is then transmitted to the main control processing module. It supports at least one of the following detection methods: capacitive sensing, pressure, strain detection, Hall effect, magnetic sensing, and optical detection. It also has front-end signal conditioning, debouncing, and effective trigger determination functions. It can eliminate environmental disturbances and false trigger signals. The output digital trigger signal includes the peg number and characteristic quantities used to calculate the plucking force and speed.

[0012] In this embodiment, the plucking input unit adopts a semi-mechanical thumb piano plucking input structure, with the core being a metal pluck assembly. The specific configuration is as follows: the metal plucks are made of 65Mn spring steel as the base material. The elasticity and stiffness of this material are suitable for the tactile requirements of thumb piano plucking, and it is not prone to fatigue or deformation, meeting the requirements of mass production and long-term use. The number of metal plucks is set to 17 according to performance needs, arranged in a fan-shaped array on the plucking area of ​​the device panel, adapting to the plucking trajectory of the human thumb. It can also be adjusted to a linear arrangement to meet product miniaturization requirements. One end of each metal pluck is a fixed end, locked to the internal metal frame by screws, while the other end is a free-plucking end, extending out of the device panel for the user's thumb to pluck. The plucking end is rounded to improve operational comfort. The thickness of the metal plucks is designed to be 0.3mm, and the length is designed to be 35-55mm (adjusted according to different pitches), ensuring a plucking action of 0.5-2.0mm. The limited elastic deformation of the plectrum is solely for tactile feedback; the plectrum itself is not the primary sound source and produces no noticeable physical vibration. The mechanical tactile structure unit and the metal plectrum of the plectrum input unit are designed to correspond one-to-one. The internal cavity of the device, located beneath the metal plectrum, consists of a limiting structure and a damping structure. This design simulates the damping of traditional thumb piano plectrums while suppressing audible physical vibrations. Specifically, the limiting structure uses high-temperature resistant silicone posts, with one post under each metal plectrum. The height of the posts is designed to be 2.0mm, based on the maximum pre-set deformation of the metal plectrum. When the user plucks the plectrum, the lower surface of the plectrum cannot deform further downwards after contacting the post, precisely limiting the maximum plectrum displacement to 0.5-2.0mm, thus preventing excessive physical vibrations caused by excessive deformation. The damping structure employs a combination of felt friction pads and silicone buffer pads. The felt friction pads are attached to the middle section of the lower surface of the metal thimble, while the silicone buffer pads are positioned at the top of the limiting post. During the plucking and rebounding of the metal thimble, the felt friction pads generate moderate friction with the thimble, and the silicone buffer pads cushion the rebound. Together with the stiffness parameters of the metal thimble, these elements precisely control the plucking trigger force within a preset range of 0.5N-2.0N, completely simulating the damped feel of a traditional thimble. Simultaneously, friction and buffering dissipate the thimble's vibrational energy, completely suppressing audible physical vibrations. The thickness, length, and elasticity parameters of the metal thimble are precisely matched and adjusted with the displacement limits of the limiting structure and the friction / buffering coefficient of the damping structure. For thimbles of different lengths, the height of the corresponding limiting post and the thickness of the friction pad are adjusted to ensure consistent plucking feel across all thimbles, with no significant difference in playing feel from a traditional thimble.The sensing and detection unit is installed at the base (fixed end) of the metal piano pieces and below the metal mounting bracket area, corresponding to each of the 17 metal piano pieces. The sensor installation position is close to the fixed end of the piano pieces, which effectively improves the detection stability of deformation / displacement signals and reduces signal crosstalk triggered by adjacent piano pieces. In this embodiment, a dual detection combination of capacitive sensing and strain detection is adopted (or any one of Hall / magnetic sensing or optical detection can be used alone). It also integrates front-end signal conditioning, debouncing, and effective trigger determination functions. The specific implementation is as follows: Capacitive sensing detection: The metal piano pieces are directly used as capacitive sensing electrodes. A fixed electrode plate is set below the base of each piano piece. The piano piece and the fixed electrode plate form a parallel plate capacitor with a distance of 1.0 mm between them. The small deformation caused by plucking the piano piece will change the electrode distance, thereby causing a change in capacitance value; Strain detection detection: A micro foil strain gauge is pasted on the side of the base of each metal piano piece. The resistance of the strain gauge changes linearly with the small deformation of the piano piece. The strain gauge is pasted with high-temperature adhesive to ensure the fit with the piano piece and avoid signal distortion. The sensing and detection unit integrates a front-end signal conditioning circuit board, which is electrically connected to the capacitive sensing electrodes and strain gauges. It performs targeted conditioning on the detected physical signals. The specific processing flow is as follows: the capacitance change signal is converted into a voltage signal using a capacitance-to-voltage conversion circuit, and the resistance change signal of the strain gauge is converted into a voltage signal using a Wheatstone bridge. The converted voltage signals are then subjected to low-pass filtering (cutoff frequency 1kHz) and precision amplification (amplification factor 100x) to filter out environmental electromagnetic interference and improve the detection accuracy of weak signals. The amplified signal is then baseline calibrated to eliminate the initial signal offset after the device is powered on. Finally, a voltage comparator is used to set the trigger threshold, converting the analog signal into a preliminary digital pulse signal. The sensing unit incorporates a microcontroller chip to perform de-jittering and valid trigger determination. The specific processing rules are as follows: a 20ms de-jittering time window is set. Within this window, for continuous digital pulse signals of the same key, only the signal that first exceeds the trigger threshold is retained, while repeated trigger signals caused by key rebound oscillations are discarded. The peak value and rising edge slope of the signal are extracted as feature quantities. A lower limit threshold for these feature quantities is set. If the signal feature quantity is below the threshold, it is determined to be a false touch signal such as environmental disturbance or finger light touch, and is directly discarded. If the feature quantity is above the threshold, it is determined to be a valid trigger signal. After de-jittering and valid trigger determination, the sensing unit encapsulates the valid trigger signal into a standard digital trigger data frame, which is transmitted to the core control unit of the main control processing module via the SPI communication interface. The data frame contains two types of core information: first, the key number (KeyID), a hexadecimal code from 00-16, corresponding one-to-one with the 17 metal key pieces; second, feature quantity data, including the voltage peak value and rising edge slope after capacitance / strain conversion, for the main control processing module to calculate the plucking force and speed.

[0013] It should be noted that, as Figure 2 The diagram shows a merging process flowchart of a digital timbre triggering and music interaction system based on a thumb piano playing structure provided in this application embodiment. The specific process is as follows: the digital trigger signal from the sensing unit is received through GPIO interrupt and polling, and a timestamp is added to the valid signal to complete the time calibration; after extracting the piano number and feature quantity, repeated triggers of the same piano are merged based on a preset de-jittering time window, retaining only the first valid event; then the current playing mode parameters are read, and the target pitch, timbre type and sampling layer are matched in combination with a predefined timbre mapping table to generate and send a timbre trigger command to the digital timbre module to retrieve and generate the corresponding timbre; at the same time, the valid trigger piano number, timestamp, target pitch and other information are synchronized to the display and music interaction module to support note matching judgment and visual feedback, and finally the next trigger signal is continuously processed in a loop.

[0014] It is important to understand that the specific process for merging repeated triggers of the same piano piece within the debouncing time window is as follows: The system receives digital trigger signals carrying the piano piece number and characteristic values ​​from the sensor detection unit via GPIO interrupt and polling. Each valid digital trigger signal received is written with a unique timestamp to complete the time calibration of the trigger event. Identify the piano piece number in the digital trigger signal, determine whether the digital trigger signal corresponding to the same piano piece number appears within the preset de-jittering time window, and if so, merge the repeated trigger events within the de-jittering time window into one valid trigger, retaining only the timestamp and feature value of the first trigger. Obtain the current performance mode parameters, including the mode type (free play, teaching, game), key settings, chord progression, key change parameters, and tempo progress. Based on the read performance mode parameters, query the predefined timbre mapping table, match the target pitch, timbre type, and timbre sampling layer with the plectrum number and the current performance mode parameters, and generate timbre trigger-related matching information.

[0015] Merging repeated triggers of the same piano within the debounce time window also includes: Based on the timbre triggering related matching information, a timbre triggering command containing the target pitch, timbre type, and timbre sampling layer is generated, and the timbre triggering command is sent to the digital timbre storage and triggering module to trigger the retrieval and generation of digital timbre; The trigger information, such as the validly triggered piano number, timestamp, matched target pitch, and trigger judgment result, is synchronized to the display and music interaction module in real time, providing data support for the note matching judgment and visual feedback of the display and music interaction module.

[0016] In this embodiment, the main control processing module uses the STM32F407 core chip, with a preset debouncing time window of 20ms. The GPIO interrupt is triggered by an external falling edge, and the polling frequency is set to 1kHz. The predefined timbre mapping table is stored in the chip's on-chip Flash. The specific implementation process is as follows: the main control processing module receives the hexadecimal KeyID of the piano piece from the sensing unit via the SPI interface in real time through a combination of GPIO external falling edge interrupt and 1kHz frequency polling. The chip writes a unique millisecond-level timestamp T0 to each valid digital trigger signal received and determined by the sensing unit, along with voltage peak value and rise time slope characteristics, to accurately time each trigger event. Subsequently, the chip's built-in signal processing program automatically identifies the key ID (KeyID) in the digital trigger signal, retrieves the timestamp T1 of the previous trigger for that KeyID, and calculates the time difference ΔT = T0 - T1. ΔT is compared with a preset 20ms debounce time window to determine if the digital trigger signal corresponding to the same key ID appears within the debounce time window. If ΔT ≤ 20ms, it is determined to be a repeated trigger caused by key rebound oscillation. The multiple repeated trigger events within the debounce time window are immediately merged into a single valid trigger, retaining only the timestamp T1 of the first trigger. The corresponding feature data is used to remove subsequent repeated trigger signal data. If ΔT > 20ms, it is determined to be a new valid trigger by the user, and the timestamp T0 and feature data of this trigger are directly retained. After the repeated trigger merging process is completed, the main control processing module reads the current performance mode parameters of the system through the on-chip bus. The performance mode parameters include the mode type of free performance / teaching / game, C major / minor key settings, chord progressions such as 1-6-4-5, modulation parameters within ±8 degrees, and the real-time tempo progress of the song. Then, based on the full set of performance mode parameters read, the predefined timbre mapping table stored in the on-chip Flash is called. By matching the key ID of the piano in the table with the double key fields of the performance mode parameters, the target pitch, piano / thumb piano timbre type, and timbre sampling layer corresponding to the valid trigger are accurately obtained. Finally, the timbre trigger related matching information containing all the above information is integrated to complete the entire process of repeated trigger merging and timbre matching information generation.

[0017] As the second module of this system, the digital timbre storage and triggering module is used to retrieve the corresponding digital timbre data from the memory according to the timbre triggering command, preprocess the digital timbre data to generate a PCM audio frame that meets the requirements and transmit it to the audio output module.

[0018] It should be understood that the specific process for generating compliant PCM audio frames and transmitting them to the audio output module is as follows: By receiving the timbre trigger command sent by the main control processing module, the timbre trigger command is parsed to extract the target pitch, timbre type, timbre sampling layer, and timbre synthesis parameters obtained by mapping the plucking force and speed of the piano. Based on the timbre synthesis parameters obtained from the analysis, the corresponding digital timbre data is retrieved. The digital timbre data includes pre-stored sampled PCM data and timbre synthesis parameters. Preprocessing includes resampling, pitch shifting, ADSR envelope overlay, filtering, reverb, and multi-channel mixing and limiting.

[0019] Generating compliant PCM audio frames and transmitting them to the audio output module also includes: If the retrieved data is sampled PCM data and its reference pitch is inconsistent with the target pitch, the sampled PCM data is resampled and pitch-shifted; if the retrieved data is timbre synthesis parameters, waveform synthesis is performed according to the parameters, and then ADSR envelope, filtering and reverb effects are superimposed on the resampled and pitch-shifted sampled PCM data or the synthesized waveform data to obtain single-channel preprocessed audio data. Determine if multiple pianos are triggered simultaneously. If so, mix the pre-processed digital timbre data and then limit the mixed digital timbre data to avoid audio distortion. If only a single piano is triggered, directly transmit the PCM audio frame. The processed digital timbre data is encapsulated into PCM audio frames that conform to the transmission standard of the audio output module. The PCM audio frames are then written into the circular buffer of the audio output module, completing the transmission to the audio output module.

[0020] In this embodiment, the digital timbre storage and triggering module performs digital timbre processing, audio frame generation, and transmission based on timbre triggering instructions. In this embodiment, the system audio sampling rate is set to 44.1kHz and 16-bit depth. The digital timbre data is stored in an external W25Q128 Flash chip. The ADSR envelope parameters, filter coefficients, and reverberation depth are all preset to standard values ​​suitable for thukade playing. The circular buffer depth is set to 1024 frames. The specific implementation process is as follows: The digital timbre storage and triggering module receives the standard timbre triggering instructions sent by the main control processing module through the SPI communication interface. It parses the instructions field by field, extracting the target pitch corresponding to the plectrum number, the timbre type of piano / acoustic thukade / electric thukade, the timbre sampling layer corresponding to light / medium / heavy plucking, and the timbre synthesis parameters such as gain, attack, and attenuation obtained by mapping the user's plucking force and speed. Based on all the parsed parameter combinations, it retrieves matching digital timbre data from the external Flash memory. This data includes pre-recorded sampled PCM. The system handles two types of data: data and timbre synthesis parameters required for FM and wavetable synthesis. Preprocessing is then performed according to preset rules. If the retrieved data is sampled PCM data with a deviation between its reference and target pitch, linear interpolation is used for resampling and semitone shifting. If the retrieved data is timbre synthesis parameters, sine, square, or triangle-based synthesized waveforms are directly generated. Standard ADSR envelopes, high-pass and low-pass filters, and spatial reverb effects are then uniformly superimposed on the shifted sampled PCM data or synthesized waveform data to obtain a single, clean preprocessed audio data stream. Simultaneously, the module monitors the current trigger status in real-time to determine if two or more pegs are triggered simultaneously. If multi-tone overlay triggering occurs, proportional mixing and overlay processing is performed on the multi-channel preprocessed audio data, and digital limiting is applied to the mixed data to restrict the audio amplitude to within 0 dBFS to avoid clipping distortion. If only a single peg is triggered, the system directly enters the frame encapsulation process. Finally, the processed audio data is encoded at 44.1kHz / 16 The standard format of 12-bit audio is encapsulated into continuous PCM audio frames. The PCM audio frames are written frame by frame into the circular buffer of the audio output module through the I2S bus, thus completing the stable transmission of high-quality, low-distortion digital timbre data to the audio output module.

[0021] As the third module of this system, the audio output module is used to receive PCM audio frames and write them into a circular buffer, and output them to the power amplifier, speakers and headphones to ensure that the end-to-end audio output delay is less than a perceptible threshold.

[0022] It should be understood that the audio output module includes a circular buffer, a power amplifier, and sound-generating components; By receiving PCM audio frames that conform to the transmission standard, the PCM audio frames are written sequentially into the built-in circular buffer according to the audio output timing, so as to realize continuous buffering of audio data and avoid data interruption. PCM audio frame data is continuously read from the circular buffer according to the preset audio sampling rate, and the digital PCM audio data is converted into analog audio electrical signals through the I2S interface and DAC digital-to-analog conversion circuit. The converted analog audio signal is transmitted to the built-in power amplifier circuit of the module. The power amplifier circuit amplifies the analog audio signal to meet the sound driving requirements of the speaker and headphones. The amplified analog audio signal is synchronously transmitted to the speaker and headphone interfaces of the device, supporting single or multi-channel audio output for speaker playback and headphone listening, to meet different listening needs of users. By using a circular buffer for buffering, high-speed I2S / DAC data conversion, and fast signal amplification by the power amplifier circuit, the entire process of audio data transmission, conversion, and amplification is time-controlled to ensure that the end-to-end output delay from digital timbre triggering to audio output is less than the user-perceptible threshold.

[0023] In this embodiment, the audio output module performs PCM audio frame reception, conversion, amplification, and low-latency output. The audio output module uses the ES8388 audio decoding chip as its core, integrating a 4096-byte FIFO ring buffer, a 3W mono Class D power amplifier circuit, and a dual-sound component consisting of a 3.5mm headphone + 8Ω / 2W miniature full-range speaker. The system's preset audio sampling rate is 44.1kHz, sampling depth is 16-bit, and mono output is enabled. The end-to-end latency threshold is set to 20ms. To improve audio output stability and adaptability, an audio signal detection and automatic interface switching sub-circuit is added. The specific implementation process is as follows: the audio output module receives the 44.1kHz / 16-bit PCM audio frame transmitted by the digital timbre storage and trigger module via the I2S high-speed communication interface. The module's built-in frame synchronization recognition unit verifies the format of standard PCM audio frames. After successful verification, the PCM audio frames are sequentially written into a FIFO circular buffer according to the audio output timing. This circular buffer employs a first-in-first-out (FIFO) buffering mechanism, continuously writing and reading data in a loop to achieve continuous and uninterrupted audio data buffering, effectively avoiding audio dropouts and stuttering caused by data transmission rate fluctuations. Subsequently, the module continuously and at a constant speed reads the PCM audio frame digital data from the circular buffer at a preset audio sampling rate of 44.1kHz. The high-precision DAC (digital-to-analog converter) circuit built into the ES8388 chip converts the digital PCM audio data into an analog audio signal with an amplitude range of 0-3.3V. During the conversion process, the chip's digital anti-aliasing filter function is activated to filter out high-frequency noise generated during conversion, ensuring the purity of the analog audio signal. The converted analog audio signal is then transmitted to the module's built-in Class D power amplifier circuit. This power amplifier circuit uses a high-efficiency digital amplification architecture to amplify the weak analog audio signal, outputting a maximum drive power of 3W, ensuring that the amplified audio signal fully meets the 8Ω / 2W requirement. The external speaker's output drive requirements are related to 3.The module meets the listening requirements of 5mm headphones, and the power amplifier circuit has built-in overcurrent and overtemperature protection mechanisms to improve the safety of module operation. The analog audio signal after power amplification is transmitted to the module's added interface automatic switching sub-circuit. This circuit can detect the plugging and unplugging status of the headphone jack in real time. When headphones are detected, it automatically cuts off the speaker path and outputs the audio signal separately to the headphone jack. When no headphones are plugged in, the audio signal is output to the speaker jack. It supports single-channel output mode for speaker playback and headphone listening, as well as seamless switching between plugging and unplugging, adapting to different listening scenarios such as users listening alone or sharing with multiple people. In the entire process of audio data transmission, conversion, and amplification, the module uses dynamic buffering scheduling of the ring buffer and 4Mbps of the I2S interface. High-speed data transmission, microsecond-level DAC digital-to-analog conversion, and rapid signal amplification by Class D amplifier circuits enable precise timing control throughout the entire process. This allows for precise compression of processing delays at each stage. Furthermore, hardware-level circuit optimization and software-level timing calibration ensure that the end-to-end output delay from digital timbre triggering to audio output is controlled within 15ms, far below the user-perceptible threshold of 20ms. This allows for zero-latency real-time synchronization between user input and audio output, guaranteeing immersive performance and real-time interaction.

[0024] As the fourth module of this system, the display and music interaction module includes a display component and an interaction control unit. It is used to load the rhythm and note sequence corresponding to the song, map the target notes to the positions of the metal pegs one by one and render the waterfall-style note track, complete the matching judgment of the plucking action and the target notes according to the trigger information and time reference, and automatically adjust the difficulty parameters according to the user's playing hit rate.

[0025] It should be noted that the interactive control unit loads the rhythm and note sequence data corresponding to the user-selected track and practice content, and shares a unified system time base with the main control processing module to keep the rendering sequence of the note track synchronized with the audio output sequence, ensuring that the position of the notes in the waterfall flow is accurately aligned with the actual sounding time. Each target note in the musical note sequence is mapped to the corresponding metal plate number according to a preset rule, establishing a one-to-one correspondence between the target note and the physical position of the plate, providing a positional basis for the waterfall flow track rendering; Based on the one-to-one correspondence between the target note and the physical position of the plectrum, the display component renders a waterfall-style note track corresponding to the plectrum position on the display interface. For the target note that is about to reach the trigger time, the corresponding track and plectrum position are highlighted in the preset preview time window to clearly indicate which plectrum the user needs to pluck and the trigger time. By receiving valid trigger information synchronized by the main control processing module, including the piano number and trigger timestamp, and combining it with the time reference to extract the preset trigger timestamp of the target note, the time deviation between the actual trigger timestamp and the target trigger timestamp is calculated. The time deviation is compared with the preset judgment tolerance window. If the time deviation is within the tolerance window range and the trigger paddle number is consistent with the paddle number mapped to the target note, the match is considered successful; otherwise, the match is considered unsuccessful.

[0026] In this embodiment, the interactive control unit and display component in the display and music interaction module work together to complete track loading, note mapping, waterfall rendering, and plucking action matching and judgment. In this embodiment, the interactive control unit uses an ESP32 chip as its core, and the display component uses a 1.8-inch TFT color LCD screen. The system uses a unified time base of a 1ms precision system timer clock, with a preset note preview time window of 1.5s and a basic judgment error tolerance window of ±80ms. It also features a creative design with multi-color graded highlight prompts and trigger accuracy graded judgments to adapt to the performance guidance and feedback needs of different users. The specific implementation process is as follows: the interactive control unit reads the standard MIDI format rhythm and note sequence data corresponding to the user-selected track or practice content through local Flash memory, and simultaneously shares a unified 1ms time base with the main control processing module via a hardware SPI bus. The precision system's time base, Clock, achieves millisecond-level alignment between the two through a clock synchronization calibration algorithm. This ensures complete synchronization between the rendering and scrolling timing of the note tracks on the LCD screen and the sound output timing of the audio module, guaranteeing a precise match between the falling trigger position of the waterfall-style notes and the actual audio sound output moment, without any timing deviation. The interactive control unit, following a preset rule that the pitch corresponds to the physical arrangement of the metal pegs in the twelve-tone equal temperament system, precisely maps each target pitch note in the song's note sequence to its corresponding unique metal peg number, KeyID, establishing a one-to-one correspondence between target notes and peg physical positions. Simultaneously, it generates a mapping table between peg numbers and LCD screen display coordinates, providing positional basis for the precise rendering of the waterfall-style note tracks. The display component, based on this mapping table, renders a color waterfall-style note track on the TFT color LCD screen, corresponding one-to-one with the physical positions of the pegs. The number of tracks is exactly the same as the number of metal pegs. Furthermore, for target notes approaching their trigger moment, a multi-color, graded highlight is provided within a preset 1.5s preview time window, ranging from 1.5s to 0.8s before the trigger moment. The indicator is light blue for 0.8s-0.3s, and bright yellow for 0.A bright red indicator appears within 3 seconds, clearly indicating the position of the plectrum to be plucked and the precise trigger time through a color gradient, enhancing the intuitiveness of the performance guidance. The interactive control unit receives valid plectrum trigger information synchronized by the main control processing module in real time via the UART serial port. This information includes the plectrum number KeyID and the millisecond-level actual trigger timestamp T0. Combined with the shared system time base Clock, the preset trigger timestamp T1 of the corresponding target note in the current waterfall flow is extracted. The time deviation Δt = T0 - T1 is accurately calculated using a difference algorithm. The interactive control unit compares the time deviation Δt with a basic judgment error tolerance window of ±80ms. At the same time, it checks whether the trigger plectrum number KeyID matches the plectrum number mapped to the target note. If the two numbers match and Δt is within the error tolerance window, it is considered a successful match. Based on |Δt|, the trigger accuracy is graded: |Δt| ≤ 20ms is perfect trigger, 20ms < |Δt| ≤ 50ms is precise trigger, and 50ms < |Δt| ≤ 80ms is accurate trigger. For a successful trigger, if the peg numbers are inconsistent or Δt exceeds the tolerance window range, the match is considered a failure. The graded judgment result will be simultaneously displayed on the corresponding track on the LCD screen, allowing users to clearly understand the accuracy of their playing and improve the focus of practice and performance.

[0027] It should be further explained that the specific process for automatically adjusting the difficulty parameters based on the user's performance hit rate is as follows: Records performance data for each plucking action in real time, including plectrum number, time deviation, matching judgment result, plucking force and speed. During or after the practice of a piece, the user's performance hit rate is calculated based on the total number of valid triggers and the number of successful matches. The statistically obtained performance hit rate is compared with the preset hit rate threshold range. If the hit rate exceeds the upper threshold, the difficulty parameters are automatically increased, including reducing the judgment tolerance window, shortening the note preview time window, and expanding the piano-note mapping range. If the hit rate is lower than the lower threshold, the difficulty parameters are automatically decreased, including expanding the judgment tolerance window, extending the note preview time window, and reducing the piano-note mapping range, thus achieving adaptive matching of difficulty parameters.

[0028] In this embodiment, the system difficulty parameters are automatically adjusted based on the user's performance hit rate. A built-in local data storage area is used to record performance data in real time. The preset basic hit rate threshold range is the lower limit to the upper limit. A creative design is added that allows for segmented adjustment of difficulty levels and gradual parameter adjustment. Different parameter adjustment gradients are configured for different performance modes to avoid sudden changes in difficulty affecting the performance experience. The specific implementation process is as follows: During the entire process of the user playing or practicing a piece, the interactive control unit records complete performance data for each plucking motion in real time through a data acquisition subroutine. This includes the key ID, the actual time deviation Δt from the preset trigger, the matching judgment result (perfect / accurate / qualified / failed), and the Velocity value mapping the plucking force and speed. All data is stored in the chip's built-in Flash data storage area in an orderly manner according to timestamps, supporting real-time statistics during performance and review statistics after performance. Subsequently, the data statistics subroutine calculates the performance hit rate based on the total number of valid triggers and the number of successful matches (perfect, accurate, and qualified judgments are all included in the successful matches) using the formula: Performance Hit Rate = Number of Successful Matches / Total Number of Valid Triggers × 100%. The system accurately calculates the user's real-time performance accuracy, calculating accuracy by phrase in game mode and by practice measure in teaching mode, adapting to different learning and performance needs. The interactive control unit compares the calculated real-time accuracy with a preset basic threshold range, and adaptively adjusts the difficulty parameters based on the current system difficulty level (beginner / intermediate / advanced). If the accuracy exceeds the upper limit threshold, it is determined that the user has adapted to the current difficulty, and the difficulty parameters will be automatically increased. Specifically, the error tolerance window is reduced according to the difficulty level gradient (±10ms for beginner, ±15ms for intermediate, and ±20ms for advanced), the note preview time window is shortened (0.2s for beginner, 0.3s for intermediate, and 0.4s for advanced), and the plectrum-note mapping range is expanded (gradually expanding from the basic 8-note mapping to the full 17-note mapping). All parameters are adjusted gradually. If the accuracy is below 60%, the system will take further action. If the lower threshold is set, and the user is deemed not yet suited to the current difficulty level, the difficulty parameters will be automatically adjusted. Specifically, the error tolerance window will be expanded according to the difficulty level gradient (±10ms for beginners, ±15ms for intermediates, and ±20ms for advanceds), the note preview time window will be extended, and the range of piano-note mapping will be narrowed (gradually shrinking from full piano mapping to the basic 8-note core mapping). At the same time, the piano mapping range that the user is good at will be retained to improve the targeting of practice. If the hit rate is within the threshold range, the current difficulty parameters will remain unchanged, allowing the user to consolidate their practice at a suitable difficulty level.Throughout the adjustment process, changes to the difficulty parameters will be subtly indicated via pop-up notifications in the display component. Simultaneously, the current difficulty level and hit rate will be displayed in real time in the corner of the interface. This achieves dynamic and precise matching between the difficulty parameters and the user's playing level, avoiding frustration caused by excessively high difficulty and preventing reduced enjoyment of practice and performance due to excessively low difficulty, thus effectively improving the user's learning efficiency and performance experience.

[0029] As the fifth module of this system, the power control module is used to provide a stable power supply and to realize basic control functions such as power on / off, mode switching, and volume adjustment.

Claims

1. A digital timbre triggering and musical interaction system based on a harp playing structure, characterized in that, include: The main control processing module, digital timbre storage and triggering module, audio output module, display and music interaction module, and power control module; The main control processing module is used to receive digital trigger signals from the sensor detection module and write timestamps, read the current playing mode parameters and query the predefined timbre mapping table, send timbre trigger commands to the digital timbre storage and trigger module, synchronize trigger information to the display and music interaction module, and can also merge repeated triggers of the same piano within the de-shake time window. The digital timbre storage and triggering module is used to retrieve corresponding digital timbre data from the memory according to the timbre triggering command, preprocess the digital timbre data to generate a PCM audio frame that meets the requirements and transmit it to the audio output module. The audio output module is used to receive PCM audio frames and write them into a circular buffer, and output them to the power amplifier, speakers and headphones to ensure that the end-to-end audio output delay is less than a perceptible threshold. The display and music interaction module includes a display component and an interaction control unit, which is used to load the rhythm and note sequence corresponding to the song, map the target notes to the positions of the metal piano pieces one by one and render the waterfall-style note track, complete the matching judgment of the plucking action and the target notes according to the trigger information and time reference, and automatically adjust the difficulty parameters according to the user's playing hit rate. The power control module is used to provide a stable power supply and to realize basic control functions such as power on / off, mode switching, and volume adjustment.

2. The digital timbre triggering and music interactive system based on the harp playing structure according to claim 1, characterized in that: The main control processing module includes a toggle input unit, a mechanical tactile structure unit, and a sensor detection unit; The plucking input unit is a semi-mechanical thumb piano plucking input structure, which includes multiple metal piano components. Each metal piano is arranged in a fan-shaped or linear manner. One end of each metal piano is the plucking end for the user's thumb to pluck. The metal piano can produce limited elastic deformation and does not serve as the main sound source. The mechanical tactile structure unit is configured in conjunction with the plucking input unit to provide plucking damping for the metal plectrum and suppress audible physical vibrations. The mechanical tactile structure unit includes a limiting structure and a damping structure that match the metal plectrum. The limiting structure limits the maximum displacement of the plectrum, and the damping structure, in conjunction with the stiffness of the metal plectrum, controls the plucking trigger force within a preset range. The thickness, length, and elastic parameters of the metal plectrum are all designed according to the standard of simulating the plucking damping of a traditional thumb piano.

3. The digital timbre triggering and music interaction system based on the harp playing structure according to claim 2, characterized in that: The sensing and detection unit is configured corresponding to each metal peg and installed in a fixed area at the base and below the metal peg. It is used to detect the physical signal generated by the user plucking the metal peg and convert it into a digital trigger signal, which is then transmitted to the main control processing module. It supports at least one of the following detection methods: capacitive sensing, pressure, strain detection, Hall effect, magnetic sensing, and optical detection. It also has front-end signal conditioning, de-jittering, and effective trigger determination functions. It can eliminate environmental disturbances and false trigger signals. The output digital trigger signal includes the peg number and a characteristic quantity used to calculate the plucking force and speed.

4. The digital timbre triggering and music interaction system based on the thumb piano playing structure as described in claim 1, characterized in that: The specific process for merging repeated triggers of the same piano piece within the debounce time window is as follows: The system receives digital trigger signals carrying the chip number and characteristic values ​​from the sensor detection unit via GPIO interrupt and polling. Each valid digital trigger signal received is written with a unique timestamp to complete the time calibration of the trigger event. Identify the piano piece number in the digital trigger signal, determine whether the digital trigger signal corresponding to the same piano piece number appears within the preset de-jittering time window, and if so, merge the repeated trigger events within the de-jittering time window into one valid trigger, retaining only the timestamp and feature value of the first trigger. Obtain the current performance mode parameters, which include the mode type (free play, teaching, game), mode settings, chord progression, modulation parameters, and tempo progress. Based on the read performance mode parameters, query the predefined timbre mapping table, match the target pitch, timbre type, and timbre sampling layer with the plectrum number and the current performance mode parameters, and generate timbre trigger-related matching information.

5. The digital timbre triggering and music interaction system based on the thumb piano playing structure as described in claim 4, characterized in that: The process of merging repeated triggers of the same piano piece within the debounce time window also includes: Based on the timbre triggering related matching information, a timbre triggering instruction containing the target pitch, timbre type, and timbre sampling layer is generated, and the timbre triggering instruction is sent to the digital timbre storage and triggering module to trigger the retrieval and generation of digital timbre; The trigger information, such as the validly triggered piano number, timestamp, matched target pitch, and trigger judgment result, is synchronized to the display and music interaction module in real time, providing data support for the note matching judgment and visual feedback of the display and music interaction module.

6. The digital timbre triggering and music interaction system based on the thumb piano playing structure as described in claim 1, characterized in that: The specific process for generating compliant PCM audio frames and transmitting them to the audio output module is as follows: By receiving the timbre trigger command sent by the main control processing module, the timbre trigger command is parsed to extract the target pitch, timbre type, timbre sampling layer, and timbre synthesis parameters obtained by mapping the plucking force and speed of the piano. Based on the timbre synthesis parameters obtained from the analysis, the corresponding digital timbre data is retrieved. The digital timbre data includes pre-stored sampled PCM data and timbre synthesis parameters. The preprocessing includes resampling, pitch shifting, ADSR envelope superposition, filtering, reverb, and multi-channel mixing and limiting processing.

7. The digital timbre triggering and music interaction system based on the thumb piano playing structure as described in claim 6, characterized in that: The process of generating compliant PCM audio frames and transmitting them to the audio output module further includes: If the retrieved data is sampled PCM data and its reference pitch is inconsistent with the target pitch, resampling and pitch shifting are performed on the sampled PCM data; if the retrieved data is timbre synthesis parameters, waveform synthesis is performed according to the parameters, and then ADSR envelope, filtering and reverb effects are superimposed on the resampled and pitch-shifted sampled PCM data or the synthesized waveform data to obtain single-channel pre-processed audio data. Determine if multiple pianos are triggered simultaneously. If so, mix the pre-processed digital timbre data and then limit the mixed digital timbre data to avoid audio distortion. If only a single piano is triggered, directly transmit the PCM audio frame. The processed digital timbre data is encapsulated into PCM audio frames that conform to the transmission standard of the audio output module. The PCM audio frames are then written into the circular buffer of the audio output module to complete the transmission to the audio output module.

8. The digital timbre triggering and music interaction system based on the thumb piano playing structure as described in claim 1, characterized in that: The audio output module includes a ring buffer, a power amplifier, and a sound-generating component; By receiving PCM audio frames that conform to the transmission standard, the PCM audio frames are sequentially written into the built-in circular buffer of the module according to the audio output timing, so as to realize continuous caching of audio data and avoid data interruption. PCM audio frame data is continuously read from the circular buffer according to the preset audio sampling rate, and the digital PCM audio data is converted into analog audio electrical signals through the I2S interface and DAC digital-to-analog conversion circuit. The converted analog audio signal is transmitted to the built-in power amplifier circuit of the module. The power amplifier circuit amplifies the analog audio signal to meet the sound driving requirements of the speaker and headphones. The amplified analog audio signal is synchronously transmitted to the speaker and headphone interfaces of the device, supporting single or multi-channel audio output for speaker playback and headphone listening, to meet different listening needs of users. By using a circular buffer for buffering, high-speed I2S / DAC data conversion, and fast signal amplification by the power amplifier circuit, the entire process of audio data transmission, conversion, and amplification is time-controlled to ensure that the end-to-end output delay from digital timbre triggering to audio output is less than the user-perceptible threshold.

9. The digital timbre triggering and music interaction system based on the thumb piano playing structure as described in claim 1, characterized in that: The interactive control unit loads the rhythm and note sequence data corresponding to the user-selected track and practice content, and shares a unified system time base with the main control processing module to keep the rendering sequence of the note track synchronized with the audio output sequence, ensuring that the position of the waterfall flow notes is accurately aligned with the actual sounding time. Each target note in the musical note sequence is mapped to the corresponding metal plate number according to a preset rule, establishing a one-to-one correspondence between the target note and the physical position of the plate, providing a positional basis for the waterfall flow track rendering; Based on the one-to-one correspondence between the target note and the physical position of the piano, the display component renders a waterfall-style note track corresponding to the position of the piano on the display interface. For the target note that is about to reach the trigger time, the corresponding track and the position of the piano are highlighted in a preset preview time window to clearly indicate which piano the user needs to pluck and the trigger time. By receiving valid trigger information synchronized by the main control processing module, including the piano number and trigger timestamp, and combining it with the time reference to extract the preset trigger timestamp of the target note, the time deviation between the actual trigger timestamp and the target trigger timestamp is calculated. The time deviation is compared with the preset judgment tolerance window. If the time deviation is within the tolerance window range and the trigger paddle number is consistent with the paddle number mapped to the target note, the match is considered successful; otherwise, the match is considered unsuccessful.

10. The digital timbre triggering and music interaction system based on the thumb piano playing structure as described in claim 1, characterized in that: The specific process for automatically adjusting the difficulty parameters based on the user's performance hit rate is as follows: Records performance data for each plucking action in real time, including plectrum number, time deviation, matching judgment result, plucking force and speed. During or after the practice of a piece, the user's performance hit rate is calculated based on the total number of valid triggers and the number of successful matches. The statistically obtained performance hit rate is compared with the preset hit rate threshold range. If the hit rate exceeds the upper limit threshold, the difficulty parameters are automatically increased, including narrowing the judgment error tolerance window, shortening the note preview time window, and expanding the piano-note mapping range. If the hit rate is lower than the lower threshold, the difficulty parameters will be automatically adjusted, including expanding the judgment tolerance window, extending the note preview time window, and narrowing the piano-note mapping range, to achieve adaptive matching of the difficulty parameters.

Images