Interactive fitness sound box system based on audio frequency extraction and multi-modal space mapping

Abstract

The application relates to the technical field of intelligent fitness, in particular to an interactive fitness sound box system based on audio frequency extraction and multi-modal space mapping, which comprises a distributed clock synchronization module, an audio real-time feature extraction module, a multi-dimensional routing matrix module, a beat prediction and adaptive degradation module, a tactile guidance execution module, a low-power motion capture module and a multi-dimensional score closed loop module; the distributed clock synchronization module is used for calculating the radio frequency transmission delay and clock offset between a sound box master control and P wearable nodes based on application layer bidirectional round trip time measurement, so that sub-millisecond global clock phase-locked synchronization is realized; through the distributed clock synchronization module, sub-millisecond time synchronization between the sound box and the multiple wearable nodes is realized, which lays a foundation for accurate motion time comparison; through an M*N to P multi-dimensional routing matrix and DSP real-time audio feature extraction, instant deconstruction and space mapping of any streaming media music are realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent fitness technology, specifically to an interactive fitness speaker system based on audio frequency extraction and multimodal spatial mapping. Background Technology

[0002] With the increasing demand for home fitness, various interactive fitness devices are emerging. Existing technologies can be mainly divided into three categories: First, dance machines and rhythm lightsaber-like devices based on pre-made music libraries, which require manual annotation of the beat and movement chart for each song. Users cannot freely select streaming music, the music library is updated late, and the usage scenarios are limited. Second, fitness mirrors and boxing target devices based on screen visual guidance, which require users to look at the screen or lighting panel throughout the process. Once they turn around, turn their backs, or bend over, they cannot obtain guidance information, which severely limits the freedom of movement. Third, wearable devices that simply follow vibrations, which only trigger vibrations when the music beat is detected. They do not take into account the inherent physiological delay of 200-300ms between human perception of muscle exertion and the actual movement, causing users' movements to always lag behind the beat, resulting in a frustrating feeling of being "half a beat behind".

[0003] In terms of multi-device time synchronization, existing wearable devices mostly use passive time synchronization based on connection events, with synchronization errors typically in the range of 10-30ms, which cannot meet the requirements for action time comparison with millisecond-level precision. In terms of action detection, existing solutions mostly use complete attitude calculation or square root operation to calculate acceleration modulus, which has high computational complexity and high power consumption, making it difficult to achieve long-term real-time detection on wearable devices powered by button batteries. Summary of the Invention

[0004] The purpose of this invention is to provide an interactive fitness speaker system based on audio frequency extraction and multimodal spatial mapping to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: An interactive fitness speaker system based on audio frequency extraction and multimodal spatial mapping includes: The distributed clock synchronization module is used to calculate the radio frequency transmission delay and clock offset between the speaker master controller and p wearable nodes based on the bidirectional round-trip time measurement at the application layer, so as to achieve sub-millisecond global clock phase-locked synchronization. The real-time audio feature extraction module is used to receive Bluetooth audio streams through DMA ping-pong buffer, call the DSP instruction set to perform fixed-point FFT, and calculate the spectral flux mutation feature values ​​of multiple channels and multiple frequency bands. The multidimensional routing matrix module is used to establish a mapping weight matrix from the number of input channels M and frequency bands N to the number of physical output nodes P, and to convert the audio feature vectors into the driving energy threshold of each node in real time. The beat prediction and adaptive degradation module is used to predict the subsequent replay time based on the exponentially weighted moving average, output guidance instructions by pre-setting physiological compensation, and calculate the beat confidence to achieve adaptive gating degradation. The haptic guidance execution module is used to generate a crisp haptic cues by controlling the linear resonant motor through overdrive and active braking timing after receiving a trigger command from the wearable node. The low-power motion capture module is used to wake up the MCU based on motion interruption, and uses the L2 norm square algorithm without square root to detect acceleration extreme points and record the time of motion occurrence under a globally unified timestamp. The multi-dimensional scoring closed-loop module is used to calculate the final score based on the Gaussian error function of the action time and the predicted retake time, combined with the action energy penalty term, and provide real-time feedback.

[0006] Preferably, in the distributed clock synchronization module, the process for calculating the radio frequency transmission delay and clock offset between the speaker master controller and P wearable nodes based on the application layer bidirectional round-trip time measurement, and achieving sub-millisecond-level global clock phase-locked synchronization, is as follows: After the connection is established, the speaker controller starts a 1MHz hardware timer, generates a timestamp t1, and sends a SYNC_REQ synchronization request packet to the wearable node. When the receive interrupt is triggered, the wearable node reads the local timer to get t2, replies with a SYNC_ACK packet within the same connection event and records the sending time t3; The speaker's main control receives a return packet, triggering an interrupt record t4. The one-way delay d is calculated using formula (1), and the clock offset is calculated using formula (2). : (1) (2) Will Write the offset to the calibration register of the wearable node, and then add this offset to all subsequent timestamps to obtain a globally unified time.

[0007] Preferably, in the real-time audio feature extraction module, the process of receiving the Bluetooth audio stream through a DMA ping-pong buffer, calling the DSP instruction set to perform a fixed-point FFT, and calculating the spectral flux abrupt change feature values ​​for multiple channels and frequency bands is as follows: The Bluetooth audio module receives A2DP protocol audio PCM data and uses a ping-pong buffering mechanism with 2×1024 sampling points through the DMA controller to transfer the data without CPU intervention. When the half-buffered transfer completes and an interrupt is triggered, the CMSIS-DSP assembly library is invoked to perform a 512-point Fast Fourier Transform using the FPU to obtain the amplitude at each frequency point. Where c is the channel number, m is the frame number, and k is the frequency point number; For each channel c and each frequency band b, calculate the spectral flux characteristics according to equation (3): (3) in It is a half-wave rectifier function. Only the rising edge of the energy is retained.

[0008] Preferably, in the multi-dimensional routing matrix module, the process of establishing a mapping weight matrix from the number of input channels M and frequency bands N to the number of physical output nodes P, and converting the audio feature vectors into the driving energy thresholds of each node in real time, is as follows: Construction dimension The routing configuration matrix W, matrix elements This represents the energy allocation weight from the combination of channel c-band b to the p-th physical node; Calculate the output energy vector according to equation (4): (4) in for A dimensional spectral flux feature column vector. Output energy vector in P dimensions; Each element corresponds to the PWM duty cycle or vibration intensity threshold of a physical node, which is written to the speaker LED driver IC via the I2C bus and simultaneously pushed to the corresponding wearable node via BLE.

[0009] Preferably, in the beat prediction and adaptive degradation module, the process of predicting subsequent replay times based on exponentially weighted moving averages, outputting guidance instructions with pre-set physiological compensation amounts, and calculating beat confidence to achieve adaptive gating degradation is as follows: To maintain a continuous IBI sequence, the average IBI is calculated using an exponentially weighted moving average according to equation (5): (5) in For smoothing coefficients; Predict the time of the next repetition according to formula (6). According to formula (7) in advance Set trigger time : (6) (7) Calculate the beat confidence level C according to formula (8): (8) in The variance of the sliding window between shots. This is the sensitivity coefficient; When C is below the confidence threshold When the position confidence flag is set to false, the output of the advanced guidance command is cut off, and the system is downgraded to real-time follow mode.

[0010] Preferably, in the tactile guidance execution module, after the wearable node receives the trigger command, the process of generating a crisp tactile cues by controlling the linear resonant motor through overdrive and active braking timing is as follows: After receiving the trigger command, the wearable node controls the motor driver IC to output a PWM waveform at the resonant frequency to the LRA linear resonant motor; An initial 10ms of 100% duty cycle is applied for overdrive to overcome the static friction of the motor and achieve rapid oscillation. At the moment the guidance period ends, a 180-degree inverted PWM waveform is output for active braking, consuming the kinetic energy of the oscillator to achieve an emergency stop. The entire tactile feedback pulse width is controlled within 30-50ms, forming a button-like tactile sensation without aftershock.

[0011] Preferably, in the low-power motion capture module, the process of waking up the MCU based on motion interruption, detecting acceleration extrema using a square root-free L2 norm square algorithm, and recording the moment of motion occurrence under a globally unified timestamp is as follows: The wearable node is normally in a low-power sleep mode. When the IMU hardware detects that the acceleration exceeds the threshold, it wakes up the MCU through the motion interrupt pin. After being woken up, the MCU performs integer multiplication and addition operations, and calculates the square of the L2 norm of dynamic acceleration according to equation (9). : (9) in , , This is the dynamic acceleration value after subtracting the gravitational component; Continuous sampling detection The zero-crossing point is the moment when the motion energy reaches its peak. ; The global uniform action time is obtained by superimposing the clock synchronization offset. The package is sent back to the speaker's main control via BLE.

[0012] Preferably, in the multi-dimensional scoring closed-loop module, the process of calculating the final score and providing real-time feedback based on the Gaussian error function of the action time and predicted retake time, combined with the action energy penalty term, is as follows: The speaker's main controller collects the motion time transmitted back from each wearable node. With peak energy ; Calculate the score for a single action using formula (10): (10) in For full marks, The standard deviation of the Gaussian function of the time error. Energy for the target action; The time error term is weighted using a Gaussian function, and the energy term is truncated using normalization to prevent cheating by light touch. Based on the score level, corresponding voice feedback is triggered, the combo count display on the panel is updated, and the interaction loop is completed.

[0013] Compared with the prior art, the beneficial effects of the present invention are: This invention achieves sub-millisecond time synchronization between the speaker and multiple wearable nodes through a distributed clock synchronization module, laying the foundation for accurate motion time comparison. Through an M×N→P multi-dimensional routing matrix and DSP real-time audio feature extraction, it enables instant deconstruction and spatial mapping of any streaming music, completely eliminating the limitations of pre-built music libraries. Users can use any music app to play any song for fitness. Through a beat prediction and adaptive degradation module, it outputs haptic guidance commands 250ms in advance, perfectly compensating for human physiological reaction delays and device physical delays, solving the "half-beat slow" problem of traditional devices. The time-confidence gating mechanism ensures that users are not misled when the rhythm is chaotic; through the square root-free extreme value detection algorithm of the low-power motion capture module, precise motion peak capture with microwatt-level power consumption is achieved on the wearable device. Combined with the pure tactile guidance method, users can obtain guidance information in any posture such as turning around, turning their back, or bending over without looking at the screen, completely liberating their vision; finally, the multi-dimensional scoring closed-loop module realizes the closed-loop verification of physical quantities across the entire chain. From time synchronization, audio analysis, predictive guidance, motion capture to scoring feedback, a complete technical closed loop is formed, providing users with an immersive, unrestricted, and highly accurate interactive fitness experience. Attached Figure Description

[0014] Figure 1 This is a module architecture diagram of the interactive fitness speaker system based on audio frequency extraction and multimodal spatial mapping of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] like Figure 1As shown, the interactive fitness speaker system based on audio frequency extraction and multimodal spatial mapping includes: The distributed clock synchronization module is used to calculate the RF transmission delay and clock offset between the speaker master controller and P wearable nodes based on the application layer bidirectional round-trip time measurement. The process of achieving sub-millisecond global clock phase-locked synchronization is as follows: After the connection is established, the speaker controller starts a 1MHz hardware timer, generates a timestamp t1, and sends a SYNC_REQ synchronization request packet to the wearable node. When the receive interrupt is triggered, the wearable node reads the local timer to get t2, replies with a SYNC_ACK packet within the same connection event and records the sending time t3; The speaker's main control receives a return packet, triggering an interrupt record t4. The one-way delay d is calculated using formula (1), and the clock offset is calculated using formula (2). : (1) (2) Will Write the offset to the calibration register of the wearable node, and then add this offset to all subsequent timestamps to obtain a globally unified time.

[0017] The real-time audio feature extraction module receives Bluetooth audio streams via DMA ping-pong buffer, calls the DSP instruction set to perform fixed-point FFT, and calculates the spectral flux abrupt change feature values ​​for multiple channels and frequency bands as follows: The Bluetooth audio module receives A2DP protocol audio PCM data and uses a ping-pong buffering mechanism with 2×1024 sampling points through the DMA controller to transfer the data without CPU intervention. When the half-buffered transfer completes and an interrupt is triggered, the CMSIS-DSP assembly library is invoked to perform a 512-point Fast Fourier Transform using the FPU to obtain the amplitude at each frequency point. Where c is the channel number, m is the frame number, and k is the frequency point number; For each channel c and each frequency band b, calculate the spectral flux characteristics according to equation (3): (3) in It is a half-wave rectifier function. Only the rising edge of the energy is retained.

[0018] The multi-dimensional routing matrix module is used to establish a mapping weight matrix from the number of input channels M and frequency bands N to the number of physical output nodes P. The process of converting audio feature vectors into driving energy thresholds for each node in real time is as follows: Construction dimension The routing configuration matrix W, matrix elements This represents the energy allocation weight from the combination of channel c-band b to the p-th physical node; Calculate the output energy vector according to equation (4): (4) in for A dimensional spectral flux feature column vector. Output energy vector in P dimensions; Each element corresponds to the PWM duty cycle or vibration intensity threshold of a physical node, which is written to the speaker LED driver IC via the I2C bus and simultaneously pushed to the corresponding wearable node via BLE.

[0019] The beat prediction and adaptive degradation module is used to predict subsequent replay times based on exponentially weighted moving averages, pre-set physiological compensation amounts to output guiding instructions, and calculate beat confidence to achieve adaptive gating degradation. The process is as follows: To maintain a continuous IBI sequence, the average IBI is calculated using an exponentially weighted moving average according to equation (5): (5) in For smoothing coefficients; Predict the time of the next repetition according to formula (6). According to formula (7) in advance Set trigger time : (6) (7) Calculate the beat confidence level C according to formula (8): (8) in The variance of the sliding window between shots. This is the sensitivity coefficient; When C is below the confidence threshold When the position confidence flag is set to false, the output of the advanced guidance command is cut off, and the system is downgraded to real-time follow mode.

[0020] The haptic guidance execution module, after receiving a trigger command from the wearable node, uses overdrive and active braking timing control of the linear resonant motor to generate a crisp haptic cues. The process is as follows: After receiving the trigger command, the wearable node controls the motor driver IC to output a PWM waveform at the resonant frequency to the LRA linear resonant motor; An initial 10ms of 100% duty cycle is applied for overdrive to overcome the static friction of the motor and achieve rapid oscillation. At the moment the guidance period ends, a 180-degree inverted PWM waveform is output for active braking, consuming the kinetic energy of the oscillator to achieve an emergency stop. The entire tactile feedback pulse width is controlled within 30-50ms, forming a button-like tactile sensation without aftershock.

[0021] The low-power motion capture module is used to wake up the MCU based on motion interrupts. It uses a square root-free L2 norm square algorithm to detect acceleration extrema and records the moment of motion occurrence under a globally unified timestamp. The wearable node is normally in a low-power sleep mode. When the IMU hardware detects that the acceleration exceeds the threshold, it wakes up the MCU through the motion interrupt pin. After being woken up, the MCU performs integer multiplication and addition operations, and calculates the square of the L2 norm of dynamic acceleration according to equation (9). : (9) in , , This is the dynamic acceleration value after subtracting the gravitational component; Continuous sampling detection The zero-crossing point is the moment when the motion energy reaches its peak. ; The global uniform action time is obtained by superimposing the clock synchronization offset. The package is sent back to the speaker's main control via BLE.

[0022] The multi-dimensional scoring closed-loop module calculates the final score based on the Gaussian error function of motion time and predicted retake time, combined with a motion energy penalty term, and provides real-time feedback. The speaker's main controller collects the motion time transmitted back from each wearable node. With peak energy ; Calculate the score for a single action using formula (10): (10) in For full marks, The standard deviation of the Gaussian function of the time error. Energy for the target action; The time error term is weighted using a Gaussian function, and the energy term is truncated using normalization to prevent cheating by light touch. Based on the score level, corresponding voice feedback is triggered, the combo count display on the panel is updated, and the interaction loop is completed.

[0023] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An interactive fitness speaker system based on audio frequency extraction and multimodal spatial mapping, characterized in that, include: The distributed clock synchronization module is used to calculate the radio frequency transmission delay and clock offset between the speaker master controller and P wearable nodes based on the bidirectional round-trip time measurement at the application layer, so as to achieve sub-millisecond global clock phase-locked synchronization. The real-time audio feature extraction module is used to receive Bluetooth audio streams through DMA ping-pong buffer, call the DSP instruction set to perform fixed-point FFT, and calculate the spectral flux mutation feature values ​​of multiple channels and multiple frequency bands. The multidimensional routing matrix module is used to establish a mapping weight matrix from the number of input channels M and frequency bands N to the number of physical output nodes P, and to convert the audio feature vectors into the driving energy threshold of each node in real time. The beat prediction and adaptive degradation module is used to predict the subsequent replay time based on the exponentially weighted moving average, output guidance instructions by pre-setting physiological compensation, and calculate the beat confidence to achieve adaptive gating degradation. The haptic guidance execution module is used to generate a crisp haptic cues by controlling the linear resonant motor through overdrive and active braking timing after receiving a trigger command from the wearable node. The low-power motion capture module is used to wake up the MCU based on motion interruption, and uses the L2 norm square algorithm without square root to detect acceleration extreme points and record the time of motion occurrence under a globally unified timestamp. The multi-dimensional scoring closed-loop module is used to calculate the final score based on the Gaussian error function of the action time and the predicted retake time, combined with the action energy penalty term, and provide real-time feedback.

2. The interactive fitness speaker system based on audio frequency extraction and multimodal spatial mapping according to claim 1, characterized in that, In the distributed clock synchronization module, the process for calculating the radio frequency transmission delay and clock offset between the speaker master controller and P wearable nodes based on the application layer bidirectional round-trip time measurement, and achieving sub-millisecond-level global clock phase-locked synchronization, is as follows: After the connection is established, the speaker controller starts a 1MHz hardware timer, generates a timestamp t1, and sends a SYNC_REQ synchronization request packet to the wearable node. When the receive interrupt is triggered, the wearable node reads the local timer to get t2, replies with a SYNC_ACK packet within the same connection event and records the sending time t3; The speaker's main control receives a return packet, triggering an interrupt record t4. The one-way delay d is calculated using formula (1), and the clock offset is calculated using formula (2). : （1） （2） Will Write the offset to the calibration register of the wearable node, and then add this offset to all subsequent timestamps to obtain a globally unified time.

3. The interactive fitness speaker system based on audio frequency extraction and multimodal spatial mapping according to claim 1, characterized in that, In the real-time audio feature extraction module, the process of receiving Bluetooth audio streams through a DMA ping-pong buffer, executing fixed-point FFT using the DSP instruction set, and calculating the spectral flux abrupt change feature values ​​across multiple channels and frequency bands is as follows: The Bluetooth audio module receives A2DP protocol audio PCM data and uses a ping-pong buffering mechanism with 2×1024 sampling points through the DMA controller to transfer the data without CPU intervention. When the half-buffered transfer completes and an interrupt is triggered, the CMSIS-DSP assembly library is invoked to perform a 512-point Fast Fourier Transform using the FPU to obtain the amplitude at each frequency point. Where c is the channel number, m is the frame number, and k is the frequency point number; For each channel c and each frequency band b, calculate the spectral flux characteristics according to equation (3): （3） in It is a half-wave rectifier function. Only the rising edge of the energy is retained.

4. The interactive fitness speaker system based on audio frequency extraction and multimodal spatial mapping according to claim 1, characterized in that, In the multi-dimensional routing matrix module, the process of establishing a mapping weight matrix from the number of input channels M and frequency bands N to the number of physical output nodes P, and converting the audio feature vectors into the driving energy thresholds of each node in real time, is as follows: Construction dimension The routing configuration matrix W, matrix elements This represents the energy allocation weight from the combination of channel c-band b to the p-th physical node; Calculate the output energy vector according to equation (4): （4） in for A dimensional spectral flux feature column vector. Output energy vector in P dimensions; Each element corresponds to the PWM duty cycle or vibration intensity threshold of a physical node, which is written to the speaker LED driver IC via the I2C bus and pushed to the corresponding wearable node via BLE.

5. The interactive fitness speaker system based on audio frequency extraction and multimodal spatial mapping according to claim 1, characterized in that, In the beat prediction and adaptive degradation module, the process of predicting subsequent replay times based on exponentially weighted moving averages, outputting guidance instructions with pre-set physiological compensation amounts, and calculating beat confidence to achieve adaptive gating degradation is as follows: To maintain a continuous IBI sequence, the average IBI is calculated using an exponentially weighted moving average according to equation (5): （5） in For smoothing coefficients; Predict the time of the next repetition according to formula (6). According to formula (7) in advance Set trigger time : （6） （7） Calculate the beat confidence level C according to formula (8): （8） in The variance of the sliding window between shots. This is the sensitivity coefficient; When C is below the confidence threshold When the position confidence flag is set to false, the output of the advanced guidance command is cut off, and the system is downgraded to real-time follow mode.

6. The interactive fitness speaker system based on audio frequency extraction and multimodal spatial mapping according to claim 1, characterized in that, In the tactile guidance execution module, after the wearable node receives the trigger command, the process of generating a crisp tactile cues by controlling the linear resonant motor through overdrive and active braking timing is as follows: After receiving the trigger command, the wearable node controls the motor driver IC to output a PWM waveform at the resonant frequency to the LRA linear resonant motor; An initial 10ms of 100% duty cycle is applied for overdrive to overcome the static friction of the motor and achieve rapid oscillation. At the moment the guidance period ends, a 180-degree inverted PWM waveform is output for active braking, consuming the kinetic energy of the oscillator to achieve an emergency stop. The entire tactile feedback pulse width is controlled within 30-50ms, forming a button-like tactile sensation without aftershock.

7. The interactive fitness speaker system based on audio frequency extraction and multimodal spatial mapping according to claim 1, characterized in that, In the low-power motion capture module, the process of waking up the MCU based on motion interruption, detecting acceleration extreme points using the L2 norm squared algorithm without square root extraction, and recording the moment of motion occurrence under a globally unified timestamp is as follows: The wearable node is normally in a low-power sleep mode. When the IMU hardware detects that the acceleration exceeds the threshold, it wakes up the MCU through the motion interrupt pin. After being woken up, the MCU performs integer multiplication and addition operations, and calculates the L2 norm square of the dynamic acceleration according to equation (9). : （9） in , , This is the dynamic acceleration value after subtracting the gravitational component; Continuous sampling detection The zero-crossing point is the moment when the motion energy reaches its peak. ; The global uniform action time is obtained by superimposing the clock synchronization offset. The package is sent back to the speaker's main control via BLE.

8. The interactive fitness speaker system based on audio frequency extraction and multimodal spatial mapping according to claim 1, characterized in that, In the multi-dimensional scoring closed-loop module, the process of calculating the final score and providing real-time feedback based on the Gaussian error function of motion time and predicted retake time, combined with the motion energy penalty term, is as follows: The speaker's main controller collects the motion time transmitted back from each wearable node. With peak energy ; Calculate the score for a single action using formula (10): (10) in For full marks, The standard deviation of the Gaussian function of the time error. Energy for the target action; The time error term is weighted using a Gaussian function, and the energy term is truncated using normalization to prevent cheating by light touch. Based on the score level, corresponding voice feedback is triggered, the combo count display on the panel is updated, and the interaction loop is completed.

Images