Intelligent fitness device interaction system based on multi-modal collaborative perception and large model

Abstract

The present application relates to the technical field of fitness equipment interaction system, and particularly relates to an intelligent fitness equipment interaction system based on multi-modal collaborative sensing and large model, which comprises a communication routing module, a multi-modal sensing module, a decision management hub module, an execution feedback module and an image updating module, the multi-modal sensing module comprises a voiceprint recognition locking unit, an instruction auxiliary recognition unit, a physiological limit evaluation unit and a motion trajectory positioning unit; the system solves technical problems such as strong noise interference, single sensing dimension and rigid feedback logic in a running scene by constructing an intelligent interaction closed loop integrating identity locking, multi-modal instruction compensation, dynamic physiology, physical monitoring and emotional logic decision; the present application can integrate voiceprint recognition, lip-aid, micro-expression monitoring and large model deep analysis, and construct a fitness interaction closed loop integrating sensing, decision and feedback.

Description

Technical Field

[0001] This invention relates to the field of fitness equipment interaction system technology, and in particular to an intelligent fitness equipment interaction system based on multimodal collaborative perception and large model. Background Technology

[0002] With the deep integration of artificial intelligence and Internet of Things (IoT) technologies, smart fitness equipment (such as smart treadmills) has gradually entered ordinary households. These devices not only provide basic exercise support functions, but also integrate preliminary interactive systems, such as touch screens and simple keyword voice control, aiming to enhance the user's exercise experience and scientific approach.

[0003] Currently, mainstream smart treadmill interaction solutions on the market mainly rely on the following technical paths: 1. Basic voice interaction technology (ASR and TTS): Existing devices typically integrate Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) modules. Users trigger the system through preset wake words and execute simple commands (such as "start running" or "increase the incline"). Some high-end models introduce keyword matching technology for processing basic task commands. 2. Single physiological monitoring technology: Most treadmills acquire users' physiological data through handgrip pulse sensors or external heart rate monitors and display exercise intensity based on heart rate zones. Some devices also have basic camera functionality for facial recognition login or simple video calls. 3. Preliminary application of Large Language Models (LLM): In recent years, some systems have begun to try to integrate large language models to achieve more natural dialogue logic. These systems process dialogue text through the Transformer architecture, generating more fluent responses than traditional rule bases.

[0004] Despite advancements in automation, current technologies still face significant limitations in complex exercise scenarios. First, they exhibit poor robustness: treadmills generate continuous motor noise and footstep impact sounds during operation. Existing ASR systems primarily focus on semantic accuracy, but recognition rates drop drastically in noisy environments and are easily interfered with by ambient noise. Second, they lack multimodal emotional and physiological depth perception: most existing systems cannot recognize and respond to the user's true physical state in real time. For example, when users reach their physical limits, they often exhibit pain through micro-expressions or distorted running postures (trajectory deviation), but current technologies rely heavily on isolated data segment analysis, making continuous tracking of emotional and physical states difficult. Third, their decision-making feedback is rigid and "irrational": system responses and device control are often static and linear. When users exhibit extreme fatigue or negative emotions, the system may still proceed with the exercise plan according to the established process, rather than prioritizing emotional reassurance or automatically reducing intensity. This lack of emotional intelligence in the interaction strategy easily leads to a deterioration in user experience and even increases the risk of sports injuries. Fourth, user profile construction is incomplete and lagging: Existing systems rely heavily on simple indicators such as call duration or task completion rate to evaluate user exercise quality, lacking a comprehensive consideration of postural evolution, emotional responses, and physiological fluctuations during exercise. User profile construction often depends on post-event analysis, lacking the ability to update in real time during interaction and guide the next exercise session.

[0005] In summary, existing smart treadmills still have significant limitations in terms of environmental adaptability, dynamic perception, and personalized services. Therefore, there is an urgent need for an interactive system that can integrate voiceprint recognition, lip reading assistance, micro-expression monitoring, and deep analysis using large models to construct a closed-loop fitness interaction system that integrates perception, decision-making, and feedback. Summary of the Invention

[0006] The purpose of this invention is to provide an intelligent fitness equipment interaction system based on multimodal collaborative perception and large model, which can integrate voiceprint recognition, lip reading assistance, micro-expression monitoring and deep analysis of large model to construct a fitness interaction closed loop integrating perception, decision-making and feedback.

[0007] To achieve the above objectives, this invention provides an intelligent fitness equipment interaction system based on multimodal collaborative perception and a large model, comprising a communication routing module, a multimodal perception module, a decision management center module, an execution feedback module, and a profile update module. The multimodal perception module includes a voiceprint recognition locking unit, a command-assisted recognition unit, a physiological limit assessment unit, and a motion trajectory positioning unit. The voiceprint recognition locking unit, the command-assisted recognition unit, the physiological limit assessment unit, and the motion trajectory positioning unit are respectively connected to the communication routing module. The decision management center module is connected to the voiceprint recognition locking unit, the command-assisted recognition unit, the physiological limit assessment unit, and the motion trajectory positioning unit. The execution feedback module is connected to the decision management center module. The profile update module is connected to both the decision management center module and the execution feedback module. The communication routing module is used to collect audio and video streams in real time during the fitness process and distribute the audio and video streams in multiple parallel streams. The voiceprint recognition and locking unit is used to extract the voiceprint features of the current user from the audio stream, lock the user's identity, and filter out environmental interference sound sources. The instruction-assisted recognition unit is used to process audio streams and video streams in parallel, fuse speech recognition and lip reading results, and finally recognize text instructions. The physiological limit assessment unit is used to identify the user's facial micro-expression features from the video stream; The motion trajectory localization unit is used to extract user limb motion trajectory features from the video stream; The decision management central module is used to extract the multimodal quantitative features output by the multimodal perception module, and convert them into structured prompt words to input into the large language model for fine-tuning in the fitness field, generating feedback instructions; The execution feedback module is used to execute emotional voice output and hardware linkage control in parallel according to feedback instructions; The profile update module is used to perform alignment analysis on multimodal data throughout the entire process and dynamically update user profiles.

[0008] The communication routing module utilizes RTP stream mirroring to perform zero-copy distribution of audio and video streams in kernel mode, simultaneously cloning one input stream into six parallel output streams.

[0009] The voiceprint recognition and locking unit uses the ECAPA-TDNN architecture as the backbone network. During the training process, it uses pre-collected pure physical noise from various models of treadmills as negative samples and uses contrastive learning to improve the model's voiceprint separation capability under extreme noise conditions.

[0010] The instruction-assisted recognition unit processes the audio and video streams in parallel and automatically adjusts the weights of audio and visual features based on the real-time signal-to-noise ratio. When the audio signal-to-noise ratio is higher than a preset threshold, the feature weights of speech recognition are increased, and when the audio signal-to-noise ratio is not higher than the preset threshold, the feature weights of visual lip reading are increased.

[0011] The physiological limit assessment unit is based on the emotion2vec_plus_large architecture and introduces a temporal attention mechanism, using video streams with exercise physiology tags for fine-tuning.

[0012] The motion trajectory positioning unit embeds human anatomical geometric constraints in the MediaPipe Holistic architecture and uses a Kalman filter based on a 30-frame sliding window motion smoother to eliminate joint jumps caused by camera motion blur and accurately calculate ground contact balance.

[0013] The decision management central module integrates a mainstream large model that has undergone full parameter fine-tuning and LoRA lightweight fine-tuning.

[0014] The execution feedback module includes a speech synthesis engine and a hardware control interface. The speech synthesis engine is used to output emotional voice guidance that is adapted to the user's current state, and the hardware control interface is used to adjust the operating parameters of the fitness equipment in conjunction with the user's current state.

[0015] This invention discloses an intelligent fitness equipment interaction system based on multimodal collaborative perception and a large model. The system uses the user's micro-expressions and movement trajectories as real-time prior knowledge to drive a large language model (LLM) to generate feedback commands with professional depth and emotional warmth. Through voiceprint locking and dual verification of "voice + lip reading," it completely solves the noise interference problem in running scenarios from both physical and semantic layers. By correlating facial micro-expressions with limb movement trajectories, it establishes a safety warning mechanism that is more sensitive and earlier than heart rate monitoring. This invention utilizes a large model as the decision-making brain, transforming multi-dimensional perceptual data into real-time guidance suggestions with professional depth and emotional resonance, achieving dynamic and humanized interaction strategies. This system solves technical challenges such as strong noise interference, single perception dimension, and rigid feedback logic in running scenarios by constructing an intelligent interaction closed loop integrating identity locking, multimodal command compensation, dynamic physiological / physical monitoring, and emotional logic decision-making. This invention integrates voiceprint recognition, lip reading assistance, micro-expression monitoring, and large model deep analysis to construct an integrated fitness interaction closed loop encompassing perception, decision-making, and feedback. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0017] Figure 1 This is a schematic diagram of the structure of an intelligent fitness equipment interaction system based on multimodal collaborative perception and a large model according to the present invention.

[0018] Figure 2 This is a flowchart illustrating the operation of an intelligent fitness equipment interaction system based on multimodal collaborative perception and a large model, according to the present invention.

[0019] Figure 3 This is a flowchart of the operation of the multimodal parallel sensing matrix and dynamic weighted fusion strategy of the present invention.

[0020] Figure 4 This is a schematic diagram of the intelligent decision-making and emotional feedback closed loop of the present invention.

[0021] Figure 5 This is a schematic diagram of the end-to-end data processing and profile update process of the present invention.

[0022] 101-Communication routing module, 102-Multimodal perception module, 103-Voiceprint recognition and locking unit, 104-Command-assisted recognition unit, 105-Physiological limit assessment unit, 106-Motion trajectory positioning unit, 107-Decision management center module, 108-Execution feedback module, 109-Portrait update module. Detailed Implementation

[0023] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0024] Please see Figures 1-5 This invention provides an intelligent fitness equipment interaction system based on multimodal collaborative perception and a large model, including a communication routing module 101, a multimodal perception module 102, a decision management center module 107, an execution feedback module 108, and a profile update module 109; the multimodal perception module 102 includes a voiceprint recognition locking unit 103, a command-assisted recognition unit 104, a physiological limit assessment unit 105, and a motion trajectory positioning unit 106; through the aforementioned scheme, voiceprint recognition, lip reading assistance, micro-expression monitoring, and deep analysis of a large model can be integrated to construct a fitness interaction closed loop integrating perception, decision-making, and feedback.

[0025] In this specific embodiment, the voiceprint recognition locking unit 103, the command-assisted recognition unit 104, the physiological limit assessment unit 105, and the motion trajectory positioning unit 106 are respectively connected to the communication routing module 101; the decision management central module 107 is respectively connected to the voiceprint recognition locking unit 103, the command-assisted recognition unit 104, the physiological limit assessment unit 105, and the motion trajectory positioning unit 106; the execution feedback module 108 is connected to the decision management central module 107; and the profile update module 109 is respectively connected to the decision management central module 107 and the execution feedback module 108. The communication routing module 101 is used to collect audio and video streams in real time during the fitness process and distribute the audio and video streams in multiple parallel streams. The voiceprint recognition and locking unit 103 is used to extract the voiceprint features of the current user from the audio stream, lock the user's identity, and filter out environmental interference sound sources. The instruction-assisted recognition unit 104 is used to process audio streams and video streams in parallel, fuse speech recognition and lip reading results, and finally recognize text instructions. The physiological limit assessment unit 105 is used to identify the user's facial micro-expression features from the video stream; The motion trajectory localization unit 106 is used to extract user limb motion trajectory features from the video stream; The decision management central module 107 is used to extract the multimodal quantitative features output by the multimodal perception module 102, and convert them into structured prompt words to input into the large language model for fine-tuning in the fitness field, generating feedback instructions; The execution feedback module 108 is used to execute emotional voice output and hardware linkage control in parallel according to feedback instructions; The profile update module 109 is used to perform alignment analysis on the multimodal data throughout the process and dynamically update the user profile.

[0026] In this embodiment, the system extracts the multimodal quantified features by performing voiceprint locking, lip-reading assisted recognition, facial micro-expression monitoring, and motion trajectory localization in parallel. These features are then transformed into structured prompt words and input into a large language model for fine-tuning in the fitness field. Based on the strategy generated by the large model, the system dynamically maps TTS voice parameters and sends hardware adjustment commands in conjunction with the model. Through end-to-end data alignment, the system analyzes the conversion efficiency between micro-expressions and trajectories, achieving closed-loop feedback.

[0027] The communication routing module 101 utilizes RTP stream mirroring to perform zero-copy distribution of audio and video streams in kernel mode, simultaneously cloning one input stream into six parallel output streams. Specifically, the communication and multimedia routing hub (based on FreeSWITCH multi-stream concurrency control) is based on the carrier-grade FreeSWITCH platform. It uses a self-developed mod_gym_media plugin to intercept and distribute underlying media streams. Its core technologies include real-time stream mirroring and clock synchronization and jitter buffering mechanisms: the real-time stream mirroring technology utilizes RTP (Real-time Transport Protocol) stream mirroring to perform zero-copy distribution of audio and video streams in kernel mode. The system clones one input stream into six parallel output streams, each supplied to a different sensing module. Clock synchronization and jitter buffering address potential camera frame rate fluctuations caused by the treadmill's high-speed movement. The system introduces a global clock reference, injecting a synchronization sequence number into the media packet header to ensure complete alignment of video frames (micro-expressions / trajectory) and audio packets (voiceprints / ASR) within 1ms. To address potential data packet loss due to motor electromagnetic interference, this module integrates an adaptive forward error correction (FEC) algorithm derived from real-world motion scenario testing, ensuring the sensing layer receives a highly complete data source.

[0028] Secondly, the voiceprint recognition locking unit 103 uses the ECAPA-TDNN architecture as its backbone network. During training, it uses pre-collected pure physical noise from various treadmill models as negative samples and employs contrastive learning to improve the model's voiceprint separation capability under extreme noise conditions. Specifically, all perception units in the multimodal perception module 102 have undergone specific parameter fine-tuning based on the "multimodal corpus of motion scenes" constructed in this invention to eliminate the performance degradation of the general model in dynamic environments. The voiceprint recognition locking unit 103 (VPR) uses the ECAPA-TDNN architecture as its backbone network. It is fine-tuned using original recordings of 10,000 real users running on treadmills. During training, the system uses pre-collected pure physical noise from 50 different treadmill models as negative samples and employs contrastive learning to improve the model's voiceprint separation capability under extreme noise conditions. Through fine-tuning, the model learns to extract a clean "speaker embedding vector" from a spectrum containing complex harmonics (motor noise). Its equivalent error rate (EER) remains below 1.5% even in a 70dB noise environment.

[0029] Simultaneously, the instruction-assisted recognition unit 104 processes the audio and video streams in parallel, automatically adjusting the weights of audio and visual features based on the real-time signal-to-noise ratio. When the audio signal-to-noise ratio is higher than a preset threshold, the feature weights for speech recognition are increased; when the audio signal-to-noise ratio is not higher than the preset threshold, the feature weights for visual lip reading are increased. For details, please refer to [link to relevant documentation]. Figure 3 The instruction-assisted recognition unit 104 (ASR + Lip-Reading) employs a dual-stream fusion strategy. The audio stream (ASR) performs LoRA fine-tuning on Whisper-Large-v3, injecting a large number of running-related proprietary terms (such as "interval running," "pace adjustment," and "increase incline"). The video stream (Lip-Reading) uses Visual Transformer (ViT) to model the user's lip movement sequence. The fine-tuned data includes lip movement features of the user in a state of rapid breathing and open-mouth panting. At the same time, the system introduces a dynamic weighting mechanism based on signal-to-noise ratio (SNR). When excessive audio background noise is detected, the system automatically increases the feature weight of visual lip reading (from 0.3 to 0.7), significantly correcting ASR transcription errors through visual redundancy verification.

[0030] Furthermore, the physiological limit assessment unit 105 is based on the emotion2vec_plus_large architecture and incorporates a temporal attention mechanism, using video streams labeled with exercise physiology tags for fine-tuning. Specifically, the physiological limit assessment unit 105 (micro-expression recognition) introduces a temporal attention mechanism on the basis of emotion2vec_plus_large, using video streams labeled with exercise physiology tags (annotated by professional sports coaches) for fine-tuning. The fine-tuned model no longer simply recognizes "anger" or "happiness," but can accurately quantify "physiological pain," "persistence," and "anxiety from hypoxia." For the intense facial shaking during running, "affine transformation enhancement" is introduced in the fine-tuning stage, enabling the model to still have stable extraction capabilities for facial features with vertical jumps of up to 10cm.

[0031] Furthermore, the motion trajectory localization unit 106 embeds human anatomical geometric constraints within the MediaPipe Holistic architecture and utilizes a Kalman filter based on a 30-frame sliding window motion smoother to eliminate joint jumps caused by camera motion blur, accurately calculating the ground contact balance. Specifically, the motion trajectory localization unit 106 (PoseEstimation) embeds human anatomical geometric constraints within the MediaPipe Holistic architecture; it performs fine-tuning using numerous running posture deformation cases (such as "dragging run," "body leaning backward," and "center of gravity tilting" due to fatigue). The system introduces a motion smoother based on a 30-frame sliding window, utilizing Kalman filtering to eliminate joint jumps caused by camera motion blur, accurately calculating the ground contact balance score.

[0032] Furthermore, the decision management central module 107 integrates a mainstream large model that has undergone full parameter fine-tuning and LoRA lightweight fine-tuning. For details, please refer to... Figure 4 The Decision Management Central Module 107 (Domain Knowledge Fine-tuning LLM) integrates mainstream large models (such as Qwen2.5-Gym version) that have undergone full parameter fine-tuning and LoRA lightweight fine-tuning. The fine-tuning corpus includes 500,000 professional fitness plans, sports injury prevention guidelines, and physiological abnormality handling protocols. By constructing a "perceptual vector-semantic description" pair, the model can understand that "micro-expression pain index 0.85+ stride stability decreased by 20%" means that the user has entered the "over-exercise risk zone," achieving multimodal instruction alignment. The functional logic is as follows: the LLM acts as an "expert-level decision-making machine," dynamically calling "encouragement mode," "early warning mode," or "intervention mode" based on the real-time status input from the perception layer, generating highly targeted feedback instructions that conform to exercise physiology. The Decision Management Central Module 107 has a prompt word engine and a dialogue management module of the Domain Knowledge Fine-tuning LLM.

[0033] Finally, the execution feedback module 108 includes a speech synthesis engine and a hardware control interface. The speech synthesis engine is used to output emotional voice guidance adapted to the user's current state, and the hardware control interface is used to adjust the operating parameters of the fitness equipment in conjunction with the user's current state. Specifically, the execution feedback module 108 includes two parts: emotional speech synthesis (TTS) and hardware control protocol. The emotional speech synthesis uses CosyVoice or GPT-SoVITS technology, and is fine-tuned based on the voice samples of professional personal trainers (including three tones: encouraging, calm, and caring). The system establishes a "state-acoustic mapping matrix". When the LLM determines that the user is in a low period, the mapping matrix will automatically increase the pitch and energy of the TTS to output an inspiring voice; when it determines that the user is at their physiological limit, it will lower the pitch and output a stable and soothing voice to reduce the user's heart rate stress. The hardware control protocol synchronously sends the decision results to the treadmill main control board through the system-integrated control layer to realize stepless adjustment of the belt speed or automatic incline compensation.

[0034] Please see Figure 5 After a call / exercise ends, the profile update module 109 initiates the data aggregation bus (DataBus) to fully align the original audio and video, sensory tag stream, LLM decision log, and hardware execution status throughout the process. It also introduces intelligent evaluation metrics: calculating the time difference between "system suggestion issued" and "user's condition improves." For example, after the system prompts "adjust breathing," it checks whether the "anxiety value" in the user's micro-expression decreases within 10 seconds, using this as feedback for model closed-loop optimization. Simultaneously, individual characteristics captured during the exercise (e.g., the user is prone to left knee valgus at a pace of 12km / h) are written as incremental knowledge into the user's dedicated vector database, providing data support for parameter initialization the next time the device is turned on.

[0035] This invention presents an intelligent fitness equipment interaction system based on multimodal collaborative perception and a large model. It abandons the sequential approach of first converting speech to text and then analyzing the emotions conveyed in the text, innovatively employing a parallel processing architecture for ASR (Automatic Speech Recognition), VPR (Voice Print Recognition), lip reading, facial micro-expression recognition, and skeletal trajectory localization. Through FreeSWITCH's streaming mirroring technology, this invention ensures absolute alignment of all perception modalities in the temporal dimension, solving the problems of low accuracy and response latency in single-modal recognition under high-decibel background noise environments like treadmills. All models in the multimodal perception module 102 do not directly use general pre-trained weights, but are specifically fine-tuned using a large amount of real-world running scenario data. For voiceprint and ASR, noise adversarial training is performed by injecting treadmill mechanical noise and footstep impact sounds. For micro-expressions and trajectory, "fatigue gradient" and "skeletal stability deviation" data are labeled, enabling the model to quantify and recognize "physiological limits" rather than just general emotions. The technical value lies in eliminating the "incompatibility" of AI models in strenuous exercise and extreme noise environments, achieving industrial-grade perception accuracy. This invention employs a dual prior knowledge injection mechanism of "physiological + physical" into the LLM decision-making process. Quantified micro-expression labels (physiological) and skeletal trajectory offsets (physiological) are converted into structured prompts, which are then injected as the highest-priority prior conditions into the large model's decision-making process. This transforms the large model's response from purely semantic-driven to policy-driven. For example, upon detecting "painful expression + center of gravity shift," the model is forced to trigger a safety intervention strategy, generating instructions with soothing and slowing suggestions. This invention establishes a real-time mapping channel from the LLM decision-making end to TTS tactile parameters and treadmill hardware control. The system can dynamically adjust the speech rate and energy based on the user's fatigue state (e.g., using a gentle, empathetic tone when fatigued) and simultaneously link the hardware to adjust the incline or speed, achieving a unity of acoustic empathy and physical execution. Through the above technical means, this invention achieves the following significant technical effects: 1. A qualitative leap in the reliability and uniqueness of interaction: Through a three-in-one fusion algorithm combining voiceprint, ASR, and lip reading, the system can accurately extract the current user's commands from noisy background noise. Experiments show that even under high-speed treadmill operation (high-decibel noise), the accuracy and anti-interference ability of command recognition have achieved a leap forward, ensuring the uniqueness of operation responses.

[0036] 2. Refined and Proactive Sports Safety Protection: The system no longer relies on lagging heart rate indicators, but instead uses micro-analysis of micro-expressions and trajectory trends to predict user fatigue or postural risks within seconds. Through automatic deceleration or verbal reminders, the system can proactively intervene before danger occurs, controlling sports risks in their nascent stage.

[0037] 3. Multimodal Fusion-Driven Empathic Fitness Guidance: LLM transforms impersonal sensor data (trajectory, facial expressions, heart rate) into context-aware prompts. The system generates feedback that includes not only professional technical advice (such as "Your center of gravity is shifting to the left, please adjust your breathing"), but also soothing or encouraging voice prompts through a TTS (Text-to-Speech) emotional control loop, greatly enhancing user comfort and engagement.

[0038] 4. From Data Compilation to In-Depth "Motion Profile" Mining: After exercise, what's generated is no longer a simple record of steps and calories, but an in-depth report containing emotional fluctuation curves, movement variation statistics, and fatigue evolution patterns. This report can update the user profile in real time, providing a truly scientific basis for users to formulate the next stage of their exercise plan, achieving a precise "one-person-one-policy" service loop.

[0039] The above-disclosed embodiments are merely one or more preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments and equivalent changes made in accordance with the claims of this application still fall within the scope of this application.

Claims

1. An intelligent fitness equipment interaction system based on multimodal collaborative perception and a large model, characterized in that, The system includes a communication routing module, a multimodal perception module, a decision management center module, an execution feedback module, and a profile update module. The multimodal perception module comprises a voiceprint recognition locking unit, a command-assisted recognition unit, a physiological limit assessment unit, and a motion trajectory positioning unit. The voiceprint recognition locking unit, the command-assisted recognition unit, the physiological limit assessment unit, and the motion trajectory positioning unit are each connected to the communication routing module. The decision management center module is connected to the voiceprint recognition locking unit, the command-assisted recognition unit, the physiological limit assessment unit, and the motion trajectory positioning unit. The execution feedback module is connected to the decision management center module. The profile update module is connected to both the decision management center module and the execution feedback module. The communication routing module is used to collect audio and video streams in real time during the fitness process and distribute the audio and video streams in multiple parallel streams. The voiceprint recognition and locking unit is used to extract the voiceprint features of the current user from the audio stream, lock the user's identity, and filter out environmental interference sound sources. The instruction-assisted recognition unit is used to process audio streams and video streams in parallel, fuse speech recognition and lip reading results, and finally recognize text instructions. The physiological limit assessment unit is used to identify the user's facial micro-expression features from the video stream; The motion trajectory localization unit is used to extract user limb motion trajectory features from the video stream; The decision management central module is used to extract the multimodal quantitative features output by the multimodal perception module, and convert them into structured prompt words to input into the large language model for fine-tuning in the fitness field, generating feedback instructions; The execution feedback module is used to execute emotional voice output and hardware linkage control in parallel according to feedback instructions; The profile update module is used to perform alignment analysis on multimodal data throughout the entire process and dynamically update user profiles.

2. The intelligent fitness equipment interaction system based on multimodal collaborative perception and large model as described in claim 1, characterized in that, The communication routing module utilizes RTP stream mirroring to perform zero-copy distribution of audio and video streams in kernel mode, simultaneously cloning one input stream into six parallel output streams.

3. The intelligent fitness equipment interaction system based on multimodal collaborative perception and large model as described in claim 1, characterized in that, The voiceprint recognition locking unit uses the ECAPA-TDNN architecture as the backbone network. During the training process, the pure physical background noise of various treadmills of different models is collected in advance as negative samples. Contrastive learning is used to improve the model's voiceprint separation ability under extreme background noise.

4. The intelligent fitness equipment interaction system based on multimodal collaborative perception and large model as described in claim 1, characterized in that, The instruction-assisted recognition unit processes the audio and video streams in parallel and automatically adjusts the weights of audio and visual features based on the real-time signal-to-noise ratio. When the audio signal-to-noise ratio is higher than a preset threshold, the feature weights of speech recognition are increased; when the audio signal-to-noise ratio is not higher than the preset threshold, the feature weights of visual lip reading are increased.

5. The intelligent fitness equipment interaction system based on multimodal collaborative perception and large model as described in claim 1, characterized in that, The physiological limit assessment unit is based on the emotion2vec_plus_large architecture and introduces a temporal attention mechanism, using video streams with motion physiology tags for fine-tuning.

6. The intelligent fitness equipment interaction system based on multimodal collaborative perception and large model as described in claim 1, characterized in that, The motion trajectory positioning unit embeds human anatomical geometric constraints in the MediaPipe Holistic architecture and uses a Kalman filter based on a 30-frame sliding window motion smoother to eliminate joint jumps caused by camera motion blur and accurately calculate ground contact balance.

7. The intelligent fitness equipment interaction system based on multimodal collaborative perception and large model as described in claim 1, characterized in that, The decision management central module integrates mainstream large models that have undergone full parameter fine-tuning and LoRA lightweight fine-tuning.

8. The intelligent fitness equipment interaction system based on multimodal collaborative perception and large model as described in claim 1, characterized in that, The execution feedback module includes a speech synthesis engine and a hardware control interface. The speech synthesis engine is used to output emotional voice guidance that is adapted to the user's current state, and the hardware control interface is used to adjust the operating parameters of the fitness equipment in conjunction with the user's current state.

Images