Intelligent sports action teaching system and method based on multi-modal large model

Abstract

The invention discloses an intelligent sports action teaching system and method based on a multi-modal large model, and relates to the technical field of intelligent teaching. According to the method, directional microphones and high-speed cameras are arranged in a badminton training field, and band-pass filtering, framing processing and short-time energy calculation of acoustic signals are combined; according to the method, the energy Ehit of the ball hitting window and the energy Ebase of the background noise can be obtained, and an acoustic energy ratio index Rst is further constructed. According to the mode, high signal-to-noise ratio extraction of the ball hitting acoustic signal in a complex environment is guaranteed, and the visual contact moment Tvis can be accurately positioned in a visual mode. Compared with an existing teaching method which only depends on visual detection, the method can still keep high robustness under the conditions of illumination variation, site noise interference and the like, and the accuracy of ball hitting action data collection and analysis is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent teaching technology, specifically to an intelligent teaching system and method for sports movements based on a multimodal large model. Background Technology

[0002] With the rapid development of intelligent technology, multimodal large models are increasingly being widely used in sports training and instructional support. Badminton, as a typical high-speed competitive sport, demands extremely high precision in the timing, speed, and accuracy of its strokes. Its instruction and training evaluation have long relied on human experience and visual observation. However, relying solely on human data or data from a single modality often fails to comprehensively capture the quality of the movements. In recent years, the fusion analysis methods of computer vision, acoustic signal processing, and artificial intelligence large models have gradually emerged, providing new possibilities for improving the accuracy of motion detection and the level of intelligent instruction.

[0003] Currently, in the specific context of badminton training, most existing technologies rely on visual data for motion detection and evaluation. This typically involves using high-speed cameras to capture the interaction between the racket and the shuttlecock, followed by image recognition algorithms to extract motion features. While visual technology can capture motion trajectories and contact timing, this method has significant drawbacks: firstly, visual acquisition is easily affected by external conditions such as changes in lighting, occlusion, and shooting angle, leading to unstable detection results; secondly, visual methods struggle to directly quantify the force intensity and swing rationality at the moment of impact, and training feedback often lacks a comprehensive depiction of motion quality, thus limiting the effectiveness of precise guidance for athletes' movements.

[0004] Under the aforementioned technology relying on a single visual modality, the system may misjudge or miss shots when ambient light is insufficient or the athlete's position deviates from the camera's angle. Even if the timing of the shot is identified, it cannot accurately distinguish whether the force exerted during the movement is appropriate. The resulting adverse effects include: even with frequent practice, athletes cannot obtain effective feedback from the system regarding swing force and shot selection, easily developing incorrect movement habits over long-term training, leading to insufficient shot speed or inappropriate shot placement in actual matches, and in severe cases, even the risk of injury. These shortcomings indicate that existing technology urgently needs to incorporate additional modal information, especially acoustic features, to compensate for the limitations of single visual feedback and improve the intelligence and scientific rigor of movement instruction. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an intelligent teaching system and method for sports movements based on a multimodal large model, which solves the problems mentioned in the background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solution, comprising the following steps: S1. By setting up directional microphones and visual acquisition devices in the badminton training court, the acoustic signal data of the athlete's hitting the ball and the visual image data of the hitting interaction are collected respectively, and the acoustic signal data is preprocessed to obtain the hitting energy data. S2. Calculate the acoustic energy ratio index Rste based on the ball energy data, and determine the ball force by comparing the acoustic energy ratio index Rste with the preset energy threshold E. Trigger the cross-modal timing correction step based on the ball force determination result. S3. In the cross-modal timing correction step, the ball hitting timing is determined. The ball hitting timing is determined by extracting the audio impact time Tah from the acoustic signal data and the visual contact time Tvis from the visual image data, and combining the acoustic energy ratio index Rste to calculate and output a comprehensive judgment index Dcomp. S4. Corrective feedback is performed based on the output of the comprehensive judgment index Dcomp. The corrective feedback generates multi-channel teaching feedback information and simultaneously executes dynamic optimization and learning mechanisms.

[0007] Preferably, S1 includes S11; S11. Place a directional microphone on each side of the training field at a symmetrical distance from the net, with the pickup axis of the directional microphone facing the hitting area. At the same time, place an auxiliary microphone on the top of the field. The raw acoustic signal is acquired using a directional microphone and then bandpass filtered. The filtering frequency band is set between 1000Hz and 8000Hz. The raw acoustic signal after bandpass filtering is then divided into frames according to a preset frame length and frame shift to obtain framed signals. Each frame is 20ms long and the interval between adjacent frames is 10ms. A Hanning window function is then superimposed on each frame signal. Based on the framed signal, the short-time energy of each frame is calculated. The method for calculating the short-time energy is as follows: the values ​​of all sampling points in each framed signal are squared one by one and then accumulated, and the weighted result of the Hanning window is superimposed to obtain the energy value corresponding to the frame. In a short-time energy sequence, when an energy mutation that is significantly more than three times the standard deviation of the background mean is detected, the time period corresponding to the energy mutation is defined as the window of the impact sound. The impact window energy Ehit is extracted from the impact sound window. At the same time, in a stable low-energy segment of several consecutive frames before the impact, the average value is extracted and calculated as the background noise energy Ebase. The energy of the hitting window (Ehit) and the background noise energy (Ebase) are then combined to obtain the hitting energy data.

[0008] Preferably, S1 further includes S12; S12. High-speed cameras are set up on the front and sides of the training ground. The frame rate of the high-speed cameras is not less than 200 frames per second and the resolution is not less than 1280×720. Each frame of the image is timestamped in milliseconds during the acquisition process. During the image acquisition process, a target detection algorithm is used to detect and track the racket and shuttlecock in real time, generating an image sequence containing the racket bounding box and the shuttlecock bounding box, and extracting the relative spatial position parameters between the racket and the shuttlecock based on the image sequence. When the distance between the nearest edge of the racket bounding box and the center of the circumcircle of the badminton bounding box is less than five pixels, and the rate of change of the diameter of the badminton bounding box exceeds 15% within three consecutive frames, a contact event is determined to have occurred, and the image frame corresponding to that moment is automatically marked as a contact keyframe. Within several frames before and after the contact keyframe, inter-frame optical flow analysis technology is used to extract the abrupt change points of the shuttlecock velocity vector, and edge recognition technology is combined to detect the contact overlap area between the racket surface and the shuttlecock outline. The abrupt change point of the shuttlecock's velocity vector is compared with the time point when the contact overlap area of ​​the shuttlecock's outline appears. The consistent moment is taken as the accurate time point when the racket hits the shuttlecock, and the accurate time point when the racket hits the shuttlecock is defined as the visual contact moment Tvis.

[0009] Preferably, S2 includes S21; S21. Construct an acoustic energy ratio model based on the ball impact energy data. Take the ball window energy Ehit and background noise energy Ebase from the real-time acquired ball impact energy as inputs into the acoustic energy ratio model. Use the ratio of the ball window energy Ehit to the background noise energy Ebase as the acoustic energy ratio index Rste to quantitatively analyze the acoustic energy and background energy levels at the moment of impact. The acoustic energy ratio index Rste is calculated and output through the following acoustic energy ratio model: Rate = Ehit / Ebase.

[0010] Preferably, S2 further includes S22; S22. In the initial stage, based on the distribution of acoustic energy ratios of successful and unsuccessful hits in historical training samples, a value that can simultaneously ensure that the false alarm rate and the false alarm rate are within an acceptable range is selected as the initial energy threshold E using statistical methods. The impact force is determined based on the real-time corrected capability threshold E and the acoustic energy ratio index Rste, and a cross-modal timing correction step is triggered based on the impact force determination result. The specific judgment content is as follows: When the acoustic energy ratio index Rste ≥ the ability threshold E, it means that the hitting force meets the requirements, and the energy is judged to be up to standard, and the action instruction is not performed. When the acoustic energy ratio index Rste < the capability threshold E, it indicates that the hitting force does not meet the requirements, and the cross-modal timing correction step is triggered.

[0011] Preferably, S3 includes S31; S31. After triggering the cross-modal timing correction step, the ball hitting timing determination is performed. The ball hitting timing determination is achieved by performing frame-by-frame energy analysis on the acoustic signal. When a peak value exceeding three times the standard deviation of the background mean is detected in the energy curve, the time point corresponding to the peak value exceeding three times the standard deviation of the background mean is defined as the audio impact time Tah. The output comprehensive judgment index Dcomp is based on the audio impact time Tah, the visual contact time Tvis, and the acoustic energy ratio index Rste.

[0012] Preferably, S3 further includes S32; S32. Based on the output of the comprehensive judgment index Dcomp, determine the timing of the shot, and trigger corrective feedback based on the judgment result. The specific judgment content is as follows: When the comprehensive judgment index Dcomp=1, the timing of the shot is judged as meeting the standard; When the comprehensive judgment index Dcomp=0, the timing of the shot is judged as unqualified.

[0013] Preferably, S4 includes S41; S41. Based on the result of the ball hitting timing judgment, trigger correction feedback. The correction feedback generates multi-channel teaching feedback information to assist in the teaching of the action. The multi-channel teaching feedback information includes a voice channel, a visual channel, and a beat channel. When the comprehensive judgment index Dcomp=1, the first prompt voice is output through the voice channel, and the ball trajectory comparison curve is superimposed on the display terminal of the visual channel in real time. When the comprehensive judgment index Dcomp=0, the second and third prompt voices are output through the voice channel, and the deviation area between the ideal motion trajectory and the actual trajectory is superimposed on the display terminal of the visual channel. Then, the beat sound signal of the time difference between the audio impact time Tah and the visual contact time Tvis is output through the speaker through the beat channel. Among them, if the acoustic energy ratio index Rste < the capability threshold E and the audio impact time Tah > the visual contact time Tvis, the second prompt voice is triggered. When the acoustic energy ratio index Rste is less than the capability threshold E and the audio impact time Tah is less than the visual contact time Tvis, the third prompt voice is triggered.

[0014] Preferably, S4 further includes S42; S42. A dynamic optimization and learning mechanism is executed simultaneously with the correction feedback. The dynamic optimization uses every fifty shots as a statistical period to dynamically optimize the energy threshold E and the time difference threshold T. The specific optimization content is as follows: When the shot pass rate is >80%, the energy threshold E will be automatically increased by 5%-10%, and the time difference threshold T will be decreased by 5%-10%. When the ball pass rate is less than 50%, the energy threshold E will be automatically reduced by 5%-10%, and the time difference threshold T will be increased by 5%-10%. Based on this, a learning mechanism is implemented, which constructs an intelligent teaching model by considering the joint distribution of the athlete's hit force judgment and hit timing judgment within a statistical period, and periodically updates the personalized intelligent teaching model based on the hit force judgment and hit timing judgment results.

[0015] A sports movement intelligent teaching system based on a multimodal large model includes a ball-hitting perception module, a ball-hitting force determination module, a ball-hitting timing determination module, and a comprehensive correction module; The ball-hitting sensing module collects acoustic signal data of the athlete's ball-hitting action and visual image data of the ball-hitting interaction by setting up directional microphones and visual acquisition devices in the badminton training court, and preprocesses the acoustic signal data to obtain ball-hitting energy data. The ball impact force determination module calculates the acoustic energy ratio index Rste based on the ball impact energy data, and determines the ball impact force by comparing the acoustic energy ratio index Rste with the preset energy threshold E. Based on the ball impact force determination result, a cross-modal timing correction step is triggered. The ball-hitting timing determination module performs ball-hitting timing determination in the cross-modal timing correction step. The ball-hitting timing determination is achieved by extracting the audio impact time Tah from the acoustic signal data and the visual contact time Tvis from the visual image data, and combining it with the acoustic energy ratio index Rste to calculate and output a comprehensive determination index Dcomp. The comprehensive correction module provides corrective feedback based on the output of the comprehensive judgment index Dcomp. This corrective feedback generates multi-channel teaching feedback information and simultaneously executes a dynamic optimization and learning mechanism.

[0016] This invention provides an intelligent teaching system and method for sports movements based on a multimodal large model. It has the following beneficial effects: (1) This method, by deploying directional microphones and high-speed cameras on the badminton training court, and combining bandpass filtering, frame processing, and short-time energy calculation of acoustic signals with target detection and optical flow analysis of visual images, can obtain the hitting window energy Ehit and background noise energy Ebase, and further construct the acoustic energy ratio index Rste. This method ensures high signal-to-noise ratio extraction of the hitting acoustic signal in complex environments and can accurately locate the visual contact moment Tvis in the visual modality. Compared with existing teaching methods that rely solely on visual detection, this method can still maintain high robustness under conditions such as changes in illumination and noise interference on the court, significantly improving the accuracy of hitting action data acquisition and analysis.

[0017] (2) This method combines the acoustic energy ratio index Rste with the audio impact time Tah and the visual contact time Tvis in steps S2 and S3, and outputs a comprehensive judgment index Dcomp. This method can simultaneously complete a multi-dimensional evaluation of the hitting force and the hitting timing. Compared with existing techniques that only determine whether the action hits, this method can further trigger cross-modal correction when "insufficient force" is determined, and combine the difference between the acoustic peak and the visual contact time to clearly distinguish between the specific problems of "insufficient force" and "inappropriate timing". On this basis, the system outputs feedback through three channels: voice prompts, visual trajectory superposition prompts, and beat prompts, so that athletes can intuitively perceive the deviation points of the action and make immediate corrections, thereby shortening the training correction cycle.

[0018] (3) By introducing a dynamic optimization and learning mechanism in steps S4 and S42, this method can dynamically adjust the energy threshold E and the time difference threshold T based on the pass rate of the ball hits in a statistical cycle of every fifty hits: when the pass rate is greater than 80%, the energy threshold E is increased by 5%-10% and the time difference threshold T is decreased by 5%-10% to increase the training difficulty; when the pass rate is less than 50%, the energy threshold E is decreased by 5%-10% and the time difference threshold T is increased by 5%-10% to avoid overtraining. At the same time, the system periodically updates the personalized intelligent teaching model based on the joint distribution of the athlete's ball hitting force judgment and ball hitting timing judgment within the statistical cycle, so that the multi-channel feedback results can be matched with the individual characteristics of different athletes, thereby realizing differentiated teaching and personalized training, and ensuring the stability and scientific nature of the training effect. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the process of a sports movement intelligent teaching method based on a multimodal large model according to the present invention.

[0020] Figure 2 This is a schematic diagram illustrating the steps of a sports movement intelligent teaching system based on a multimodal large model according to the present invention.

[0021] Figure 3 This is a layout diagram of the training ground. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1: Please refer to Figure 1 This invention provides an intelligent teaching method for sports movements based on a multimodal large model. To achieve the above objectives, this invention is implemented through the following technical solution, including the following steps: S1. By setting up directional microphones and visual acquisition devices in the badminton training court, the acoustic signal data of the athlete's hitting and the visual image data of the hitting interaction are collected respectively, and the acoustic signal data is preprocessed to obtain the hitting energy data. S2. Calculate the acoustic energy ratio index Rste based on the ball energy data, and determine the ball force by comparing the acoustic energy ratio index Rste with the preset energy threshold E. Trigger the cross-modal timing correction step based on the ball force determination result. S3. In the cross-modal timing correction step, the ball hitting timing is determined. The ball hitting timing is determined by extracting the audio impact time Tah from the acoustic signal data and the visual contact time Tvis from the visual image data, and combining the acoustic energy ratio index Rste to calculate and output the comprehensive judgment index Dcomp. S4. Based on the output of the comprehensive judgment index Dcomp, corrective feedback is provided. The corrective feedback generates multi-channel teaching feedback information and simultaneously executes dynamic optimization and learning mechanisms.

[0024] In this embodiment, method S1 ensures the synchronous acquisition of the acoustic signal and visual image of the hit by deploying directional microphones and visual acquisition devices in the badminton training court. The directional microphones, with their pickup axis facing the hitting area, significantly improve the signal-to-noise ratio of the hit sound to the background noise. If only environmental microphones are used, background conversations or echoes from the court often drown out the hit signal, leading to inaccurate extraction of hit energy data. Therefore, this deployment method effectively avoids noise interference and achieves clear separation of hit energy Ehit and background energy Ebase. Step S2 introduces the acoustic energy ratio index Rste, which essentially offsets the influence of the overall environmental energy level through the ratio. Even with high background noise, it can accurately reflect the true force intensity at the moment of impact through normalization, thus making the hit force determination more robust. Step S3 establishes a cross-modal comprehensive judgment index Dcomp by comparing the difference between the audio impact moment Tah and the visual contact moment Tvis. Its physical meaning lies in truly depicting the time alignment relationship between "sound generation" and "racket contact". If the sound of the impact is significantly delayed compared to the visual contact, it often indicates a swing that is too late, and vice versa. A single modality is insufficient to reveal this issue; therefore, cross-modal correction can accurately pinpoint movement deviations. Step S4 generates multi-channel feedback in three categories: voice, vision, and beat, and combines this with a dynamic optimization mechanism. This allows the system to provide immediate alerts for "too early" or "too late," and automatically reduce the training pace after multiple consecutive misjudgments, preventing athletes from repeatedly practicing at the wrong rhythm. Thus, this method not only improves the accuracy and anti-interference capabilities of impact detection but also achieves personalized and adaptive adjustments in the feedback process, ultimately ensuring that the training process is both scientific and efficient.

[0025] Example 2: Please refer to Figure 1 and Figure 3 Specifically: S1 includes S11; S11. A directional microphone is placed symmetrically on the left and right sides of the training field at a distance from the net, with the pickup axis of the directional microphone facing the hitting area to enhance the signal-to-noise ratio of the hitting sound signal and the ambient noise. At the same time, an auxiliary microphone is placed at the top of the field to collect the overall ambient noise, thereby achieving multi-source fusion of acoustic signals and noise suppression. The raw acoustic signal is acquired using a directional microphone and then bandpass filtered. The filtering frequency band is set between 1000Hz and 8000Hz to remove low-frequency human voice interference and high-frequency electromagnetic noise, in addition to the sound of hitting the ball. The raw acoustic signal after bandpass filtering is then divided into frames according to a preset frame length and frame shift to obtain framed signals. Each frame is 20ms long and the interval between adjacent frames is 10ms. A Hanning window function is superimposed on each frame signal to reduce spectral leakage. Based on the framed signal, the short-time energy of each frame is calculated. The method for calculating the short-time energy is as follows: the values ​​of all sampling points in each frame signal are squared one by one and then accumulated, and the weighted result of the Hanning window is superimposed to obtain the energy value corresponding to the frame. In a short-time energy sequence, when an energy mutation significantly exceeding three standard deviations of the background mean is detected, the time period corresponding to the energy mutation is defined as the window of the impact sound. The impact window energy Ehit is extracted from the impact sound window. At the same time, in a stable low-energy segment of several consecutive frames before the impact, the average value is extracted and calculated as the background noise energy Ebase, which is used for the subsequent calculation and evaluation of the acoustic energy ratio index. The energy of the hitting window (Ehit) and the background noise energy (Ebase) are then combined to obtain the hitting energy data.

[0026] S1 also includes S12; S12. High-speed cameras are set up on the front and sides of the training ground. The frame rate of the high-speed cameras is not less than 200 frames per second and the resolution is not less than 1280×720. During the acquisition process, each frame image is timestamped at the millisecond level to ensure that the image data and acoustic signal data are synchronized in the time dimension. During the image acquisition process, the target detection algorithm is used to detect and track the racket and shuttlecock in real time, generating an image sequence containing the racket bounding box and the shuttlecock bounding box, and extracting the relative spatial position parameters between the racket and the shuttlecock based on the image sequence. When the distance between the nearest edge of the racket bounding box and the center of the circumcircle of the badminton bounding box is less than five pixels, and the rate of change of the diameter of the badminton bounding box exceeds 15% within three consecutive frames, a contact event is determined to have occurred, and the image frame corresponding to that moment is automatically marked as a contact keyframe. Within several frames before and after the contact keyframe, inter-frame optical flow analysis is used to extract the abrupt change points of the shuttlecock velocity vector, and edge recognition technology is combined to detect the contact overlap area between the racket surface and the shuttlecock outline. The abrupt change point of the shuttlecock velocity vector is compared with the time point when the contact overlap area of ​​the shuttlecock contour appears. The consistent time is taken as the accurate time point when the racket hits the shuttlecock. The accurate time point when the racket hits the shuttlecock is defined as the visual contact moment Tvis, which is used in the subsequent cross-modal timing correction step.

[0027] In this embodiment, in S11, directional microphones are placed symmetrically on either side of the net, with the pickup axis pointing towards the hitting area. This physically improves the signal-to-noise ratio between the hitting sound and ambient noise. If the microphones are facing the spectator area or placed outside the court, the sound of the athlete hitting the ball is easily masked by human voices and the sound of shoes rubbing, making the short-term energy change point less obvious and leading to misjudgment of Ehit. By symmetrically placing the microphones and using a top auxiliary microphone, the system can compare the energy differences of different sound sources and achieve noise suppression after multi-source fusion, ensuring the stability and separability of the hitting acoustic signal. In the signal processing stage of S11, the bandpass filter frequency is set to 1kHz–8kHz to remove low-frequency human voices and high-frequency electromagnetic interference. The sound pressure energy at the moment of impact is mainly concentrated in the mid-to-high frequency range. If this frequency band is not set, the low-frequency "boom" of human voices will raise Ebase and directly lower Rste, causing the system to misjudge "insufficient force". Through this frequency band filtering, it can be ensured that the energy extraction stage is more consistent with the true physical meaning: Ehit represents the instantaneous intensity of the impact sound, and Ebase represents the stable noise level. In S12, the high-speed camera's frame rate is set to no less than 200fps because a standard 30fps camera, in actual training, causes the ball contact process to span multiple frames, making it impossible to capture the precise contact moment and easily leading to Tvis delay bias. For example, if the ball hits between two frames, a standard camera can only estimate within ±15ms, while 200fps can compress the error to ±2.5ms, thus ensuring more accurate alignment between Tah and Tvis and avoiding time difference amplification during cross-modal correction. Through the above implementation, the system achieves effective separation of the ball's impact sound from environmental noise, accurate energy capture at the moment of impact, and millisecond-level alignment of acoustic and visual signals at the physical level. This design not only avoids misjudgments caused by excessive noise or image blurring in traditional single-modal systems but also improves the accuracy and robustness of training feedback, providing a reliable data foundation for subsequent cross-modal judgment and dynamic optimization.

[0028] Example 3: Please refer to Figure 1 Specifically: S2 includes S21; S21. Based on the ball impact energy data, construct an acoustic energy ratio model. Take the ball window energy Ehit and background noise energy Ebase from the real-time acquired ball impact energy as inputs into the acoustic energy ratio model. Use the ratio of ball window energy Ehit to background noise energy Ebase as the acoustic energy ratio index Rste to quantitatively analyze the acoustic energy and background energy levels at the moment of impact. The acoustic energy ratio index Rste is calculated and output through the following acoustic energy ratio model: Rate = Ehit / Ebase. Origin and Basic Principles of the Formula: The measurement of acoustic energy is a classic formula in the field of signal processing, derived from physics and the principles of digital signal processing (DSP). In acoustic signal processing, short-time energy is an important indicator of sound intensity. The classic formula is: Short-time energy = sum of the squares of the amplitudes of each sampling point. This is derived from the principle in physics that "energy is proportional to the square of amplitude". Based on this, this formula defines the short-time energy Ehit of the impact window and the background noise energy Ebase, and constructs the acoustic energy ratio index Rste. The calculation method for short-time energy itself follows the classical formula (based on physics); the improvement of this formula is that it distinguishes the hitting window from the static background window, avoiding misjudgment caused by environmental noise fluctuations; and it uses the acoustic energy ratio index Rste instead of a single energy value, which improves robustness to different venues and different noise environments. Dimensionality analysis: Ehit, Ebase: The unit is "sum of squares of amplitude", which can correspond to the relative quantity of sound pressure square (Pa²); Rste=Ehit / Ebase: Since it is a ratio, it is dimensionless in the end, which is consistent with physical common sense; Specific examples: Background noise energy Ebase: 10 frames were selected from the still segment, and the energy of each frame was [102, 98, 95, 100, 97, 96, 101, 99, 94, 98]; the average value = (102+98+95+100+97+96+101+99+94+98) / 10 = 98; therefore, Ebase = 98; Impact window energy Ehit: The short-time energy calculated within the window where the impact sound is detected is 560; therefore, Ehit = 560. Acoustic energy ratio Rste: Rste=Ehit / Ebase=560 / 98≈5.71.

[0029] S2 also includes S22; S22. In the initial stage, based on the distribution of acoustic energy ratios of successful and unsuccessful hits in historical training samples, a value that can simultaneously ensure that the false alarm rate and the false alarm rate are within an acceptable range is selected as the initial energy threshold E using statistical methods. The impact force is determined based on the real-time corrected capability threshold E and the acoustic energy ratio index Rste, and a cross-modal timing correction step is triggered based on the impact force determination result. The specific judgment content is as follows: When the acoustic energy ratio index Rste ≥ the ability threshold E, it means that the hitting force meets the requirements, and the energy is judged to be up to standard, and the action instruction is not performed. When the acoustic energy ratio index Rste < the capability threshold E, it indicates that the hitting force does not meet the requirements, and the cross-modal timing correction step is triggered.

[0030] In this embodiment, in S21, the method ratios the energy of the hitting window, Ehit, to the background noise energy, Ebase, to obtain the acoustic energy ratio index, Rste. Its core purpose is to offset random fluctuations in environmental noise. If only Ehit is used as the criterion, sudden noises in the training area (such as coughing from spectators or the sound of shoes rubbing) can cause these noise peaks to be misidentified as hitting energy, leading to misjudgments. By using a ratio, Rste can reflect the increase in hitting impact energy relative to background noise in real time, possessing a physical "signal intensity contrast," thus significantly improving the stability and robustness of the judgment. In S22, a statistical method based on the distribution of historical training samples is introduced to set an initial energy threshold E. This avoids one-size-fits-all misjudgments due to individual or venue differences. For example, a beginner's Ehit is generally low when hitting the ball. If the energy threshold of a high-level athlete is directly applied, the system will frequently judge it as "unqualified," resulting in excessive negative feedback and disrupting the training experience. Setting the threshold through statistical distribution ensures that the false alarm rate and false negative rate remain within a reasonable range, achieving personalized and scientific thresholding. This implementation method, in a physical sense, ensures that the hit judgment is not merely a measure of "absolute energy," but rather a measure of "relative energy increase." In terms of training scenarios, it guarantees that athletes of different levels and in different environments can receive fair and accurate force assessments. The resulting benefits include: significantly reducing the risk of misjudgment due to environmental noise interference, improving the system's adaptability and the accuracy of training feedback, and enabling subsequent cross-modal timing corrections to be triggered more reliably, avoiding invalid or redundant correction steps.

[0031] Example 4: Please refer to Figure 1 Specifically: S3 includes S31; S31. After triggering the cross-modal timing correction step, the ball hitting timing determination is performed. The ball hitting timing determination is achieved by performing frame-by-frame energy analysis on the acoustic signal. When a peak value exceeding three times the standard deviation of the background mean is detected in the energy curve, the time point corresponding to the peak value exceeding three times the standard deviation of the background mean is defined as the audio impact time Tah. Based on the audio impact time Tah, the visual contact time Tvis, and the acoustic energy ratio index Rste, the output comprehensive judgment index Dcomp is calculated. The specific output formula of the output comprehensive judgment index Dcomp is: Dcomp=1[Rste≥E]×1[|Tah-Tvis|<T]; where T represents the time difference threshold, which is set in the range of 20ms-30ms. The formulas are derived from physical meanings: signal processing and detection logic. In the acoustic part, the acoustic energy ratio index Rste originates from the classic short-time energy formula, based on physics and digital signal processing (DSP) principles, and is used to measure the ratio of impact sound energy to background noise energy. In the visual part, the visual contact time Tvis originates from target detection and motion analysis algorithms, and is a commonly used detection method in computer vision. Time difference analysis: the difference between Tah and Tvis |Tah-Tvis| belongs to the classic time alignment algorithm, used to detect the synchronization of multimodal signals. Improvements to this formula: It integrates two independent modalities (acoustic energy and visual contact) into a binary judgment formula; it constructs a comprehensive judgment index Dcomp through logical multiplication to achieve an overall evaluation of the hitting action, rather than a single index judgment. In the formula, 1[condition] represents an exponential function; it means that when the condition is true, it equals 1, and when the condition is false, it equals 0. 1[Rste≥E] means the hitting power meets the standard, and 1[|Tah-Tvis|<T] means the hitting timing meets the standard; The reason why the cross-modal timing correction step is triggered when the hitting power does not meet the requirements is that if the power is sufficient, it means that the athlete's basic strength of movement is not the problem. There is no need to immediately enter the complex cross-modal analysis process. It is sufficient to directly give positive feedback that "the power meets the requirements". This can reduce the computational burden on the system and avoid too much meaningless analysis. When the hitting power is insufficient, simply telling the athlete "the power is not enough" is not helpful. The root cause of insufficient power is often not a problem with muscle strength, but an error in the timing of the movement (such as swinging the racket too early or too late, resulting in not hitting the ball at the optimal point of force). Therefore, when Rste < E, cross-modal correction is triggered. Only by combining the acoustic peak Tah with the visual contact time Tvis can the problem be located as "insufficient power" or "inappropriate timing".

[0032] S3 also includes S32; S32. Based on the output of the comprehensive judgment index Dcomp, determine the timing of the shot, and trigger corrective feedback based on the judgment result. The specific judgment content is as follows: When the comprehensive judgment index Dcomp=1, the timing of the shot is judged as meeting the standard; When the comprehensive judgment index Dcomp=0, the timing of the shot is judged as unqualified.

[0033] In this embodiment, during the implementation of S31, the method frames the acoustic signal and uses "three standard deviations" as the peak detection threshold to locate the audio impact moment Tah. The purpose is to avoid false triggering by random noise in the environment. For example, the common sounds of shoe sole friction or audience clapping in a badminton hall can cause instantaneous energy changes, but these noises are often insufficient to exceed three standard deviations of the background mean. This criterion allows for accurate identification of the actual hitting sound, ensuring that the physical meaning of Tah is clear: "the impact sound of the racket hitting the ball." In cross-modal correction, Tah is time-aligned with the visual contact moment Tvis. The core objective is to verify whether the timing of the force exertion matches the actual contact moment of the ball. For example, if Tah is significantly later than Tvis, it indicates that the athlete's force exertion is delayed. Although the force is sufficient, the hitting point has deviated from the optimal position, resulting in a decrease in hitting quality. In this case, Dcomp outputs 0, and the system can directly attribute the problem to "inappropriate timing," preventing athletes from being misled by simply reporting "insufficient force." In S32, a concise and efficient feedback judgment logic is implemented through the binary output of Dcomp (1 = qualified, 0 = unqualified). The advantages of this logic lie in its real-time performance and interpretability: during training, the system can clearly tell the athlete whether the movement is "passed" or "needs correction" without complex calculations. Simultaneously, because the formula logic integrates both force and timing dimensions, it avoids the one-sidedness of single-index judgment, making the teaching feedback more targeted. The beneficial effects of this implementation are: on the one hand, the dual-modal coupling of acoustic and visual modes ensures high accuracy and robustness in hitting timing detection; on the other hand, the concise Dcomp output makes the feedback both fast and intuitive, effectively improving the athlete's learning efficiency in training and reducing the cost of repetitive training due to incorrect judgments.

[0034] Example 5: Please refer to Figure 1 Specifically: S4 includes S41; S41. Based on the timing of the shot, trigger corrective feedback. This feedback generates multi-channel teaching feedback information to assist in instruction. The multi-channel teaching feedback information includes a voice channel, a visual channel, and a beat channel. The voice channel outputs positive prompts to alert the athlete to deviations in their movements. The visual channel overlays the afterimages of the standard swing trajectory and the athlete's actual trajectory on the display terminal, allowing the athlete to visually identify the location of the deviation. The beat channel plays a beat sound signal proportional to the time difference through a speaker to guide the athlete to gradually correct their shot timing. When the comprehensive judgment index Dcomp=1, the first prompt voice "the shot is powerful and the timing is right" is output through the voice channel, and the shot trajectory comparison curve is superimposed on the display terminal of the visual channel in real time to strengthen the athlete's memory of the correct action; When the comprehensive judgment index Dcomp=0, the second and third prompt voices are output through the voice channel, and the deviation area between the ideal motion trajectory and the actual trajectory is superimposed on the display terminal of the visual channel. Then, the beat sound signal of the time difference between the audio impact time Tah and the visual contact time Tvis is output through the speaker through the beat channel. If the acoustic energy ratio index Rste < the capability threshold E and the audio impact time Tah > the visual contact time Tvis, the second prompt voice is triggered, prompting "Swing too late, please swing earlier"; When the acoustic energy ratio index Rste is less than the capability threshold E and the audio impact time Tah is less than the visual contact time Tvis, a third voice prompt is triggered, indicating "Swing too early, please postpone".

[0035] S4 also includes S42; S42. While providing corrective feedback, a dynamic optimization and learning mechanism is implemented. The dynamic optimization uses every fifty shots as a statistical period to dynamically optimize the energy threshold E and the time difference threshold T. The specific optimization content is as follows: When the ball pass rate is >80%, the energy threshold E is automatically increased by 5%-10%, and the time difference threshold T is decreased by 5%-10% to tighten the judgment criteria. When the ball pass rate is less than 50%, the energy threshold E will be automatically reduced by 5%-10%, and the time difference threshold T will be increased by 5%-10% to avoid the negative effects of overtraining. Based on this, a learning mechanism is implemented. This mechanism constructs an intelligent teaching model by analyzing the joint distribution of the athlete's hit force and hit timing judgments within a statistical period. The personalized intelligent teaching model is then periodically updated based on the hit force and hit timing judgment results, so that multi-channel teaching feedback information can adapt to the movement characteristics of different athletes and provide differentiated guidance.

[0036] In this embodiment, the core of the multi-channel feedback design in S41 is to reduce the ambiguity that a single prompt may cause. For example, when relying solely on the voice prompt "movement deviation," athletes may find it difficult to understand the specific problem instantly. However, by superimposing the standard trajectory and comparing the afterimage, athletes can intuitively see the deviation point. Simultaneously, the beat channel is designed to convert the abstract time difference |Tah-Tvis| into an auditory rhythm, making it easier for athletes to correct their movement timing through rhythm during actual training. The physical meaning of this setting is to transform the judgment results of multimodal data into multi-sensory feedback, minimizing information delay and comprehension bias. For example, when the audio impact moment Tah lags behind the visual contact moment Tvis, a simple prompt "too late" is often difficult for beginners to grasp, while the early sound of the beat can instantly create a conditioned reflex, thereby quickly improving the timing of the shot. In S42, the purpose of introducing a dynamic optimization mechanism is to prevent training standards from being "too lenient" or "too strict." If the energy threshold E remains unchanged for a long period, the previously easily attainable standard will lead to a lack of challenge in training and stagnation as the athlete's skill level improves. Conversely, if the threshold is set too strictly, it may cause frustration due to repeated failures. By statistically analyzing 50 shots and dynamically fine-tuning the ability threshold E and the time difference threshold T, the system can adjust the difficulty of judgment based on actual performance. For example, if the pass rate exceeds 80%, the judgment standard is tightened, which is equivalent to gradually narrowing the allowable error range of the movement, helping athletes enter a higher level of refined training. Conversely, when the pass rate is below 50%, the threshold is appropriately relaxed to ensure positive feedback during training and avoid overload. This strategy is physically equivalent to "progressive load training," using a dynamic adaptation mechanism to ensure that the feedback always matches the athlete's ability boundaries. The beneficial effect of this implementation method is that the system can not only correct movements in real time but also self-adjust the standard as the athlete's skill level improves, achieving "personalized progressive teaching." The ultimate improvement is reflected in: more intuitive training feedback, more efficient movement adjustment, more scientific training rhythm, and athletes maintaining motivation and progress in a continuous dynamic balance.

[0037] Example 6: Please refer to Figure 1 and Figure 2 A sports movement intelligent teaching system based on a multimodal large model includes a ball-hitting perception module, a ball-hitting force determination module, a ball-hitting timing determination module, and a comprehensive correction module. The hit perception module collects acoustic signal data of the athlete's hit and visual image data of the hit interaction by setting up directional microphones and visual acquisition devices in the badminton training court, and preprocesses the acoustic signal data to obtain the hit energy data. The impact force determination module calculates the acoustic energy ratio index Rste based on the impact energy data, and determines the impact force by comparing the acoustic energy ratio index Rste with the preset energy threshold E. Based on the impact force determination result, the cross-modal timing correction step is triggered. The ball-hitting timing determination module performs ball-hitting timing determination in the cross-modal timing correction step. The ball-hitting timing determination is achieved by extracting the audio impact time Tah from the acoustic signal data and the visual contact time Tvis from the visual image data, and combining it with the acoustic energy ratio index Rste to calculate and output a comprehensive determination index Dcomp. The comprehensive correction module provides corrective feedback based on the output of the comprehensive judgment index Dcomp. This corrective feedback generates multi-channel teaching feedback information and simultaneously executes dynamic optimization and learning mechanisms.

[0038] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention.

Claims

1. A smart teaching method for sports movements based on a multimodal large model, characterized in that: Includes the following steps: S1. By setting up directional microphones and visual acquisition devices in the badminton training court, the acoustic signal data of the athlete's hitting the ball and the visual image data of the hitting interaction are collected respectively, and the acoustic signal data is preprocessed to obtain the hitting energy data. S2. Calculate the acoustic energy ratio index Rste based on the ball energy data, and determine the ball force by comparing the acoustic energy ratio index Rste with the preset energy threshold E. Trigger the cross-modal timing correction step based on the ball force determination result. S3. In the cross-modal timing correction step, the ball hitting timing is determined. The ball hitting timing is determined by extracting the audio impact time Tah from the acoustic signal data and the visual contact time Tvis from the visual image data, and combining the acoustic energy ratio index Rste to calculate and output a comprehensive judgment index Dcomp. S4. Corrective feedback is performed based on the output of the comprehensive judgment index Dcomp. The corrective feedback generates multi-channel teaching feedback information and simultaneously executes dynamic optimization and learning mechanisms.

2. The intelligent teaching method for sports movements based on a multimodal large model according to claim 1, characterized in that: S1 includes S11; S11. Place a directional microphone on each side of the training field at a symmetrical distance from the net, with the pickup axis of the directional microphone facing the hitting area. At the same time, place an auxiliary microphone on the top of the field. The raw acoustic signal is acquired using a directional microphone and then bandpass filtered. The filtering frequency band is set between 1000Hz and 8000Hz. The raw acoustic signal after bandpass filtering is then divided into frames according to a preset frame length and frame shift to obtain framed signals. Each frame is 20ms long and the interval between adjacent frames is 10ms. A Hanning window function is then superimposed on each frame signal. Based on the framed signal, the short-time energy of each frame is calculated. The method for calculating the short-time energy is as follows: the values ​​of all sampling points in each framed signal are squared one by one and then accumulated, and the weighted result of the Hanning window is superimposed to obtain the energy value corresponding to the frame. In a short-time energy sequence, when an energy mutation that is significantly more than three times the standard deviation of the background mean is detected, the time period corresponding to the energy mutation is defined as the window of the impact sound. The impact window energy Ehit is extracted from the impact sound window. At the same time, in a stable low-energy segment of several consecutive frames before the impact, the average value is extracted and calculated as the background noise energy Ebase. The energy of the hitting window (Ehit) and the background noise energy (Ebase) are then combined to obtain the hitting energy data.

3. The intelligent teaching method for sports movements based on a multimodal large model according to claim 2, characterized in that: S1 further includes S12; S12. High-speed cameras are set up on the front and sides of the training ground. The frame rate of the high-speed cameras is not less than 200 frames per second and the resolution is not less than 1280×720. Each frame of the image is timestamped in milliseconds during the acquisition process. During the image acquisition process, a target detection algorithm is used to detect and track the racket and shuttlecock in real time, generating an image sequence containing the racket bounding box and the shuttlecock bounding box, and extracting the relative spatial position parameters between the racket and the shuttlecock based on the image sequence. When the distance between the nearest edge of the racket bounding box and the center of the circumcircle of the badminton bounding box is less than five pixels, and the rate of change of the diameter of the badminton bounding box exceeds 15% within three consecutive frames, a contact event is determined to have occurred, and the image frame corresponding to that moment is automatically marked as a contact keyframe. Within several frames before and after the contact keyframe, inter-frame optical flow analysis technology is used to extract the abrupt change points of the shuttlecock velocity vector, and edge recognition technology is combined to detect the contact overlap area between the racket surface and the shuttlecock outline. The abrupt change point of the shuttlecock's velocity vector is compared with the time point when the contact overlap area of ​​the shuttlecock's outline appears. The consistent moment is taken as the accurate time point when the racket hits the shuttlecock, and the accurate time point when the racket hits the shuttlecock is defined as the visual contact moment Tvis.

4. The intelligent teaching method for sports movements based on a multimodal large model according to claim 3, characterized in that: S2 includes S21; S21. Construct an acoustic energy ratio model based on the ball impact energy data. Take the ball window energy Ehit and background noise energy Ebase from the real-time acquired ball impact energy as inputs and input them into the acoustic energy ratio model. Use the ratio of the ball window energy Ehit to the background noise energy Ebase as the acoustic energy ratio index Rste to perform quantitative analysis of the acoustic energy and background energy levels at the moment of ball impact.

5. The intelligent teaching method for sports movements based on a multimodal large model according to claim 4, characterized in that: S2 further includes S22; S22. In the initial stage, based on the distribution of acoustic energy ratios of successful and unsuccessful hits in historical training samples, a value that can simultaneously ensure that the false alarm rate and the false alarm rate are within an acceptable range is selected as the initial energy threshold E using statistical methods. The impact force is determined based on the real-time corrected capability threshold E and the acoustic energy ratio index Rste, and a cross-modal timing correction step is triggered based on the impact force determination result. The specific judgment content is as follows: When the acoustic energy ratio index Rste ≥ the ability threshold E, it means that the hitting force meets the requirements, and the energy is judged to be up to standard, and the action instruction is not performed. When the acoustic energy ratio index Rste < the capability threshold E, it indicates that the hitting force does not meet the requirements, and the cross-modal timing correction step is triggered.

6. The intelligent teaching method for sports movements based on a multimodal large model according to claim 5, characterized in that: S3 includes S31; S31. After triggering the cross-modal timing correction step, the ball hitting timing determination is performed. The ball hitting timing determination is achieved by performing frame-by-frame energy analysis on the acoustic signal. When a peak value exceeding three times the standard deviation of the background mean is detected in the energy curve, the time point corresponding to the peak value exceeding three times the standard deviation of the background mean is defined as the audio impact time Tah. The output comprehensive judgment index Dcomp is based on the audio impact time Tah, the visual contact time Tvis, and the acoustic energy ratio index Rste.

7. The intelligent teaching method for sports movements based on a multimodal large model according to claim 6, characterized in that: S3 further includes S32; S32. Based on the output of the comprehensive judgment index Dcomp, determine the timing of the shot, and trigger corrective feedback based on the judgment result. The specific judgment content is as follows: When the comprehensive judgment index Dcomp=1, the timing of the shot is judged as meeting the standard; When the comprehensive judgment index Dcomp=0, the timing of the shot is judged as unqualified.

8. The intelligent teaching method for sports movements based on a multimodal large model according to claim 7, characterized in that: S4 includes S41; S41. Based on the result of the ball hitting timing judgment, trigger correction feedback. The correction feedback generates multi-channel teaching feedback information to assist in the teaching of the action. The multi-channel teaching feedback information includes a voice channel, a visual channel, and a beat channel. When the comprehensive judgment index Dcomp=1, the first prompt voice is output through the voice channel, and the ball trajectory comparison curve is superimposed on the display terminal of the visual channel in real time. When the comprehensive judgment index Dcomp=0, the second and third prompt voices are output through the voice channel, and the deviation area between the ideal motion trajectory and the actual trajectory is superimposed on the display terminal of the visual channel. Then, the beat sound signal of the time difference between the audio impact time Tah and the visual contact time Tvis is output through the speaker through the beat channel. Among them, if the acoustic energy ratio index Rste < the capability threshold E and the audio impact time Tah > the visual contact time Tvis, the second prompt voice is triggered. When the acoustic energy ratio index Rste is less than the capability threshold E and the audio impact time Tah is less than the visual contact time Tvis, the third prompt voice is triggered.

9. The intelligent teaching method for sports movements based on a multimodal large model according to claim 8, characterized in that: S4 also includes S42; S42. A dynamic optimization and learning mechanism is executed simultaneously with the correction feedback. The dynamic optimization uses every fifty shots as a statistical period to dynamically optimize the energy threshold E and the time difference threshold T. The specific optimization content is as follows: When the ball-hitting success rate is >80%, the energy threshold E will be automatically increased by 5%-10%, and the time difference threshold T will be decreased by 5%-10%. When the ball pass rate is less than 50%, the energy threshold E will be automatically reduced by 5%-10%, and the time difference threshold T will be increased by 5%-10%. Based on this, a learning mechanism is implemented, which constructs an intelligent teaching model by considering the joint distribution of the athlete's hit force judgment and hit timing judgment within a statistical period, and periodically updates the personalized intelligent teaching model based on the hit force judgment and hit timing judgment results.

10. A sports movement intelligent teaching system based on a multimodal large model, applied to the sports movement intelligent teaching method based on a multimodal large model as described in any one of claims 1-9, characterized in that: It includes a ball-sensing module, a ball-force determination module, a ball-timing determination module, and a comprehensive correction module; The ball-hitting sensing module collects acoustic signal data of the athlete's ball-hitting action and visual image data of the ball-hitting interaction by setting up directional microphones and visual acquisition devices in the badminton training court, and preprocesses the acoustic signal data to obtain ball-hitting energy data. The ball impact force determination module calculates the acoustic energy ratio index Rste based on the ball impact energy data, and determines the ball impact force by comparing the acoustic energy ratio index Rste with the preset energy threshold E. Based on the ball impact force determination result, a cross-modal timing correction step is triggered. The ball-hitting timing determination module performs ball-hitting timing determination in the cross-modal timing correction step. The ball-hitting timing determination is achieved by extracting the audio impact time Tah from the acoustic signal data and the visual contact time Tvis from the visual image data, and combining it with the acoustic energy ratio index Rste to calculate and output a comprehensive determination index Dcomp. The comprehensive correction module provides corrective feedback based on the output of the comprehensive judgment index Dcomp. This corrective feedback generates multi-channel teaching feedback information and simultaneously executes a dynamic optimization and learning mechanism.

Images