Sports action intelligent teaching system and method based on multi-modal large model

Abstract

The application discloses a sports action intelligent teaching system and method based on a multimodal large model, and relates to the technical field of intelligent teaching.The method is characterized in that a directional microphone and a high-speed camera are arranged in a badminton training ground, and the method combines band-pass filtering, frame processing and short-time energy calculation of acoustic signals, and target detection and optical flow analysis of visual images, so that the method can obtain a hitting window energy Ehit and a background noise energy Ebase, and further construct an acoustic energy ratio index Rste.The method guarantees high signal-to-noise ratio extraction of the hitting acoustic signal in a complex environment, and can accurately locate a visual contact time Tvis in a visual mode.Compared with the existing teaching method which only relies on visual detection, the method still has high robustness under the conditions of illumination change and field noise interference, and significantly improves the accuracy of hitting action data acquisition and analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent teaching, in particular to a sports action intelligent teaching system and method based on a multi-modal large model. BACKGROUND

[0002] With the rapid development of intelligent technology, multi-modal large models have been widely used in the field of sports training and teaching assistance. As a typical high-speed confrontation sport, badminton has very high requirements for the timing, speed and accuracy of the hitting action. Its action teaching and training evaluation have long relied on manual experience and visual observation. However, relying solely on manual or single-modal data often fails to fully capture the quality of the action. In recent years, the fusion analysis method of computer vision, acoustic signal processing and artificial intelligence large model has gradually emerged, providing new possibilities for improving the accuracy of sports action detection and the level of teaching intelligence.

[0003] At present, in the specific scene of badminton training, the existing technical methods mostly rely on visual data for action detection and evaluation, usually by high-speed cameras to collect the interaction process of the racket and the shuttlecock, and then by image recognition algorithms to extract sports features. Although visual technology can capture the action trajectory and contact timing, this method has significant drawbacks: on the one hand, visual acquisition is easily disturbed by external conditions such as changes in field lighting, occlusion, and shooting angle, resulting in unstable detection results; on the other hand, the visual method cannot directly quantify the strength of the hitting moment and the rationality of the racket swing, and the training feedback often lacks a comprehensive portrayal of the quality of the action, thereby limiting the effect of precise guidance on the athlete's action.

[0004] Under the above technical conditions relying on a single visual modality, when the environmental light is insufficient or the athlete's position deviates from the camera's collection angle, the system may produce false positives or false negatives; even if the hitting timing is recognized, it is still impossible to accurately distinguish whether the action force is reasonable. The resulting abnormal effects include: even if the athlete practices frequently, they cannot obtain effective feedback on the racket swing force and hitting point selection from the system, and long-term training can easily form incorrect action habits, leading to insufficient hitting speed or unreasonable hitting points in actual competition, and in severe cases, it may even cause action injury risks. These deficiencies indicate that the existing technology urgently needs to introduce additional modal information, especially acoustic features, to make up for the limitations of single visual feedback and improve the intelligence and scientific nature of action teaching. SUMMARY

[0005] To address the deficiencies of the prior art, the present application provides a sports action intelligent teaching system and method based on a multi-modal large model, which solves the problems mentioned in the background art.

[0006] To achieve the above purpose, the present application is implemented by the following technical solution:

[0007] S1, by arranging directional microphones and visual acquisition devices on a badminton training ground, collecting the athlete's hitting acoustic signal data and visual image data of the hitting interaction respectively, and preprocessing the acoustic signal data to obtain hitting energy data;

[0008] S2, calculating the acoustic energy ratio index Rste based on the hitting energy data, and determining the hitting strength by comparing the acoustic energy ratio index Rste with the preset energy threshold E, and triggering the cross-modal timing correction step based on the hitting strength determination result;

[0009] S3, in the cross-modal timing correction step, the hitting timing determination is performed, the hitting timing determination is performed by extracting the audio impact time Tah from the acoustic signal data, and extracting the visual contact time Tvis from the visual image data, and calculating and outputting the comprehensive determination index Dcomp combined with the acoustic energy ratio index Rste;

[0010] S4, according to the output result of the comprehensive determination index Dcomp, the correction feedback is performed, the correction feedback is performed by generating multi-channel teaching feedback information, and simultaneously performing dynamic optimization and learning mechanism.

[0011] Preferably, the S1 comprises S11;

[0012] S11, one directional microphone is arranged on each side of the training ground symmetrically to the position of the net, and the pickup main shaft of the directional microphone is directed towards the hitting area, and an auxiliary microphone is arranged on the top of the ground;

[0013] The original acoustic signal is collected based on the directional microphone, and the original acoustic signal is subjected to band-pass filtering processing, wherein the filtering frequency band is set to 1000Hz-8000Hz, the original acoustic signal subjected to the band-pass filtering processing is subjected to frame processing according to a preset frame length and frame shift, and a frame signal is obtained, wherein the length of each frame is 20ms, the interval between adjacent frames is 10ms, and a Hanning window function is superimposed on each frame signal;

[0014] Based on the frame signal, the short-time energy of each frame is calculated, and the calculation method of the short-time energy is as follows: the values of all sampling points in each frame signal are squared and added one by one, and the weighted result of the Hanning window is superimposed, so as to obtain the energy value corresponding to the frame;

[0015] In the short-time energy sequence, when it is detected that there is an energy mutation that is obviously more than three times the standard deviation of the background mean value, the time period corresponding to the energy mutation is defined as the window of the hitting impact sound, and the hitting window energy Ehit is extracted from the window of the hitting impact sound; At the same time, in the stable low-energy segment of a plurality of frames before hitting, the average value is extracted and counted as the background noise energy Ebase;

[0016] The ball hitting window energy Ehit and the background noise energy Ebase are summarized to obtain the ball hitting energy data.

[0017] Preferably, the S1 further comprises S12.

[0018] S12, high-speed cameras are arranged on the front and side of the training court, the frame rate of the high-speed cameras is not less than 200 frames per second, the resolution is not less than 1280*720, and a millisecond-level time stamp is added to each frame of image during collection;

[0019] During image collection, a target detection algorithm is used to detect and track the racket and the shuttlecock in real time, to generate an image sequence containing a racket bounding box and a shuttlecock bounding box, and to extract relative spatial position parameters between the racket and the shuttlecock based on the image sequence;

[0020] When the distance between the nearest edge of the racket bounding box and the center of the circumscribed circle of the shuttlecock bounding box is less than five pixels, and the diameter change rate of the shuttlecock bounding box exceeds 15% in three consecutive frames, it is determined that a contact event occurs, and the image frame corresponding to the moment is automatically marked as a contact key frame;

[0021] In a number of frames before and after the contact key frame, an inter-frame optical flow analysis technique is used to extract the mutation point of the shuttlecock velocity vector, and an edge recognition technique is used to detect the contact overlap area of the racket face and the shuttlecock profile;

[0022] The mutation point of the shuttlecock velocity vector and the time point at which the contact overlap area of the shuttlecock profile appears are compared, and the consistent time point is taken as the accurate time point at which the racket hits the shuttlecock, and the accurate time point at which the racket hits the shuttlecock is defined as the visual contact time Tvis.

[0023] Preferably, the S2 comprises S21.

[0024] S21, based on the ball hitting energy data, an acoustic energy ratio model is constructed, the ball window energy Ehit and the background noise energy Ebase in the real-time obtained ball hitting energy are taken as input items, the acoustic energy ratio model is input, the ratio of the ball hitting window energy Ehit to the background noise energy Ebase is taken as an acoustic energy ratio index Rste, the acoustic energy and the background energy level at the ball hitting moment are quantitatively analyzed; the acoustic energy ratio index Rste is calculated and output by the following acoustic energy ratio model: Rate=Ehit / Ebase.

[0025] Preferably, the S2 further comprises S22.

[0026] S22, in the initial stage, according to the acoustic energy ratio distribution of the ball hitting standard and non-standard in the historical training sample, the value capable of guaranteeing the false positive rate and the false negative rate in the acceptable range is selected as the initial energy threshold E through a statistical method;

[0027] The ball hitting force is determined based on the ability threshold E and the acoustic energy ratio index Rste corrected in real time, and the cross-modal timing correction step is triggered based on the ball hitting force determination result, and the specific determination content is as follows:

[0028] When the acoustic energy ratio index Rste is greater than or equal to the ability threshold E, it indicates that the ball hitting force meets the requirements, and the energy standard is determined, and no action teaching is performed;

[0029] When the acoustic energy ratio index Rste is less than the ability threshold E, it indicates that the ball hitting force does not meet the requirements, and the cross-modal timing correction step is triggered.

[0030] Preferably, the S3 comprises S31;

[0031] S31, after triggering the cross-modal timing correction step, the ball hitting timing determination is performed, and the ball hitting timing determination is performed by analyzing the frame energy of the acoustic signal, and when a peak value exceeding three times the standard deviation of the background mean value is detected, the time point corresponding to the peak value exceeding three times the standard deviation of the background mean value is defined as the audio impact time Tah;

[0032] Based on the audio impact time Tah, the visual contact time Tvis and the acoustic energy ratio index Rste, the output comprehensive determination index Dcomp is determined.

[0033] Preferably, the S3 further comprises S32;

[0034] S32, based on the output result of the comprehensive determination index Dcomp, the ball hitting timing determination is performed, and the deviation correction feedback is triggered based on the determination result, and the specific determination content is as follows:

[0035] When the comprehensive determination index Dcomp is 1, the ball hitting timing determination is standard;

[0036] When the comprehensive determination index Dcomp is 0, the ball hitting timing determination is not standard.

[0037] Preferably, the S4 comprises S41;

[0038] S41, based on the ball hitting timing determination result, the deviation correction feedback is triggered, and the deviation correction feedback is generated through the generation of multi-channel teaching feedback information to assist in action teaching, and the generation of multi-channel teaching feedback information includes voice channel, visual channel and rhythm channel;

[0039] When the comprehensive determination index Dcomp = 1, the first prompt voice is output through the voice channel, and the ball hitting trajectory contrast curve is superimposed in real time on the display terminal of the visual channel;

[0040] When the comprehensive determination index Dcomp = 0, the second prompt voice and the third prompt voice are output through the voice channel, and the deviation area of the ideal action trajectory and the actual trajectory is superimposed on the display terminal of the visual channel, and then the beat sound signal of the time difference between the audio impact time Tah and the visual contact time Tvis is output through the speaker through the beat channel;

[0041] If the acoustic energy ratio index Rste < the ability threshold E and the audio impact time Tah > the visual contact time Tvis, the second prompt voice is triggered;

[0042] If the acoustic energy ratio index Rste < the ability threshold E and the audio impact time Tah < the visual contact time Tvis, the third prompt voice is triggered.

[0043] Preferably, the S4 further comprises S42;

[0044] S42, the dynamic optimization and learning mechanism are executed at the same time of the deviation correction feedback, the dynamic optimization is performed by dynamically optimizing the energy threshold E and the time difference threshold T with every fifty times of hitting as a statistical period, and the specific optimization contents are as follows:

[0045] When the hitting pass rate > 80%, the energy threshold E is automatically increased by 5%-10%, and the time difference threshold T is automatically reduced by 5%-10%;

[0046] When the hitting pass rate < 50%, the energy threshold E is automatically reduced by 5%-10%, and the time difference threshold T is automatically increased by 5%-10%;

[0047] On this basis, the learning mechanism is executed, the intelligent teaching model is constructed by the joint distribution of the hitting force determination and the hitting time determination of the athletes in the statistical period, and the personalized intelligent teaching model is periodically updated based on the hitting force determination and the hitting time determination results.

[0048] A sports action intelligent teaching system based on a multi-modal large model, comprising a hitting perception module, a hitting force determination module, a hitting time determination module, and a comprehensive correction module;

[0049] The hitting perception module is arranged with directional microphones and visual acquisition devices in a badminton training field, and the hitting acoustic signal data and the visual image data of the hitting interaction of the athletes are collected respectively, and the acoustic signal data is preprocessed to obtain the hitting energy data;

[0050] The ball hitting force determination module determines the ball hitting force by calculating an acoustic energy ratio index Rste based on the ball hitting energy data, and determining the ball hitting force by comparing the acoustic energy ratio index Rste with a preset energy threshold E, and triggering a cross-modal timing correction step based on the ball hitting force determination result;

[0051] The ball hitting timing determination module determines the ball hitting timing by extracting an audio impact time Tah from the acoustic signal data and extracting a visual contact time Tvis from the visual image data, and calculating and outputting a comprehensive determination index Dcomp in combination with the acoustic energy ratio index Rste in the cross-modal timing correction step.

[0052] The comprehensive correction module performs deviation correction feedback according to the output result of the comprehensive determination index Dcomp, and simultaneously performs dynamic optimization and learning mechanism by generating multi-channel teaching feedback information.

[0053] The present application provides a sports action intelligent teaching system and method based on a multi-modal large model.

[0054] (1) The method can obtain the ball hitting window energy Ehit and the background noise energy Ebase by arranging directional microphones and high-speed cameras in the badminton training field, and combining the band-pass filtering, frame processing and short-time energy calculation of acoustic signals, and the target detection and optical flow analysis of visual images, and further constructing the acoustic energy ratio index Rste. This method ensures high signal-to-noise ratio extraction of the ball hitting acoustic signal in a complex environment, and can accurately locate the visual contact time Tvis in the visual modality. Compared with the existing teaching method which only relies on visual detection, this method still maintains high robustness under conditions such as light changes and site noise interference, and significantly improves the accuracy of ball hitting action data acquisition and analysis.

[0055] (2) The method combines the acoustic energy ratio index Rste with the audio impact time Tah and the visual contact time Tvis, and outputs the comprehensive determination index Dcomp in steps S2 and S3, which can simultaneously complete multi-dimensional evaluation of the ball hitting force and the ball hitting timing. Compared with the existing technology which only determines whether the action hits or not, this method can further trigger cross-modal correction when determining "insufficient force", and combine the differences between the acoustic peak value and the visual contact time to clearly distinguish the specific problems of "insufficient force" and "improper timing". On this basis, the system outputs feedback through three channels of voice prompt, visual trajectory superposition prompt and beat prompt sound, so that the athlete can intuitively perceive the action deviation point and make immediate correction, thereby shortening the training deviation correction period.

[0056] (3) The method introduces a dynamic optimization and learning mechanism in steps S4 and S42, and the method can dynamically adjust the energy threshold E and the time difference threshold T according to the passing rate of each fifty times of hitting based on a statistical cycle: when the passing rate is greater than 80%, the energy threshold E is increased by 5%-10% and the time difference threshold T is reduced by 5%-10% to increase the training difficulty; when the passing rate is less than 50%, the energy threshold E is reduced by 5%-10% and the time difference threshold T is increased by 5%-10% to avoid overtraining. At the same time, the system periodically updates the personalized intelligent teaching model according to the joint distribution of the hitting force determination and the hitting time determination of the athletes in the statistical cycle, so that the multi-channel feedback results can match the individual characteristics of different athletes, thereby realizing differentiated teaching and personalized training, and ensuring the stability and scientific nature of the training effect. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 A flowchart of a sports action intelligent teaching method based on a multi-modal large model.

[0058] Figure 2 A flowchart of a sports action intelligent teaching method based on a multi-modal large model.

[0059] Figure 3 A training site layout. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0061] Embodiment 1: Please refer to Figure 1 The present application provides a sports action intelligent teaching method based on a multi-modal large model. To achieve the above purpose, the present application is implemented by the following technical solutions: comprising the following steps:

[0062] S1, directional microphones and visual acquisition devices are arranged on a badminton training site to collect the hitting acoustic signal data and the visual image data of the hitting interaction of the athletes, and the acoustic signal data is preprocessed to obtain the hitting energy data;

[0063] S2, the acoustic energy ratio index Rste is calculated based on the hitting energy data, and the acoustic energy ratio index Rste is compared with the preset energy threshold E to determine the hitting force, and the cross-modal timing correction step is triggered based on the hitting force determination result;

[0064] S3, in the cross-modal timing correction step, a hitting timing determination is performed, the hitting timing determination is calculated by extracting an audio impact time Tah from the acoustic signal data, extracting a visual contact time Tvis from the visual image data, and combining an acoustic energy ratio indicator Rste to output a comprehensive determination indicator Dcomp;

[0065] S4, according to the output result of the comprehensive determination indicator Dcomp, a correction feedback is performed, the correction feedback is performed by generating multi-channel teaching feedback information, and a dynamic optimization and learning mechanism is simultaneously performed.

[0066] In this embodiment, the method S1 ensures the synchronous acquisition of the hitting acoustic signal and the visual image by arranging the directional microphone and the visual acquisition device in the badminton training field, wherein the pickup main shaft of the directional microphone is directed to the hitting area, which can significantly improve the signal-to-noise ratio of the hitting sound and the background noise. If only an environmental microphone is used, the background conversation sound or the venue echo will often drown the hitting signal, resulting in inaccurate extraction of the hitting energy data, therefore, this arrangement can effectively avoid noise interference and realize clear separation of the hitting energy Ehit and the background energy Ebase. The acoustic energy ratio indicator Rste is introduced in step S2, the substantial significance of which is to offset the influence of the overall environmental energy level through the ratio, so that the real strength of the hitting moment can be accurately reflected through the normalization method even if the background noise of the site is large, so that the hitting strength determination is more robust. In step S3, the difference between the audio impact time Tah and the visual contact time Tvis is compared to establish the comprehensive determination indicator Dcomp of the cross-modal, the physical significance of which is to truly depict the time alignment relationship between “sound production” and “racket contact”. If the hitting sound is obviously delayed compared with the visual contact, it often represents that the racket is swung too late, and vice versa, a single mode is difficult to reveal this problem, therefore, the cross-modal correction can accurately locate the action deviation. In step S4, three types of multi-channel feedback of voice, vision and rhythm are generated, and a dynamic optimization mechanism is combined, so that the system can prompt “too early or too late” in real time, and automatically reduce the training rhythm when the continuous multiple misjudgments occur, so as to prevent the athletes from repeatedly practicing under the wrong rhythm. Therefore, the method not only improves the accuracy and anti-interference of the hitting detection, but also realizes personalized and adaptive adjustment in the feedback link, finally ensures that the training process is scientific and efficient.

[0067] Embodiment 2: please refer to Figure 1 and Figure 3 Specifically, S1 includes S11;

[0068] S11, each of the two directional microphones is arranged at a symmetrical position of the net on the left and right sides of the training ground, and the pickup main shaft of the directional microphone is directed to the hitting area to enhance the signal-to-noise ratio of the hitting sound signal and the environmental noise, and an auxiliary microphone is arranged on the top of the training ground to collect the overall environmental noise, so as to realize multi-source fusion of acoustic signals and noise suppression;

[0069] The original acoustic signal is collected based on the directional microphone, and the original acoustic signal is subjected to band-pass filtering processing, wherein the filtering frequency band is set to 1000Hz-8000Hz to remove low-frequency human voice interference and high-frequency electromagnetic noise in addition to the hitting sound, and the original acoustic signal subjected to the band-pass filtering processing is subjected to frame processing according to a preset frame length and frame shift to obtain a frame signal, wherein the length of each frame is 20ms, the interval between adjacent frames is 10ms, and a Hanning window function is superimposed on each frame signal to reduce spectral leakage;

[0070] Based on the frame signal, the short-time energy of each frame is calculated, and the calculation method of the short-time energy is that the values of all sampling points in each frame signal are squared and added one by one, and the weighted result of the Hanning window is superimposed, so as to obtain the energy value corresponding to the frame;

[0071] In the short-time energy sequence, when it is detected that there is an energy mutation that obviously exceeds three times the standard deviation of the background mean value, the time period corresponding to the energy mutation is defined as the window of the hitting impact sound, and the hitting window energy Ehit is extracted from the window of the hitting impact sound; at the same time, in the stable low-energy segment of a plurality of frames before hitting, the average value is extracted and counted as the background noise energy Ebase for subsequent calculation and evaluation of the acoustic energy ratio index;

[0072] The hitting window energy Ehit and the background noise energy Ebase are collected to obtain the hitting energy data.

[0073] S1 also includes S12;

[0074] S12, a high-speed camera is arranged on the front and side of the training ground, the frame rate of the high-speed camera is not less than 200 frames per second, the resolution is not less than 1280*720, and a millisecond-level time stamp is added to each frame of image in the collection process to ensure the synchronization of image data and acoustic signal data in the time dimension;

[0075] In the process of image acquisition, a target detection algorithm is used to detect and track the racket and the shuttlecock in real time, to generate an image sequence containing the racket bounding box and the shuttlecock bounding box, and to extract the relative spatial position parameters between the racket and the shuttlecock based on the image sequence;

[0076] When the racket bounding box nearest edge is less than five pixels from the center of the shuttlecock bounding box circumscribed circle, and the diameter of the shuttlecock bounding box changes by more than 15% in three consecutive frames, a contact event is determined to have occurred, and the image frame corresponding to the time is automatically marked as a contact key frame;

[0077] Within several frames before and after the contact key frame, the frame-to-frame optical flow analysis technique is used to extract the mutation point of the shuttlecock speed vector, and the edge recognition technique is used to detect the contact overlap area of the racket face and the shuttlecock profile;

[0078] The mutation point of the shuttlecock speed vector and the time point of the appearance of the contact overlap area of the shuttlecock profile are compared, and the consistent time is taken as the accurate time point of the racket hitting the shuttlecock, and the accurate time point of the racket hitting the shuttlecock is defined as the visual contact time Tvis for subsequent cross-modal time correction steps.

[0079] In this embodiment, in S11, the directional microphone is arranged at the left and right symmetrical positions of the ball net, and the sound pickup main shaft is directed towards the hitting area, which can improve the signal-to-noise ratio of the hitting sound and the environmental noise at the physical level. If the microphone is directly facing the audience area or placed at the periphery of the field, the sound of the player hitting the ball is easily covered by the human voice and the sound of the shoe bottom rubbing, resulting in that the short-time energy mutation point is not obvious, and the Ehit is misjudged. Through symmetrical arrangement and cooperation with the auxiliary microphone at the top, the system can compare the energy difference of different sound sources, realize the noise suppression effect after multi-source fusion, and ensure the stability and separability of the hitting sound signal. In the signal processing link of S11, the band-pass filter frequency band is set to 1 kHz-8 kHz, and the purpose is to remove the low-frequency human voice and high-frequency electromagnetic interference. The sound pressure energy at the hitting moment is mainly concentrated in the medium and high frequency band, and if this frequency band is not set, the low-frequency "rumbling" of the human voice will raise the Ebase, directly lower the Rste, and make the system misjudge as "insufficient force". Through this frequency band screening, it can ensure that the energy extraction link is more in line with the true physical meaning: Ehit represents the instantaneous intensity of impact sound, and Ebase represents the stable noise level. In S12, the frame rate of the high-speed camera is set to not less than 200 fps, because the ordinary 30 fps camera will cause the hitting contact process to span multiple frames in actual training, and cannot capture the accurate contact time, which is easy to cause Tvis delay deviation. For example, if the hitting occurs between two frames, the ordinary camera can only estimate within ±15 ms, while 200 fps can compress the error to ±2.5 ms, so as to ensure that the alignment determination of Tah and Tvis is more accurate, and to avoid the time difference amplification effect in the cross-modal correction process. Through the above implementation mode, the system realizes effective separation of the hitting sound and the environmental noise at the physical level, accurate energy capture at the hitting moment, and millisecond-level alignment of the acoustic and visual signals. This design not only avoids the misjudgment caused by too much noise or image blur in the traditional single mode, but also improves the accuracy and robustness of the training feedback, so that the subsequent cross-modal determination and dynamic optimization have a reliable data basis.

[0080] Embodiment 3: please refer to Figure 1 , specifically: S2 includes S21;

[0081] S21, constructing an acoustic energy ratio model based on the hitting energy data, taking the ball window energy Ehit and the background noise energy Ebase in the real-time acquired hitting energy as input items, inputting the acoustic energy ratio model, taking the ratio of the hitting window energy Ehit and the background noise energy Ebase as the acoustic energy ratio index Rste, and quantitatively analyzing the acoustic energy and the background energy level at the hitting moment; the acoustic energy ratio index Rste is calculated and output by the following acoustic energy ratio model: Rate = Ehit / Ebase;

[0082] Origin and basic principle of the formula: the measurement of acoustic energy belongs to the classical formula in the field of signal processing, which is derived from the principles of physics and digital signal processing (DSP): in acoustic signal processing, short-time energy is an important indicator to characterize the intensity of sound. The classical formula is: short-time energy = cumulative sum of the square of the amplitude of each sampling point; this is derived from the principle of physics that "energy is proportional to the square of the amplitude". On this basis, the formula defines the short-time energy Ehit of the hitting window and the background noise energy Ebase, and constructs the acoustic energy ratio index Rste;

[0083] The calculation method of short-time energy itself follows the classical formula (physical basis); the improvement of the formula is that the hitting window and the stationary background window are distinguished, which avoids misjudgment caused by environmental noise fluctuations; the acoustic energy ratio index Rste is used instead of a single energy value, which improves the robustness in different venues and different noise environments;

[0084] Dimensional consistency analysis: Ehit, Ebase: the unit is "amplitude square sum", which can correspond to the relative amount of sound pressure square (Pa²); Rste = Ehit / Ebase: since it is a ratio, it is finally dimensionless, which conforms to the physical principle;

[0085] Specific examples:

[0086] Background noise energy Ebase: 10 frames are selected in the stationary section, and the energy of each frame is [102, 98, 95, 100, 97, 96, 101, 99, 94, 98]; the average value is (102+98+95+100+97+96+101+99+94+98) / 10=98; therefore, Ebase=98;

[0087] Hitting window energy Ehit: in the window where the hitting sound is detected, the short-time energy is calculated to be 560; therefore, Ehit=560;

[0088] Acoustic energy ratio Rste: Rste=Ehit / Ebase=560 / 98≈5.71.

[0089] S2 also includes S22;

[0090] S22, in the initial stage, according to the distribution of acoustic energy ratio of hitting standard and non-standard in the history training sample, the value that can ensure the false positive rate and the false negative rate in the acceptable range is selected as the initial energy threshold E through statistical method;

[0091] The ability threshold E based on real-time correction and the acoustic energy ratio index Rste are used to determine the hitting force, and the cross-modal timing correction step is triggered based on the hitting force determination result. The specific determination content is as follows:

[0092] When the acoustic energy ratio index Rste is greater than the ability threshold E, it indicates that the hitting force meets the requirements, and the energy is determined to meet the requirements, and no action teaching is performed.

[0093] When the acoustic energy ratio index Rste is less than the ability threshold E, it indicates that the hitting force does not meet the requirements, and the cross-modal opportunity correction step is triggered.

[0094] In this embodiment, in S21, the ball hitting window energy Ehit is ratio processed with the background noise energy Ebase to obtain the acoustic energy ratio index Rste, and the core purpose is to offset the random fluctuations of environmental noise. If only Ehit is used as the basis for judgment, when there is sudden noise (such as audience coughing sound, shoe friction sound) in the training ground, these noise peaks will be misidentified as hitting energy, resulting in misjudgment. By using the ratio form, Rste can reflect the increase amplitude of the hitting impact energy relative to the background noise in real time, has the "signal intensity contrast" in the physical sense, and thus significantly improves the stability and robustness of the judgment. In S22, a statistical method based on the distribution of historical training samples is introduced to set the initial energy threshold E, which can avoid one-size-fits-all misjudgment caused by individual differences or site differences. For example, a beginner's Ehit is generally low when hitting, and if the energy threshold of a high-level athlete is directly applied, the system will frequently determine that it does not meet the requirements, resulting in too much negative feedback and damaging the training experience. By setting the threshold through statistical distribution, the false positive rate and false negative rate can be maintained within a reasonable range, and the threshold can be personalized and scientific. In the physical sense, this implementation ensures that the ball hitting judgment is not only a measure of "absolute energy", but also a measure of "relative energy increase"; in the training scene, it ensures that athletes of different levels and in different environments can obtain fair and accurate force evaluation. The beneficial effects brought by this are: significantly reducing the misjudgment risk caused by environmental noise interference, improving the adaptability of the system and the accuracy of the training feedback, so that the subsequent cross-modal opportunity correction can be triggered on a more reliable basis, avoiding invalid or redundant correction steps.

[0095] Embodiment 4: Please refer to Figure 1 , specifically: S3 includes S31;

[0096] S31, after triggering the cross-modal opportunity correction step, the ball hitting opportunity is determined, and the ball hitting opportunity is determined by analyzing the energy of the acoustic signal in frames. When a peak value that exceeds three times the standard deviation of the background mean is detected, the time point corresponding to the peak value that exceeds three times the standard deviation of the background mean is defined as the audio impact time Tah.

[0097] Based on the audio impact moment Tah, visual contact moment Tvis and acoustic energy ratio index Rste, the output comprehensive judgment index Dcomp is output, and the specific output formula of the output comprehensive judgment index Dcomp is: Dcomp=1[Rste≥E]×1[|Tah-Tvis|<T]; In the formula, T represents a time difference threshold, which is set in the interval of 20ms-30ms;

[0098] The formula is derived from physical meaning: signal processing and detection logic is derived from, the acoustic part: the acoustic energy ratio index Rste is derived from the classical short-time energy formula, physics and digital signal processing DSP principle, which is used to measure the ratio of ball hitting sound energy and background noise energy; The visual part: the visual contact moment Tvis is derived from target detection and motion analysis algorithm, which is a common detection means in computer vision; Time difference analysis: the difference between Tah and Tvis |Tah-Tvis| belongs to the classical time alignment algorithm, which is used to detect the synchronization of multi-modal signals;

[0099] The improvement of the formula is: two independent modalities (acoustic energy and visual contact) are fused into a binary decision formula; Through logical product operation, the comprehensive judgment index Dcomp is constructed to realize the overall evaluation of the hitting action, instead of single index judgment;

[0100] 1[condition] in the formula represents the exponential function; That is, when the condition is true, =1, and when the condition is not true, =0;

[0101] 1[Rste≥E] represents that the hitting strength meets the requirements, and 1[|Tah-Tvis|<T] represents that the hitting timing meets the requirements;

[0102] Why does the hitting strength not meet the requirements? Trigger the cross-modal timing correction step, because if the strength is enough, it means that the athlete's action basis strength has no problem, and there is no need to immediately enter the complex cross-modal analysis link, and directly give the positive feedback of "strength meets the requirements", which can reduce the system calculation burden and avoid too much meaningless analysis. When the hitting strength is insufficient, simply telling the athlete "the strength is not enough" has no guiding significance. The root cause of insufficient strength is often not muscle strength, but action timing error (such as hitting the ball too early or too late, resulting in no best force hitting the ball), so when Rste<E, trigger the cross-modal correction, and compare the acoustic peak Tah and the visual contact moment Tvis, to determine whether the problem is "insufficient force" or "improper timing".

[0103] S3 also includes S32;

[0104] S32, based on the output result of the comprehensive judgment index Dcomp, the hitting timing is judged, and the correction feedback is triggered based on the judgment result, and the specific judgment content is as follows:

[0105] When the comprehensive determination index Dcomp = 1, the ball hitting timing is determined to be up to standard;

[0106] When the comprehensive determination index Dcomp = 0, the ball hitting timing is determined to be not up to standard.

[0107] In this embodiment, in the implementation process of S31, the audio impact moment Tah is located by framing the acoustic signal and using "three times the standard deviation" as the peak detection threshold, which aims to avoid the false triggering of random noise in the environment. For example, the shoe bottom friction sound or the audience's clapping sound commonly seen in a badminton hall can also cause a sudden change in energy, but the amplitude of these noises is often insufficient to exceed three times the standard deviation of the background mean. Through this determination condition, the real ball hitting sound can be accurately identified, thereby ensuring that Tah has a clear physical meaning, that is, the impact sound at the moment of the racket hitting the ball. In the cross-modal correction, Tah and the visual contact moment Tvis are time-aligned and calculated, and the core goal is to verify whether the action force timing matches the actual contact timing of the ball. For example, if Tah is significantly later than Tvis, it means that the athlete hits the ball with a lag in force, although the force is up to standard, but the hitting point has deviated from the optimal position, resulting in a decrease in the quality of the hit. At this time, Dcomp outputs 0, and the system can directly attribute the problem to "improper timing" to avoid the athlete's misunderstanding due to the simple feedback of "insufficient force". In S32, through the binary output of Dcomp (1 = up to standard, 0 = not up to standard), a simple and efficient feedback determination logic is realized. The advantage of this logic is real-time and interpretability: during training, the system can clearly tell the athlete whether the action is "passed" or "needs to be corrected" without complex calculations. At the same time, since the formula logic integrates the force and timing dimensions, it avoids the one-sidedness brought by single-index determination, making the teaching feedback more targeted. The beneficial effects of this implementation are: on the one hand, through the coupling of acoustic and visual dual modalities, the high precision and robustness of the ball hitting timing detection are ensured, and on the other hand, through the simple Dcomp output, the feedback is both fast and intuitive, which can effectively improve the learning efficiency of athletes in training and reduce the repeated training cost caused by false determination.

[0108] Embodiment 5: Please refer to Figure 1 , specifically: S4 includes S41;

[0109] ​S41, based on the ball hitting timing determination result, trigger the deviation correction feedback, the deviation correction feedback generates multi-channel teaching feedback information to assist in action teaching, the multi-channel teaching feedback information includes a voice channel, a visual channel and a rhythm channel; wherein the voice channel outputs positive prompt information to prompt the athlete's action deviation; the visual channel superimposes the standard swing trajectory and the athlete's actual trajectory residual image on the display terminal, so that the athlete can intuitively identify the deviation position; the rhythm channel plays a rhythm sound signal proportional to the time difference through the loudspeaker to guide the athlete to gradually correct the ball hitting timing;

[0110] When the comprehensive determination index Dcomp=1, the first prompt voice "the ball hitting is powerful and the timing is appropriate" is output through the voice channel, and the ball hitting trajectory comparison curve is superimposed on the display terminal in the visual channel in real time to strengthen the athlete's memory of correct action;

[0111] When the comprehensive determination index Dcomp=0, the second prompt voice and the third prompt voice are output through the voice channel, and the deviation area of the ideal action trajectory and the actual trajectory is superimposed on the display terminal in the visual channel, and then the rhythm channel outputs the rhythm sound signal of the time difference between the audio impact time Tah and the visual contact time Tvis through the loudspeaker;

[0112] When the acoustic energy ratio index Rste is less than the ability threshold E and the audio impact time Tah is greater than the visual contact time Tvis, the second prompt voice is triggered to prompt "swing too late, please advance";

[0113] When the acoustic energy ratio index Rste is less than the ability threshold E and the audio impact time Tah is less than the visual contact time Tvis, the third prompt voice is triggered to prompt "swing too early, please delay".

[0114] S4 also includes S42;

[0115] S42, simultaneously with the deviation correction feedback, a dynamic optimization and learning mechanism is executed, the dynamic optimization optimizes the energy threshold E and the time difference threshold T by taking every fifty times of hitting as a statistical period, and the specific optimization content is as follows:

[0116] When the hitting pass rate is greater than 80%, the energy threshold E is automatically increased by 5%-10%, and the time difference threshold T is decreased by 5%-10% to tighten the determination standard;

[0117] When the hitting pass rate is less than 50%, the energy threshold E is automatically decreased by 5%-10%, and the time difference threshold T is increased by 5%-10% to avoid the negative effects caused by excessive training;

[0118] On this basis, a learning mechanism is executed, the learning mechanism constructs an intelligent teaching model by a joint distribution of the ball hitting force determination and the ball hitting time determination of the athlete in a statistical period, and periodically updates the personalized intelligent teaching model based on the ball hitting force determination and the ball hitting time determination result, so that the multi-channel teaching feedback information can adapt to the action characteristics of different athletes and provide differentiated guidance.

[0119] In the embodiment, in the implementation of S41, the core of the design of the multi-channel feedback is to reduce the ambiguity that a single prompt may bring. For example, when only relying on the voice prompt "action deviation", the athlete has difficulty in understanding the specific problem at a moment, and by superimposing the standard trajectory and the residual image for comparison, the athlete can directly see the deviation point. At the same time, the design of the rhythm channel is to convert the abstract time difference |Tah-Tvis| into an auditory rhythm, and the athlete is more likely to correct the action timing through the rhythm in actual training. The physical meaning of this setting is to convert the determination results of multi-modal data into multi-sensory feedback, which maximizes the reduction of information delay and understanding deviation. For example, when the audio impact moment Tah falls behind the visual contact moment Tvis, simply prompting "too late" is often difficult for beginners to understand, but the rhythm sound starting in advance can immediately form an action conditioned reflex, thereby quickly improving the ball hitting timing. In S42, the purpose of introducing the dynamic optimization mechanism is to prevent the training standard from being "too wide" or "too strict". If the energy threshold E remains unchanged for a long time, when the athlete's level improves, the standard that was originally easy to meet will lead to a lack of challenge in training and stagnation of progress; and if it is set too strictly, it may lead to frustration of the athlete under the continuous failure feedback. By counting 50 times of hitting and dynamically adjusting the ability threshold E and the time difference threshold T, the system can adjust the determination difficulty according to the real performance. For example, if the pass rate exceeds 80%, the determination standard is tightened, which is equivalent to gradually reducing the allowed error interval of the action, helping the athlete to enter a higher level of fine training; on the contrary, when the pass rate is less than 50%, the threshold is appropriately relaxed to ensure that the training process maintains positive feedback and avoids excessive load. This strategy is equivalent to "progressive load training" in physical meaning, and through the dynamic adaptation mechanism, the feedback always fits the ability boundary of the athlete. The beneficial effects of this implementation are that the system can not only correct the action immediately, but also adjust the standard as the athlete's level improves, realizing "personalized progressive teaching". The ultimate improvement is reflected in: the training feedback is more intuitive, the action adjustment is more efficient, the training rhythm is more scientific, and the athlete can maintain motivation and progress in the constant dynamic balance.

[0120] Embodiment 6: Please refer to Figure 1 and Figure 2 A sports action intelligent teaching system based on a multi-modal large model, comprising a ball hitting perception module, a ball hitting force determination module, a ball hitting time determination module, and a comprehensive correction module.

[0121] The ball hitting perception module is configured to set up directional microphones and visual acquisition devices on a badminton training court, to collect the acoustic signal data of the ball hitting of the athletes and the visual image data of the ball hitting interaction, and to preprocess the acoustic signal data to obtain the ball hitting energy data;

[0122] The ball hitting force determination module is configured to calculate an acoustic energy ratio index Rste based on the ball hitting energy data, to determine the ball hitting force by comparing the acoustic energy ratio index Rste with a preset energy threshold E, and to trigger the cross-modal timing correction step based on the ball hitting force determination result.

[0123] The ball hitting timing determination module is configured to perform the ball hitting timing determination in the cross-modal timing correction step, to extract an audio impact time Tah from the acoustic signal data, to extract a visual contact time Tvis from the visual image data, and to calculate and output a comprehensive determination index Dcomp in combination with the acoustic energy ratio index Rste.

[0124] The comprehensive correction module is configured to perform the correction feedback according to the output result of the comprehensive determination index Dcomp, to generate the multi-channel teaching feedback information, and to simultaneously perform the dynamic optimization and learning mechanism.

[0125] Although the embodiments of the present application have been shown and described, it can be understood by those skilled in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application.

Claims

1. A multi-modal large model-based sports action intelligent teaching method, characterized in that: The method comprises the following steps: S1, by arranging directional microphones and visual acquisition devices on the badminton training ground, collecting the sound signal data of the player's hitting and the visual image data of the hitting interaction, and preprocessing the sound signal data to obtain the hitting energy data; S2, based on the hitting energy data, calculate the acoustic energy ratio index Rste, and determine the hitting strength of the acoustic energy ratio index Rste and the preset energy threshold E, and trigger the cross-modal timing correction step based on the hitting strength determination result; S3, in the cross-modal timing correction step, the hitting timing determination is performed, the hitting timing determination is performed by extracting the audio impact time Tah from the acoustic signal data, and the visual contact time Tvis is extracted from the visual image data, and the acoustic energy ratio index Rste is combined to calculate and output the comprehensive determination index Dcomp; S4, according to the output result of the comprehensive determination index Dcomp, the correction feedback is carried out, the correction feedback is carried out by generating multi-channel teaching feedback information, and dynamic optimization and learning mechanism are carried out at the same time.

2. The sports action intelligent teaching method based on a multi-modal large model according to claim 1, characterized in that: The S1 comprises S11; S11, one directional microphone is arranged on the left and right sides of the training ground symmetrically to the position of the net, and the pickup main shaft of the directional microphone is directed to the hitting area, and an auxiliary microphone is arranged on the top of the field; Based on the directional microphone, the original acoustic signal is collected, and the original acoustic signal is processed by band-pass filtering, wherein the filter band is set to 1000Hz-8000Hz, the original acoustic signal processed by the filter is processed by frame processing according to the preset frame length and frame shift, the frame signal is obtained, wherein the length of each frame is 20ms, the interval of adjacent frames is 10ms, and the Hann window function is superimposed on each frame signal; Based on the frame signal, the short-time energy of each frame is calculated, the calculation method of the short-time energy is: the square of all sample point values in each frame signal is added one by one, and the weighted result of the Hann window is added, so as to obtain the energy value corresponding to the frame; In the short-time energy sequence, when it is detected that there is an energy mutation obviously exceeding three times of the background mean value, the time period corresponding to the energy mutation is defined as the hitting impact sound window, the hitting window energy Ehit is extracted from the hitting impact sound window; At the same time, in the stable low energy segment of a plurality of frames before hitting, the average value is extracted and counted as the background noise energy Ebase; The hitting window energy Ehit and the background noise energy Ebase are collected to obtain the hitting energy data.

3. The method of claim 2, wherein the method is characterized in that: The S1 further comprises S12; S12, high-speed cameras are arranged on the front and side of the training ground, the frame rate of the high-speed cameras is not less than 200 frames per second, the resolution is not less than 1280*720, and a millisecond-level time stamp is added to each frame of image during the collection process; In the process of image acquisition, the target detection algorithm is used for real-time detection and tracking of the racket and the shuttlecock, the image sequence containing the racket bounding box and the shuttlecock bounding box is generated, and the relative spatial position parameters between the racket and the shuttlecock are extracted based on the image sequence; When the racket bounding box nearest edge is less than five pixels away from the center of the badminton bounding box circumscribed circle, and the diameter variation rate of the badminton bounding box exceeds 15% in three consecutive frames, it is determined that a contact event occurs, and the image frame corresponding to the moment is automatically marked as a contact key frame; Within a certain number of frames before and after the contact key frame, the frame inter-flow analysis technique is used to extract the mutation point of the badminton speed vector, and the edge recognition technique is used to detect the contact overlap area of the racket face and the badminton profile; The mutation point of the badminton speed vector and the time point of the contact overlap area of the badminton profile are compared, and the consistent time point is taken as the accurate time point of the racket hitting the badminton, and the accurate time point of the racket hitting the badminton is defined as the visual contact time Tvis.

4. The multi-modal large model-based sports action intelligent teaching method according to claim 3, characterized in that: The S2 includes S21; S21, based on the ball hitting energy data, an acoustic energy ratio model is constructed, the ball window energy Ehit and the background noise energy Ebase obtained in real time are taken as input items, the acoustic energy ratio model is input, the ratio of the ball window energy Ehit and the background noise energy Ebase is taken as the acoustic energy ratio index Rste, and the acoustic energy and the background energy level at the ball hitting moment are quantitatively analyzed.

5. The multi-modal large model-based sports action intelligent teaching method according to claim 4, characterized in that: The S2 also includes S22; S22, in the initial stage, according to the acoustic energy ratio distribution of the ball hitting standard and non-standard in the historical training sample, the value that can ensure the false positive rate and the false negative rate in the acceptable range is selected as the initial energy threshold E through statistical method; Based on the real-time corrected ability threshold E and the acoustic energy ratio index Rste, the ball hitting strength is determined, and based on the ball hitting strength determination result, the cross-modal time correction step is triggered, and the specific determination content is as follows: When the acoustic energy ratio index Rste is greater than the ability threshold E, it indicates that the ball hitting strength meets the requirements, and it is determined that the energy meets the requirements, and no action teaching is performed; When the acoustic energy ratio index Rste is less than the ability threshold E, it indicates that the ball hitting strength does not meet the requirements, and the cross-modal time correction step is triggered.

6. The multi-modal large model-based sports action intelligent teaching method according to claim 5, characterized in that: The S3 includes S31; S31, after triggering the cross-modal time correction step, the ball hitting time determination is performed, the ball hitting time determination is performed through frame energy analysis on the acoustic signal, when a peak value exceeding three times the standard deviation of the background mean value is detected, the time point corresponding to the peak value exceeding three times the standard deviation of the background mean value is defined as the audio impact time Tah; Based on the audio impact time Tah, the visual contact time Tvis and the acoustic energy ratio index Rste, the output comprehensive determination index Dcomp is determined.

7. The multi-modal large model-based sports action intelligent teaching method according to claim 6, characterized in that: The S3 also includes S32; S32, based on the output result of the comprehensive determination index Dcomp, the ball hitting time determination is performed, and based on the determination result, the deviation correction feedback is triggered, and the specific determination content is as follows: When the comprehensive determination index Dcomp is 1, the ball hitting time determination is standard; When the comprehensive determination index Dcomp is 0, the ball hitting time determination is not standard.

8. The multi-modal large model-based sports action intelligent teaching method according to claim 7, characterized in that: The S4 includes S41; S41, based on the ball hitting timing determination result, trigger the deviation correction feedback, the deviation correction feedback generates multi-channel teaching feedback information to assist in action teaching, and the generation of multi-channel teaching feedback information includes voice channel, visual channel and rhythm channel; When the comprehensive determination index Dcomp=1, the first prompt voice is output through the voice channel, and the ball hitting trajectory contrast curve is superimposed in real time on the display terminal of the visual channel; When the comprehensive determination index Dcomp=0, the second prompt voice and the third prompt voice are output through the voice channel, and the deviation area of the ideal action trajectory and the actual trajectory is superimposed on the display terminal of the visual channel, and then the beat sound signal of the time difference between the audio impact time Tah and the visual contact time Tvis is output through the speaker through the rhythm channel; Wherein, if the acoustic energy ratio index Rste is less than the ability threshold E and the audio impact time Tah is greater than the visual contact time Tvis, the second prompt voice is triggered; When the acoustic energy ratio index Rste is less than the ability threshold E and the audio impact time Tah is less than the visual contact time Tvis, the third prompt voice is triggered.

9. The multi-modal large model-based sports action intelligent teaching method according to claim 8, characterized in that: The S4 further comprises S42; S42, while the deviation correction feedback is executed, a dynamic optimization and learning mechanism is executed, the dynamic optimization is executed by taking every fifty times of hitting as a statistical period to dynamically optimize the energy threshold E and the time difference threshold T, and the specific optimization content is as follows: When the hitting pass rate is greater than 80%, the energy threshold E is automatically increased by 5%-10%, and the time difference threshold T is decreased by 5%-10%; When the hitting pass rate is less than 50%, the energy threshold E is automatically decreased by 5%-10%, and the time difference threshold T is increased by 5%-10%; On this basis, the learning mechanism is executed, the learning mechanism constructs an intelligent teaching model according to the joint distribution of the hitting force determination and the ball hitting timing determination of the athletes in the statistical period, and periodically updates the individualized intelligent teaching model based on the hitting force determination and the ball hitting timing determination result.

10. A sports action intelligent teaching system based on a multi-modal large model, applied to the sports action intelligent teaching method based on the multi-modal large model in any one of claims 1-9, characterized in that: It comprises a ball hitting perception module, a hitting force determination module, a ball hitting timing determination module and a comprehensive correction module; The ball hitting perception module is arranged with a directional microphone and a visual acquisition device in a badminton training field, and the ball hitting acoustic signal data and the visual image data of the hitting interaction of the athletes are collected respectively, and the acoustic signal data is preprocessed to obtain the hitting energy data; The hitting force determination module calculates the acoustic energy ratio index Rste based on the hitting energy data, and determines the hitting force based on the acoustic energy ratio index Rste and the preset energy threshold E, and triggers the cross-modal timing correction step based on the hitting force determination result; The ball hitting timing determination module executes the ball hitting timing determination in the cross-modal timing correction step, extracts the audio impact time Tah from the acoustic signal data, extracts the visual contact time Tvis from the visual image data, and calculates and outputs the comprehensive determination index Dcomp in combination with the acoustic energy ratio index Rste; The comprehensive correction module performs correction feedback according to the output result of the comprehensive judgment index Dcomp, and the correction feedback is performed by generating multi-channel teaching feedback information, and simultaneously performing dynamic optimization and learning mechanism.

Images