Real-time swimming stroke recognition method and system for swimmer in indoor swimming pool environment and storage device

Abstract

The invention discloses a real-time swimming stroke recognition method and system for a swimmer in an indoor swimming pool environment and a storage device, and belongs to the field of human body action recognition. The method comprises the following steps: collecting video data in real time, wherein the video data comprises action sequence information, body posture information and time sequence dynamic information of a swimmer under different swimming strokes; intercepting video clips from the video data, classifying the video clips, performing frame extraction processing to obtain multiple frames of images, storing the multiple frames of images according to a time sequence, and arranging the multiple frames of images into a structured swimming stroke data set; constructing a swimming stroke recognition and classification model by adopting a YOLOv8 and Recogniter2D model framework, and performing training by utilizing the structured swimming stroke data set, so that the trained swimming stroke recognition and classification model can perform swimming stroke classification according to action sequence information, body posture information and time sequence dynamic information; and inputting video data acquired in real time into the trained swimming stroke classification model, and outputting a swimming stroke type. According to the technical scheme, real-time monitoring, accurate classification and quantitative analysis of swimming strokes are achieved, and an efficient, accurate and practical solution is provided for swimming training and swimming pool monitoring.

Description

Technical Field

[0001] The present invention relates to the field of human motion recognition, and in particular to a method, system and storage device for real-time swimming posture recognition of swimmers in an indoor swimming pool environment. Background Art

[0002] With the rapid development of computer vision and deep learning technologies, action recognition based on video data is playing an increasingly important role in fields such as sports training, health monitoring, and security surveillance. As a key research area in pattern recognition, swimming stroke recognition has broad application prospects in areas such as swimming training assistance, athlete motion analysis, and pool safety monitoring.

[0003] Traditional swimming style recognition methods rely primarily on manual observation or wearable devices. However, since manual observation relies on the coach's experience and visual judgment, it is susceptible to subjective factors and environmental interference, making it difficult to achieve real-time, accurate monitoring. Furthermore, manual observation cannot provide quantitative data support, limiting trainers' in-depth analysis and improvement of their own techniques. While wearable device-based methods can collect swimmers' motion data and provide certain quantitative analysis, wearing the devices can affect the swimmer's athletic experience. Furthermore, the high cost and complex maintenance make them difficult to promote on a large scale in ordinary swimming venues. Summary of the Invention

[0004] In view of the problems that the above-mentioned manual observation method cannot achieve real-time monitoring and accurate quantitative analysis, and that wearable devices affect the sports experience, are costly and difficult to promote on a large scale, the present invention provides a method, system and storage device for real-time swimming posture recognition of swimmers in an indoor swimming pool environment, and adopts YOLOv8 and Recognizer2D model framework to process the swimming posture classification task in the present invention. Among them, YOLOv8 can quickly locate and track the positions of multiple swimmers, and generate independent motion segments for each target through bounding box detection and temporal association, thereby providing high-quality single-target input for the subsequent Recognizer2D model to ensure the accuracy of swimming posture classification; the Recognizer2D model framework selects PPTSM_v2 as the backbone network, combining the lightweight design of PP-LCNetV2 and the temporal modeling capability of TSM (Temporal Shift Module) to extract the spatial features and temporal dynamic information of video frames. PPTSM_v2 improves the efficiency and performance of the model through technologies such as deep separable convolution, reparameterization, channel attention mechanism and residual connection. Then, MoViNetHead is used as the classification head to map the features extracted by the backbone network into the category space to generate swimming style classification results. This enables real-time monitoring, precise classification and quantitative analysis of swimming styles, providing an efficient, accurate and practical solution for swimming training and pool monitoring.

[0005] In order to solve the above technical problems, the technical solution of the present invention is:

[0006] According to a first aspect of the technical solution of the present invention, a method for real-time swimming stroke recognition of a swimmer in an indoor swimming pool environment is provided, wherein the method comprises the following steps:

[0007] S1: Real-time acquisition of video data, including motion sequence information, body posture information, and temporal dynamic information of the swimmer in different swimming strokes;

[0008] S2: extracting video clips from the video data, classifying the video clips and performing frame extraction processing to obtain multiple frames of images, storing the multiple frames of images in chronological order and organizing them into a structured swimming style dataset;

[0009] S3: constructing a swimming stroke recognition and classification model using the YOLOv8 and Recognizer2D model framework, and using the structured swimming stroke dataset for training, so that the trained swimming stroke recognition and classification model can classify swimming strokes based on action sequence information, body posture information, and temporal dynamic information;

[0010] S4: Inputting the real-time collected video data into the trained swimming style classification model to output the swimming style type.

[0011] Furthermore, in said S1:

[0012] The motion sequence information includes: the trajectory, frequency and amplitude of the arm stroke, the rhythm, strength and method of the leg kick, the coordination of the body and the breathing action; it is used to capture the typical motion patterns of different swimming styles;

[0013] The body posture information includes: the positions of key body joints, body angles and the overall outline information of the swimmer; it is used to describe the swimmer's static posture and assist in identifying the swimming style;

[0014] The temporal dynamic information includes: periodic changes in stroke, kicking and breathing, changes in the speed of arm stroke and leg kicking, and transition information between different actions; it is used to capture the continuity and dynamic changes of actions.

[0015] Furthermore, in S2, the swimming style data set is divided into five categories: breaststroke, freestyle, butterfly stroke, backstroke and uncertain.

[0016] Furthermore, in S2, the steps of constructing the swimming style dataset are as follows:

[0017] S21 video data capture: capturing a video clip of a single person swimming from the video data;

[0018] S22: Classifying and labeling swimming styles: classifying the video clips according to swimming style categories, and labeling the video clips by category;

[0019] S23 video frame extraction processing: performing frame extraction processing on the classified video segments, converting each video segment into multiple frame images, and storing them in a folder corresponding to each video segment in chronological order;

[0020] S24 Dataset organization and storage: organizing the folders into a structured swimming stroke dataset, and extracting all folder paths into a list format as input data for a swimming stroke recognition and classification model;

[0021] S25 Dataset Division and Usage: The structured swimming stroke dataset is divided into a training set, a validation set, and a test set, which are used for swimming stroke recognition and classification model training, tuning, and performance evaluation, respectively.

[0022] Furthermore, in S21, a video capture tool is used to capture a video clip of a single person swimming from the video data, specifically including:

[0023] Drag and draw a rectangular box to select the swimmer's area in real time;

[0024] Dynamically adjust the frame position and size to ensure that the target person is completely contained in the frame;

[0025] After clicking the "Save" button, the video content of the selected area within the current playback time period will be automatically captured, the redundant images outside the frame will be cropped out, and the high-definition image quality will be retained, and the video clip will be output as a single person swimming.

[0026] Furthermore, if segmented capture is required, the selection range can be adjusted by pausing multiple times, and the current segment can be saved after clicking “Stop” each time. This will eventually generate multiple cropped videos that precisely focus on the swimmer’s movements, providing high-quality input for subsequent analysis.

[0027] Furthermore, in S21, each segment is 3-10 seconds long, ensuring that it contains at least one complete swimming action.

[0028] Furthermore, in said S22, the swimming stroke categories are divided into five categories: breaststroke, freestyle, butterfly stroke, backstroke and uncertain.

[0029] Furthermore, in S3, the YOLOv8 model simultaneously detects and distinguishes multiple swimmers in the swimming pool scene, and stably outputs the precise bounding box and tracking ID of each target. Through bounding box detection and temporal association, an independent motion segment is generated for each target, thereby providing single target input for the subsequent Recognizer2D model.

[0030] Furthermore, in S3, the Recognizer2D model framework selects the PPTSM_v2 model as the backbone network, fuses spatial features and temporal features to generate a comprehensive spatiotemporal feature representation, and uses MoViNetHead as the classification head for classification;

[0031] Among them, the PPTSM_v2 model includes the PP-LCNetV2 model and the TSM (Temporal Shift Module) module.

[0032] Furthermore, the spatial features are extracted by the PP-LCNetV2 model based on the action sequence information and body posture information; and the temporal features are extracted by the TSM module based on the temporal dynamic information.

[0033] Furthermore, in said S3, said swimming style recognition and classification model includes an encoder and an output layer;

[0034] The encoder contains multiple stacked coding layers to capture the spatial features in the video data and understand the dynamic changes of movements through temporal features to obtain spatiotemporal features; the output layer converts the spatiotemporal features extracted by the encoder into probability distributions of different swimming stroke categories through MoViNetHead to achieve accurate classification.

[0035] According to a second aspect of the technical solution of the present invention, a system for real-time swimming stroke recognition of swimmers in an indoor swimming pool environment is provided, the system comprising: a processor and a memory for storing executable instructions; wherein the processor is configured to execute the executable instructions to perform the real-time swimming stroke recognition method for swimmers in an indoor swimming pool environment as described in any of the above aspects.

[0036] According to a third aspect of the technical solution of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the method for real-time swimming posture recognition of swimmers in an indoor swimming pool environment as described in any of the above aspects is implemented.

[0037] Beneficial effects of the present invention

[0038] The present invention provides a method for constructing a swimming stroke database of swimmers in an indoor swimming pool environment and for real-time swimming stroke recognition. By making full use of the spatiotemporal information in video data, the inherent laws and correlations of swimmers' movements are deeply explored to achieve accurate recognition of swimming strokes. Based on the video classification method of YOLOv8 target detection technology and PPTSM_v2, combined with deep learning technology, a model that can accurately identify swimming strokes is constructed. Since the model is trained based on a large amount of real video data, its recognition accuracy and real-time performance are significantly improved. At the same time, this method is not limited by the subjectivity of manual observation and the limitations of wearable devices, and can automatically adapt to the changes in movements of different swimmers. This method can provide technical support for swimming training assistance, athlete movement analysis and swimming pool safety monitoring, and is expected to optimize training effects and improve the technical level and safety of swimmers. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0040] Figure 1 This is a flow chart of the method for constructing a swimming posture database and real-time swimming posture recognition for swimmers in an indoor swimming pool environment according to the present invention;

[0041] Figure 2 This is a rendering of the intelligent interception tool VideoTool in an embodiment of the present invention;

[0042] Figure 3 It is an inference diagram of the swimming style recognition classification model based on PPTSM_v2 in an embodiment of the present invention;

[0043] Figure 4 2 is a schematic diagram of the prediction results of the swimming style recognition classification model in an embodiment of the present invention.

[0044] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0045] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0046] The terms "first," "second," and the like in the description and claims of the present disclosure are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present disclosure described herein can, for example, be implemented in orders other than those illustrated or described herein.

[0047] In addition, the terms "comprises" and "having" and any variations thereof are intended to cover a non-exclusive inclusion. For example, a process, method, system, product or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, product or apparatus.

[0048] Multiple includes two or more.

[0049] It should be understood that the term "and / or" used in this disclosure simply describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. This reduces manpower input and facilitates business automation, offering universal, efficient, and high-precision features.

[0050] The technical solution of the present invention first provides a method for real-time swimming posture recognition of a swimmer in an indoor swimming pool environment, wherein the method comprises the following steps:

[0051] S1: Real-time acquisition of video data, including motion sequence information, body posture information, and temporal dynamic information of the swimmer in different swimming strokes;

[0052] Furthermore, in said S1:

[0053] The motion sequence information includes: the trajectory, frequency and amplitude of the arm stroke, the rhythm, strength and method of the leg kick, the coordination of the body and the breathing action; it is used to capture the typical motion patterns of different swimming styles;

[0054] The body posture information includes: the positions of key body joints, body angles and the overall outline information of the swimmer; it is used to describe the swimmer's static posture and assist in identifying the swimming style;

[0055] The temporal dynamic information includes: periodic changes in stroke, kicking and breathing, changes in the speed of arm stroke and leg kicking, and transition information between different actions; it is used to capture the continuity and dynamic changes of actions.

[0056] S2: extracting video segments from the video data, classifying the video segments and performing frame extraction processing to obtain multiple frames of images, storing the multiple frames of images in chronological order and organizing them into a structured swimming style dataset.

[0057] Furthermore, in S2, the swimming style data set is divided into five categories: breaststroke, freestyle, butterfly stroke, backstroke and uncertain.

[0058] Furthermore, in S2, the steps of constructing the swimming style dataset are as follows:

[0059] S21 video data capture: capturing a video clip of a single person swimming from the video data;

[0060] S22: Classifying and labeling swimming styles: classifying the video clips according to swimming style categories, and labeling the video clips by category;

[0061] S23 video frame extraction processing: performing frame extraction processing on the classified video segments, converting each video segment into multiple frame images, and storing them in a folder corresponding to each video segment in chronological order;

[0062] S24 Dataset organization and storage: organizing the folders into a structured swimming stroke dataset, and extracting all folder paths into a list format as input data for a swimming stroke recognition and classification model;

[0063] S25 Dataset Division and Usage: The structured swimming stroke dataset is divided into a training set, a validation set, and a test set, which are used for swimming stroke recognition and classification model training, tuning, and performance evaluation, respectively.

[0064] Furthermore, in S21, a video capture tool is used to capture a video clip of a single person swimming from the video data, specifically including:

[0065] Drag and draw a rectangular box to select the swimmer's area in real time;

[0066] Dynamically adjust the frame position and size to ensure that the target person is completely contained in the frame;

[0067] After clicking the "Save" button, the video content of the selected area within the current playback time period will be automatically captured, the redundant images outside the frame will be cropped, the high-definition image quality will be retained, and the video will be output as an independent video file.

[0068] Furthermore, if segmented capture is required, the selection range can be adjusted by pausing multiple times, and the current segment can be saved after clicking “Stop” each time. This will eventually generate multiple cropped videos that precisely focus on the swimmer’s movements, providing high-quality input for subsequent analysis.

[0069] Furthermore, in S21, each segment is 3-10 seconds long, ensuring that it contains at least one complete swimming action.

[0070] Furthermore, in said S22, the swimming stroke categories are divided into five categories: breaststroke, freestyle, butterfly stroke, backstroke and uncertain.

[0071] S3: A swimming stroke recognition and classification model is constructed using the YOLOv8 and Recognizer2D model framework, and the structured swimming stroke dataset is used for training, so that the trained swimming stroke recognition and classification model can classify swimming strokes according to action sequence information, body posture information and temporal dynamic information.

[0072] Furthermore, in S3, the YOLOv8 is based on an improved CSPDarknet backbone network and an Anchor-free detection head design, adopts a single-stage detection architecture, and simultaneously detects and distinguishes multiple swimmers in the swimming pool scene through a dynamic label allocation strategy and an enhanced anchor box mechanism. It also stably outputs the precise bounding box and tracking ID of each target, and generates independent motion segments for each target through bounding box detection and temporal association, thereby providing single target input for the subsequent Recognizer2D model.

[0073] Furthermore, in S3, the Recognizer2D model framework selects the PPTSM_v2 model as the backbone network, fuses spatial features and temporal features to generate a comprehensive spatiotemporal feature representation, and uses MoViNetHead as the classification head;

[0074] Among them, the PPTSM_v2 model includes the PP-LCNetV2 model and the TSM (Temporal Shift Module) module.

[0075] Furthermore, the spatial features are extracted by the PP-LCNetV2 model based on the action sequence information and body posture information; and the temporal features are extracted by the TSM module based on the temporal dynamic information.

[0076] Furthermore, in said S3, said swimming style recognition and classification model includes an encoder and an output layer;

[0077] The encoder contains multiple stacked coding layers to capture the spatial features in the video data and understand the dynamic changes of movements through temporal features to obtain spatiotemporal features; the output layer converts the spatiotemporal features extracted by the encoder into probability distributions of different swimming stroke categories through MoViNetHead to achieve accurate classification.

[0078] S4: Inputting the real-time collected video data into the trained swimming style classification model to output the swimming style type.

[0079] The technical solution of the present invention further provides a system for real-time swimming stroke recognition of swimmers in an indoor swimming pool environment, the system comprising: a processor and a memory for storing executable instructions; wherein the processor is configured to execute the executable instructions to perform the real-time swimming stroke recognition method for swimmers in an indoor swimming pool environment as described above.

[0080] The technical solution of the present invention also provides a computer-readable storage medium, characterized in that a computer program is stored thereon, and when the computer program is executed by a processor, the method for real-time swimming posture recognition of swimmers in an indoor swimming pool environment as described above is implemented.

[0081] Example

[0082] This embodiment provides a method for real-time swimming posture recognition of swimmers in an indoor swimming pool environment. Figure 1 As shown, including:

[0083] S101: Video data of people in the swimming pool is collected in real time through the camera to obtain the complete action sequence, body posture and temporal dynamic information of the swimmers in different swimming strokes, which is used as training data for the model.

[0084] The collected video data must include the following information:

[0085] 1) Movement sequence information: This includes the trajectory, frequency, and amplitude of the arm stroke (e.g., alternating strokes in freestyle and symmetrical strokes in breaststroke); the rhythm, force, and pattern of the leg kick (e.g., alternating kicks in freestyle and dolphin kicks in butterfly); body coordination (e.g., rotation, undulation, and balance); and breathing movements (e.g., head rotation and breathing rate). This information is used to capture the typical movement patterns of different swimming styles.

[0086] 2) Body posture information: This includes the positions of key joints such as the shoulders, elbows, wrists, hips, knees, and ankles; body angles such as the angle between the body and the horizontal plane and the angle between the arms and the body; and the swimmer's overall silhouette. This information is used to describe the swimmer's static posture and assist in identifying swimming styles.

[0087] 3) Temporal dynamics: This includes the periodic changes in stroke, kick, and breathing, the speed variations of arm and leg strokes, and the transitions between different actions (such as the transition from stroke to breath). This information captures the continuity and dynamics of the movements, enhancing the model's understanding of temporal features.

[0088] S102: Using the collected video data, a swimming stroke dataset is constructed. The dataset is divided into five categories: breaststroke, freestyle, butterfly stroke, backstroke, and uncertain, which serves as input data for a swimming stroke recognition and classification model.

[0089] The specific steps for constructing the swimming style dataset are as follows:

[0090] 1) Video data capture: Use the video capture software VideoTool to process the collected swimming video data and capture swimming clips of a single person. Each clip is 3-10 seconds long and is guaranteed to contain at least one complete swimming stroke.

[0091] S2: Based on real-time multi-stroke swimming video data, we developed an intelligent capture tool, VideoTool, which supports manually dragging the timeline to accurately capture individual video clips. This tool provides an intelligent capture tool for building high-quality structured datasets and provides high-quality input for subsequent swimming stroke recognition and classification model training.

[0092] The intelligent capture tool VideoTool includes the following functions:

[0093] 1) Video Selection and Target Path Settings: Supports importing a variety of common video formats, automatically detects compatibility, and prompts for unusual formats. Provides a default storage path and supports custom directories, automatically generating subfolders by swimming style tags.

[0094] 2) Playback Control: Provides basic play, pause, stop, and exit functions, supporting fast forward, slow down, and return to original speed. Videos are displayed in their original proportions (no stretching or compression), avoiding motion distortion and improving browsing and capture accuracy. Drag the timeline slider to precisely locate the time, improving video browsing and capture efficiency.

[0095] 3) Capture function: During video playback, you can manually drag and draw a rectangular frame to select the swimmer's area in real time, dynamically adjust the frame position and size to ensure that the target person is completely contained in the frame; after clicking the "Save" button, the video content of the selected area within the current playback time period will be automatically captured, and the redundant images outside the frame will be cropped while retaining the high-definition image quality, and output as an independent video file. If you need to capture in segments, you can pause and adjust the selection range multiple times, and save the current segment after clicking "Stop" each time, and finally generate multiple cropped videos that accurately focus on the swimmer's movements, providing high-quality input for subsequent analysis. The effect of the capture tool is very good. Figure 2 shown.

[0096] In summary, the VideoTool video capture tool accurately captures swimming video clips, ensuring that each clip focuses on the target swimming stroke, eliminating redundant background interference, and achieving high-quality data collection. During the capture process, VideoTool fully preserves the temporal dynamics of swimming strokes, providing a temporal modeling foundation for the Temporal Shift Module (TSM) in PPTSM_v2, enabling the model to accurately capture movement continuity and enhance its ability to learn the temporal characteristics of swimming strokes.

[0097] 2) Swimming style classification and labeling: The captured swimming clips are classified into five categories according to the swimming style: breaststroke, freestyle, butterfly stroke, backstroke, and uncertain. Each video clip is labeled to ensure the accuracy of the category label.

[0098] 3) Video frame extraction: The classified video clips are subjected to frame extraction, each video clip is converted into multiple frames of images, and stored in a separate folder corresponding to each video clip in chronological order.

[0099] 4) Dataset organization and storage: The extracted image folders are organized into a structured dataset, and all folder paths are extracted into a list format as input data for the swimming style recognition and classification model to facilitate subsequent model training and verification.

[0100] 5) Dataset Division and Usage: The organized dataset is divided into training, validation, and test sets, which are used for model training, tuning, and performance evaluation, respectively, to ensure the diversity and representativeness of the dataset, covering different swimming styles, different swimmers, and different environmental conditions.

[0101] S103: Build a swimming style recognition and classification model using YOLOv8 and the Recognizer2D model framework of the PaddlePaddle deep learning framework. Calculate predicted values through forward propagation, compare them with the true labels to calculate the loss function, and use the backpropagation algorithm to update model parameters and optimize model performance.

[0102] When building a swimming style recognition and classification model, the present invention uses the YOLOv8 and Recognizer2D model frameworks, which are suitable for the swimming style classification task in the present invention. YOLOv8 is an efficient target detection model suitable for multi-target scenarios in swimming pools. There may be multiple swimming target objects in the swimming pool. The Recognizer2D model framework can only process swimming video clips of a single target. The YOLOv8 model can accurately detect and track the positions of multiple swimmers in real time, and generates a separate swimming video clip for each target through bounding box detection and temporal association. These clips provide high-quality single-target input for the Recognizer2D model, ensuring the accuracy of swimming style classification.

[0103] YOLOv8 is an efficient target detection model based on an improved CSPDarknet backbone network and an anchor-free detection head design. It can accurately detect multiple targets in images or videos in real time. It adopts an advanced single-stage detection architecture and significantly improves the detection ability of overlapping targets and small-sized targets through a dynamic label allocation strategy and an enhanced anchor box mechanism. The model supports simultaneous detection and differentiation of multiple swimmers in swimming pool scenes, and can stably output accurate bounding boxes and tracking IDs for each target. In multi-target swimming pool scenes, YOLOv8 can quickly locate and track the positions of multiple swimmers, and generate independent motion segments for each target through bounding box detection and temporal association, thereby providing high-quality single-target input for the subsequent Recognizer2D model and ensuring the accuracy of swimming style classification. Its high precision and real-time performance are particularly suitable for motion analysis in multi-person swimming pool environments, providing reliable multi-target detection support for intelligent swimming training.

[0104] Recognizer2D provides a general framework for processing video classification tasks. It extracts the spatial features of video frames through 2D convolutional neural networks and classifies them in combination with temporal information. In the present invention, the Recognizer2D model framework selects PPTSM_v2 as the backbone network, combines the lightweight design of PP-LCNetV2 and the temporal modeling capability of TSM (Temporal Shift Module), and extracts the spatial features and temporal dynamic information of video frames. PPTSM_v2 improves the efficiency and performance of the model through technologies such as deep separable convolution, reparameterization, channel attention mechanism and residual connection. Then, using MoViNetHead as the classification head, the features extracted by the backbone network are mapped to the category space to generate swimming style classification results. The reasoning process of the swimming style recognition classification model based on PPTSM_v2 is as follows: Figure 3 shown.

[0105] TSM is a temporal modeling module that captures temporal dynamic information between video frames through feature shifting operations. It enhances the model's ability to model temporal features without significantly increasing computational overhead. By shifting the features of some channels along the time dimension, information interaction between frames is achieved. It is suitable for tasks that require temporal modeling, such as video classification and action recognition. The traditional TSM model uses ResNet50 as the backbone network, which has some shortcomings. ResNet50 has a large number of parameters, resulting in high computational complexity, slow model inference speed, and poor real-time performance. The TSM module is usually inserted in the latter layers of ResNet50. The shallow features lack effective temporal modeling and it is difficult to capture short-term details of the swimming posture (such as rapid arm swings).

[0106] PP-TSM is optimized based on the ResNet50 backbone network, using the superior ResNet50_vd as the model's backbone. By improving the downsampling structure and convolutional layer design, it enhances feature extraction capabilities, improving model accuracy while maintaining the original number of parameters. PP-TSMv2 inherits some of PP-TSM's optimization strategies, focusing on model tuning in areas such as the backbone network, TSM module tuning, and input frame optimization. To accelerate the model, PP-LCNetV2 removes some residual connections and uses depthwise separable convolution to reduce computation and parameter count. Reparameterization technology is introduced to improve model inference efficiency. Support for the channel attention mechanism (SE module) enhances the model's focus on key features. Residual connections (shortcuts) mitigate the vanishing gradient problem and enhance the model's expressiveness. Compared with the traditional TSM and ResNet50 combination, the PP-TSMv2 model is more suitable for swimming stroke recognition tasks. Combined with a lightweight backbone network, it improves the model's computational efficiency and better meets the real-time requirements of swimming stroke recognition.

[0107] When constructing a swimming style recognition and classification model, this paper adopts the Recognizer2D model framework and uses existing datasets to train and test the TSM, PP-TSM, and PP-TSMv2 models. The specific configurations and results are shown in Table 1. The experimental results show that the TSM model has low test accuracy and a long average inference time. The PP-TSM model has greatly improved in test accuracy and is more suitable for swimming style recognition tasks, but the average inference time has not been improved. The PP-TSMv2 model has achieved improvements in both test accuracy and efficiency thanks to its innovative lightweight design and timing modeling optimization.

[0108] Table 1

[0109]

[0110] The core of the swimming style recognition function lies in using the PPTSM_v2 model to capture spatiotemporal features in video data. During this process, YOLOv8 first performs real-time multi-target detection and tracking in the pool scene, accurately locating each swimmer's position and generating independent motion trajectory segments. This provides the PPTSM_v2 model with high-quality single-target time series data consisting of consecutive video frames. The PPTSM_v2 model combines the lightweight design of PP-LCNetV2 with the temporal modeling capabilities of the TSM (Temporal Shift Module) to extract spatial features and temporal dynamic information from video frames. Specifically, the PPTSM_v2 model extracts temporal information through feature shifting operations and combines it with a convolutional neural network to extract spatial features (such as body posture and joint positions). This integration of spatiotemporal information enables accurate classification of swimming styles. The input data includes the swimmer's motion sequence (e.g., arm stroke, leg kick), body posture information (e.g., joint positions, body angles), and temporal dynamic information (e.g., motion period, velocity variation). After preprocessing, this data is input into the PPTSM_v2 model in time series order. The model captures the continuity and dynamic changes of movements by performing spatiotemporal modeling of these features, and ultimately outputs the swimming style type (such as freestyle, breaststroke, backstroke, and butterfly stroke).

[0111] The specific steps for swimming style recognition and classification modeling are as follows:

[0112] 1) Data Collection: Use cameras to collect videos of people swimming in the pool, ensuring coverage of various swimming styles and different perspectives to provide basic data for model training.

[0113] 2) Data Preprocessing: First, we use YOLOv8 to perform multi-target detection and tracking on the original video, generating independent motion trajectory segments for each swimmer. We then perform keyframe sampling, normalization, resizing, data augmentation, and temporal segmentation on the extracted single-person video data to convert the original video into a format suitable for model input.

[0114] 3) Model Construction: Build a swimming style classification model, consisting of an encoder and an output layer. The encoder includes multiple stacked encoding layers that combine PPTSM_v2 with 2D convolutional layers. This allows the encoder to capture spatial features in the video while understanding the dynamics of the movement through temporal information exchange. The output layer converts the spatiotemporal features extracted by the encoder into a probability distribution of different swimming style categories, enabling accurate classification.

[0115] 4) Model Training: The model is trained using the training dataset. During training, the predicted values are calculated through forward propagation and compared with the true labels to calculate the loss function. The model parameters are then updated using the backpropagation algorithm to minimize the loss function and optimize model performance.

[0116] 5) Model Evaluation and Optimization: Use an independent test set to evaluate model performance, using metrics such as accuracy, precision, and recall to assess the model's predictive accuracy. Based on the evaluation results, fine-tune the model, including adjusting model parameters and changing the model structure.

[0117] 6) Prediction and Application: Use the trained model to predict swimming styles. In practice, real-time captured and preprocessed video clips are fed into the trained model to extract spatiotemporal features for accurate swimming style classification. A visual interface displays the swimming style classification results in real time, allowing coaches and users to instantly review and analyze them, supporting personalized training plan development and technical improvement guidance.

[0118] In summary, the swimming style recognition method based on YOLOv8 and PP-TSMv2 achieves accurate classification of swimming styles by capturing spatial features and temporal dynamics in video data. This modeling approach combines convolutional neural networks from deep learning with temporal modeling techniques. The entire modeling and classification process is completed through steps including data acquisition, preprocessing, model construction, training, evaluation and optimization, and real-time application. The swimming style classification model can monitor and analyze swimmers' movements in real time, providing efficient and accurate technical support for swimming training assistance and pool safety monitoring.

[0119] S104: Input the video data collected by the camera in real time into the trained swimming style classification model, extract the temporal features and spatial features, fuse them to achieve accurate swimming style classification, and output the swimming style type.

[0120] Temporal features are dynamic information extracted from video frames to describe the continuity and variation of a swimmer's movements. These include movement cycles (such as the periodic changes in arm stroke, leg kick, and breathing), movement speeds (such as the speed changes of arm stroke and leg kick), movement transitions (such as the transition from stroke to breathing), and dynamic changes (such as body rotation, undulation, and changes in balance). These features are extracted through the Temporal Series Modeling Module (TSM), which captures the dynamic relationships between frames and provides temporal information support for swimming style classification.

[0121] Spatial features are static information extracted from a single image frame to describe the swimmer's body posture and local movements. They primarily include body posture (such as joint positions and body angles), local movements (such as the static posture of arm strokes and leg kicks), and body silhouette (the swimmer's overall outline). These features are extracted using a 2D convolutional neural network (CNN), which captures detailed information within a single frame and provides spatial information support for swimming style classification.

[0122] The extracted spatial and temporal features are fused to generate a comprehensive spatiotemporal feature representation. The fused features can simultaneously describe the swimmer's static posture and dynamic movements, providing comprehensive information support for swimming style classification.

[0123] S105: The swimming style classification results are displayed in real time through a visual interface, so as to facilitate real-time viewing and analysis by the coach or user.

[0124] Here, the predicted swimming stroke categories are divided into five categories: breaststroke, freestyle, butterfly stroke, backstroke and uncertain, which are represented by breaststroke, freestyle, butterfly_stroke, backstroke and Uncertain respectively. The model prediction results of this embodiment are as follows Figure 4 As shown, the swimming stroke types of swimmers in an indoor pool environment can be predicted in real time, and the results of the swimming stroke classification are displayed in real time through a visual interface, making it easy for coaches or users to view and analyze them immediately. Specifically, all people are identified, drawn with green boxes, and labeled with their IDs. In addition to the person ID, people in the pool are also labeled with tags such as "person act:butterfly_stroke" to indicate butterfly stroke, "person act:backstroke" to indicate breaststroke, and "person act:Uncertain" to indicate an uncertain type.

[0125] In summary, the present invention uses a camera to collect video data of people in the swimming pool in real time, obtains the complete action sequence, body posture and temporal dynamic information of the swimmer in different swimming strokes, and uses it as training data for the model; uses the collected video data to construct a swimming stroke data set, which is divided into five categories: breaststroke, freestyle, butterfly stroke, backstroke and uncertain, and serves as input data for a swimming stroke recognition and classification model; constructs a swimming stroke recognition and classification model, and adopts the Recognizer2D model framework of the YOLOv8 and PaddlePaddle deep learning framework. The predicted value is calculated by forward propagation, and the loss function is calculated by comparing it with the real label. The model parameters are updated using the backpropagation algorithm to optimize the model performance. The video data collected by the camera in real time is input into the trained swimming stroke classification model, and the temporal features and spatial features are extracted. After fusion, accurate classification of swimming strokes is achieved, and the swimming stroke type is output; the swimming stroke classification results are displayed in real time through a visual interface, which is convenient for coaches or users to view and analyze immediately.

[0126] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0127] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0128] Through the description of the above embodiments, those skilled in the art can clearly understand that the above implementation method can be implemented by means of software plus the necessary general hardware platform, or of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0129] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are protected by the present invention.

Claims

1. A method for real-time swimming stroke recognition of swimmers in an indoor swimming pool environment, characterized in that: The method comprises the following steps: S1: Real-time acquisition of video data, including motion sequence information, body posture information, and temporal dynamic information of the swimmer in different swimming strokes; S2: extracting video clips from the video data, classifying the video clips and performing frame extraction processing to obtain multiple frames of images, storing the multiple frames of images in chronological order and organizing them into a structured swimming style dataset; S3: constructing a swimming stroke recognition and classification model using the YOLOv8 and Recognizer2D model framework, and using the structured swimming stroke dataset for training, so that the trained swimming stroke recognition and classification model can classify swimming strokes based on action sequence information, body posture information, and temporal dynamic information; S4: Inputting the real-time collected video data into the trained swimming style classification model to output the swimming style type.

2. The method according to claim 1, characterized in that In S1: The motion sequence information includes: the trajectory, frequency and amplitude of the arm stroke, the rhythm, strength and method of the leg kick, the coordination of the body and the breathing action; it is used to capture the typical motion patterns of different swimming styles; The body posture information includes: the positions of key body joints, body angles and the overall outline information of the swimmer; it is used to describe the swimmer's static posture and assist in identifying the swimming style; The temporal dynamic information includes: periodic changes in stroke, kicking and breathing, changes in the speed of arm stroke and leg kicking, and transition information between different actions; it is used to capture the continuity and dynamic changes of actions.

3. The method according to claim 1, characterized in that In S2, the steps for constructing the swimming style dataset are as follows: S21 video data capture: capturing a video clip of a single person swimming from the video data; S22: Classifying and labeling swimming styles: classifying the video clips according to swimming style categories, and labeling the video clips by category; S23 video frame extraction processing: performing frame extraction processing on the classified video segments, converting each video segment into multiple frame images, and storing them in a folder corresponding to each video segment in chronological order; S24 Dataset organization and storage: organizing the folders into a structured swimming stroke dataset, and extracting all folder paths into a list format as input data for a swimming stroke recognition and classification model; S25 Dataset Division and Usage: The structured swimming stroke dataset is divided into a training set, a validation set, and a test set, which are used for swimming stroke recognition and classification model training, tuning, and performance evaluation, respectively.

4. The method according to claim 1, wherein In S2, the swimming style data set is divided into five categories: breaststroke, freestyle, butterfly stroke, backstroke and uncertain.

5. The method according to claim 1, characterized in that In S3, the YOLOv8 model simultaneously detects and distinguishes multiple swimmers in the swimming pool scene, and stably outputs the precise bounding box and tracking ID of each target. Through bounding box detection and temporal association, an independent motion segment is generated for each target, thereby providing single target input for the subsequent Recognizer2D model.

6. The method according to claim 1, characterized in that In S3, the Recognizer2D model framework selects the PPTSM_v2 model as the backbone network, fuses spatial features and temporal features to generate a comprehensive spatiotemporal feature representation, and uses MoViNetHead as the classification head; Among them, the PPTSM_v2 model includes the PP-LCNetV2 model and the TSM (Temporal Shift Module) module.

7. The method according to claim 6, characterized in that The spatial features are extracted by the PP-LCNetV2 model based on the action sequence information and body posture information; the temporal features are extracted by the TSM module based on the temporal dynamic information.

8. The method according to claim 6, characterized in that In S3, the swimming style recognition and classification model includes an encoder and an output layer; The encoder contains multiple stacked coding layers to capture the spatial features in the video data and understand the dynamic changes of movements through temporal features to obtain spatiotemporal features; the output layer converts the spatiotemporal features extracted by the encoder into probability distributions of different swimming stroke categories through MoViNetHead to achieve accurate classification.

9. A real-time swimming stroke recognition system for swimmers in an indoor swimming pool environment, the system comprising: A processor and a memory for storing executable instructions; characterized in that the processor is configured to execute the executable instructions to perform the real-time swimming stroke recognition method for swimmers in an indoor swimming pool environment according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for real-time swimming stroke recognition of a swimmer in an indoor swimming pool environment according to any one of claims 1 to 8 is implemented.

Images