A non-contact athlete physiological state evaluation method and system based on feature fusion and a medium

Abstract

The application discloses a feature fusion-based non-contact athlete physiological state evaluation method and system and a medium, relates to multiple technical fields of computer vision, image processing and physiological signal detection, and comprises the following steps: S1, constructing an rPPG signal feature extraction model, extracting rPPG signal features from the preprocessed video through the rPPG signal feature extraction model; S2, constructing an information and action feature extraction model and training, and extracting information features and action features of athletes from the preprocessed video, fusing the information features and the action features to obtain motion information features; S3, constructing a multi-source signal fusion network based on a deep learning model, taking the obtained rPPG signal features and motion information features as inputs, and continuously predicting the fatigue degree of the athletes. The application effectively extracts rPPG signals and biomechanical parameter features, combines feature coding and feature fusion technology, and realizes prediction of the fatigue degree of the athletes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical fields of computer vision, image processing and physiological signal detection, and particularly relates to a non-contact athlete physiological state evaluation method and system based on feature fusion and a medium. BACKGROUND

[0002] rPPG technology, i.e. remote photoplethysmography, is a technology that captures the periodic changes in skin color caused by the heart cycle through sensors such as cameras. By capturing these tiny color changes, the blood volume pulse signal (BVP signal) can be extracted, and then the heart rate (HR), respiratory rate (RR), and heart rate variability (HRV) and other heart cycle-related physiological indicators can be measured. rPPG technology plays a very important role in scenarios that require non-contact physiological parameter monitoring.

[0003] Heart rate, heart rate variability, and blood oxygen saturation are important indicators of autonomic nervous system activity. By analyzing the time and frequency domain features of rPPG signals, the trends of core indicators such as heart rate (HR), heart rate variability (HRV), and blood oxygen saturation (SpO2) can be evaluated, and the individual's stress response and health status can be understood. In the video signal, the corresponding exercise load (cumulative sprint distance, average speed, etc.) and motion characteristics (number of changes in direction, jumping height, etc.) parameters can be inferred, which are beneficial for us to evaluate the fatigue degree of athletes. For example, when athletes are tired, their physical condition will show heavy breathing, increased heart rate, and slower speed, etc. Therefore, the fusion of the above multiple indicators can obtain more accurate and comprehensive information. The multi-modal fusion analysis method not only improves the accuracy and reliability of the athlete activity evaluation, but also provides strong support for subsequent health management.

[0004] Currently, basketball teams lack real-time fatigue monitoring tools for athletes during games, and mainly rely on wearable devices (such as chest straps or wristwatches) to monitor heart rate and HRV. Changes in low-frequency / high-frequency heart rate variability can reflect the fatigue state of the autonomic nervous system (such as an increase in sympathetic nervous activity), which can cause discomfort for some athletes and cannot be accurately worn by every athlete, resulting in the inability of the coaching staff to obtain data for some athletes. SUMMARY

[0005] The present application aims to overcome the shortcomings of the prior art and provide a non-contact athlete physiological state evaluation method and system based on feature fusion and a medium.

[0006] The object of the application is achieved by the following technical solutions:

[0007] In a first aspect, the application discloses a non-contact athlete physiological state evaluation method based on feature fusion, comprising the following steps:

[0008] S1, collect the video of the athlete's movement and preprocess it; construct an rPPG signal feature extraction model, use domain adaptation as an unsupervised learning scheme to train the rPPG signal feature extraction model; use the trained rPPG signal feature extraction model to extract rPPG signal features from the preprocessed video, including extracting local spatiotemporal features by spatiotemporal feature extraction 3D-CNN, and combining Transform modeling long time series dependence to obtain rPPG signal features;

[0009] S2, construct and train an information and action feature extraction model, extract the information features and action features of the athlete from the preprocessed video through the trained information and action feature extraction model, fuse the information features and action features, and obtain the movement information features;

[0010] S3, construct a multi-source signal fusion network based on a deep learning model, take the obtained rPPG signal features and movement information features as inputs, and continuously predict the fatigue degree of the athlete.

[0011] Based on the first aspect, step S1 specifically comprises the following steps:

[0012] S11, preprocess the video ; wherein T represents the time step, and HxW represents the spatial resolution, specifically including: converting the RGB color channel of the video to the YCbCr color channel, and the mixing relationship of the rPPG signal in the YCbCr color channel is: ; wherein represents the blue difference change amount, represents the red difference change amount, Motion ( t ) represents the noise signal caused by movement, and alpha and gamma represent the pulse signal gain, and beta and delta represent the movement interference coefficient; then input into the motion compensation to eliminate the interference of the motion artifact on the rPPG signal; wherein the formula of the motion compensation is: , represents t the skin region pixel intensity signal extracted from the video in the RGB color channel at time t, represents the motion displacement gradient signal calculated by the optical flow method, represents the signal after eliminating the motion artifact; obtain the preprocessed video ;

[0013] S12, the spatio-temporal feature extraction 3D-CNN processes the pre-processed video by a 3D convolution kernel on the pre-processed video , wherein represents the input channel, h x w represents the spatial resolution, represents the output channel, and the spatio-temporal feature extraction output is obtained : , represents the time step, represents the spatial resolution, and RELU represents the activation function, and BatchNorm represents the batch normalization;

[0014] S13, the spatio-temporal feature extraction output is flattened into a time sequence feature sequence and position encoding is added , ; wherein represents the time, , represents flattening the spatio-temporal feature;

[0015] S14, the global dependence is captured by the multi-head self-attention MSA and the layer normalization LN of the encoder Transformer: , ; wherein represents the output feature of the Transformer model at the layer, FFN represents the feedforward network, l is the layer index; and the rPPG signal feature is obtained .

[0016] Based on the first aspect, step S2 specifically comprises:

[0017] S21, the detection frame is output by the YOLOv8s model , wherein is the left upper corner coordinate of the detection frame, is the right lower corner coordinate of the detection frame, is the player category confidence; the size of the anchor box is redesigned using K-Means clustering to adapt to the body shape of the basketball player, that is, the average width-height ratio is 0.4:1;

[0018] S22, the position sensitivity of the detection frame is enhanced by a loss function , wherein represents the classification loss, represents the weight coefficient of the classification loss, represents the boundary box regression loss, a weight coefficient representing a bounding box regression loss, wherein the bounding box regression loss is calculated by wherein represents the Euclidean distance, represents the overlap ratio of the predicted box and the real box, c represents the diagonal line of the minimum enclosing box, v represents the aspect ratio consistency term, represents the predicted box, represents the real box, represents the weight coefficient;

[0019] S23, track the athletes using the DeepSort algorithm, extract the information features and action features of the athletes from the preprocessed video through the trained information and action feature extraction model, and predict the next frame state of the athletes through Kalman filtering wherein F represents a state transition matrix, represents a state vector, , x’ represents the horizontal coordinate of the center point, y represents the vertical coordinate of the center point, w represents the width of the bounding box, and h’ represents the height of the bounding box, represents the horizontal velocity, represents the vertical velocity;

[0020] S24, match the detection box and the tracking track through the Hungarian algorithm, the cost matrix C includes the Mahalanobis distance and the appearance feature cosine similarity ; the calculation formula of the Mahalanobis distance is wherein S represents the covariance matrix of Kalman filtering, represents a track, T1 represents a transpose, and the calculation formula of the appearance feature cosine similarity is wherein is the i-th appearance feature vector, is the appearance feature mean value of the j-th track;

[0021] S25, track the athlete information by integrating the cost , wherein is the weight of the motion cost, is the weight of the appearance cost; obtain the information features, and the athlete information includes the athlete running distance, the athlete foot center point, the athlete speed, and the actual speed;

[0022] S26, estimate the joint coordinates of AlphaPose based on the posture in the tracking box wherein k is the index number of the joint, and the action features are calculated: , fuse information features and action features, and output the extracted motion information features .

[0023] Based on the first aspect, step S3 specifically includes the following steps:

[0024] S31, extracting the rPPG signal features and motion information features Input into the multi-source signal fusion network, first extract the local spatiotemporal information through the 3DCNN network, then input the local spatiotemporal information into the LSTM model to extract the temporal features, and then extract the temporal features through the rPPG signal features. Get the first output feature , through motion information features Get the second output feature ;

[0025] S32, the first output feature Perform short-time Fourier transform (STFT) to obtain the spectrum , and then perform inverse transformation back to the time domain feature to obtain the first enhanced feature ;

[0026] S33, from the second output feature Analyze joint coordinates ,in Represent feature dimensions and calculate joint importance weights , and then the second enhanced feature is obtained by weighted aggregation ;

[0027] S34, the first enhancement feature and the second enhancement feature Fusion , obtain fusion features , the fusion features Input into the fatigue prediction model to obtain the relevant features of the athlete's fatigue level, including signal predicted heart rate HR, heart rate variability HRV, blood oxygen saturation SpO2, exercise load and exercise characteristics. The neuromuscular fatigue index NFI index is calculated based on the relevant features of the athlete's fatigue level, that is, ;in Indicates other, represents the weight of the signal predicting heart rate HR, Indicates the ratio of low frequency to high frequency of heart rate variability The weight of Indicates the weight of the exercise load, represents the weight of motion features, Indicates the weight of blood oxygen saturation SpO2, Indicates other The weight of the NFI index is used to predict the fatigue degree of the athlete.

[0028] In a second aspect, the present application discloses a non-contact athlete physiological state evaluation system based on feature fusion, which is used for the non-contact athlete physiological state evaluation method based on feature fusion.

[0029] The 3DCNN Transform encoder-based rPPG signal extraction module is used to construct an rPPG signal feature extraction model, and domain adaptation is used as an unsupervised learning scheme to train the rPPG signal feature extraction model; the trained rPPG signal feature extraction model is used to extract rPPG signal features from the preprocessed video.

[0030] The YOLO DeepSort 3DCNN-based player information extraction module is used to extract information features and action features of athletes from the preprocessed video through the trained information and action feature extraction model, fuse the information features and the action features, and obtain motion information features.

[0031] The continuity-based body state evaluation module based on fused signals is used to construct a multi-source signal fusion network based on a deep learning model, take the obtained rPPG signal features and motion information features as inputs, and continuously predict the fatigue degree of the athlete.

[0032] In a third aspect, the present application discloses a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the non-contact athlete physiological state evaluation method based on feature fusion.

[0033] The present application has the following advantages:

[0034] 1) The present application innovatively proposes a non-contact body state evaluation technology combining remote photoplethysmography rPPG signal to predict core indicators such as heart rate (HR), heart rate variability (HRV), and blood oxygen saturation (SpO2) and fuse with sports biomechanical parameters; by analyzing video clips containing basketball games, the rPPG signal, the exercise load, and the exercise characteristics of the athletes are extracted, respectively, to realize feature fusion of different modal information, and then a multi-source signal fusion network of a designed deep learning model is used to continuously and real-time evaluate the fatigue degree of the athletes. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The present application discloses a camera placement position schematic diagram;

[0036] Figure 2 A step flowchart of an embodiment of the present application;

[0037] Figure 3 A positioning athlete joint node flowchart of an embodiment of the present application;

[0038] Figure 4 A flowchart of an embodiment of the present application based on YOLO DeepSort 3DCNN to extract athlete information;

[0039] Figure 5 A flowchart of an embodiment of the present application based on 3D-CNN Transform to extract rPPG signals;

[0040] Figure 6 A flowchart of an embodiment of the present application for continuous prediction of the fatigue degree of an athlete by fusing signal input into a multi-source signal fusion network. DETAILED DESCRIPTION

[0041] The technical solutions of the present application will be described in detail below with reference to the embodiments, obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0042] The present application discloses a non-contact athlete physiological state evaluation method based on feature fusion, which combines remote photoplethysmography (rPPG, Remote Photoplethysmography) signal to predict core indicators such as heart rate (HR), heart rate variability (HRV), and blood oxygen saturation (SpO2), and fuses non-contact body state evaluation technology of sports biomechanical parameters. By analyzing the video clips containing basketball games, the rPPG signals, exercise loads, and exercise characteristics of athletes are extracted, and the feature fusion of different modal information is realized. In order to further improve the fusion efficiency and effect, a deep regression model based on multi-source signal fusion is designed, which can continuously and real-time evaluate the activity of athletes, and the step flowchart is as shown in Figure 2 The method specifically comprises the following steps:

[0043] S1, collect the video of the athlete's movement and pre-process; construct a rPPG signal feature extraction model, adopt domain adaptation as an unsupervised learning scheme to train the rPPG signal feature extraction model; use the trained rPPG signal feature extraction model to extract rPPG signal features from the pre-processed video, including extracting local spatio-temporal features by spatio-temporal feature extraction 3D-CNN, and combining Transform to model long time series dependence to obtain rPPG signal features;

[0044] Exemplarily, corresponding cameras are placed at corresponding positions of the basketball court, which is based on specific real work conditions and refers to the current mainstream camera placement positions. The camera placement position diagram is shown as Figure 1 The specific operation is as follows: above the center line of the audience stand, shoot the whole field from the top, capture the overall competition, and clearly observe the running data of the athletes; located at the edge of the audience area, about 4 meters high, focus on shooting the details of the athletes' confrontation, to capture the athletes' action information such as breakthrough, shooting and the like; the cameras placed above the basket, behind the basketball stand and on both sides of the center line are used to identify the rppg signals of the athletes' faces and necks in real time.

[0045] S2, construct and train an information and action feature extraction model, extract the information features and action features of the athletes from the pre-processed video through the trained information and action feature extraction model, fuse the information features and action features, and obtain the movement information features;

[0046] S3, construct a multi-source signal fusion network based on a deep learning model, take the obtained rPPG signal features and movement information features as inputs, and continuously predict the fatigue degree of the athletes.

[0047] Specifically, the rPPG signal extraction process based on 3D-CNN Transform is shown in the diagram Figure 5 as shown, step S1 specifically includes the following steps:

[0048] S11, pre-process the video , including color channel conversion and motion compensation, convert the color channel of the video to YCbCr channel, the signal-to-noise ratio (Signal-to-Noise Ratio) of YCbCr space is significantly improved in the motion scene, especially the robustness of Cr channel to light change, wherein T represents the time step, HxW represents the spatial resolution, specifically including: converting the RGB color channel of the video to the YCbCr color channel, the mixing relationship of the rPPG signal in the YCbCr color channel is: ; wherein represents the blue difference change amount, represents the red difference change amount, Motion ​t ) represents the noise signal caused by motion, and α and γ represent the pulse signal gain, and β and δ represent the motion interference coefficient; then input to motion compensation to eliminate the interference of motion artifacts on the rPPG signal; wherein the formula of motion compensation is: , represents t the pixel intensity signal of the skin region extracted from the RGB color channel of the video at time t, represents the signal after removing motion artifacts, represents the motion displacement gradient signal calculated by the optical flow method, which is used to represent the pixel-level displacement change of the athlete's face or neck skin in the motion (such as skin deformation caused by head shaking and neck muscle stretching); obtain the preprocessed video ;

[0049] S12, spatio-temporal feature extraction 3D-CNN processes the preprocessed video , that is, through 3D convolution kernel process the preprocessed video , wherein represents the input channel, and h x w represents the spatial resolution, represents the output channel, and obtains the spatio-temporal feature extraction output : , represents the time step, represents the spatial resolution, and RELU represents the activation function, and BatchNorm represents the batch normalization;

[0050] S13, flatten the spatio-temporal feature extraction output into a time sequence feature sequence and add position encoding , ; wherein represents the time t, , represents flattening the spatio-temporal feature;

[0051] S14, capture global dependencies through the multi-head self-attention MSA and layer normalization LN of the encoder Transformer: , ; wherein represents the output feature of the Transformer model at the layer, FFN represents the feedforward network, l is the layer index; obtain the rPPG signal feature .

[0052] Specifically, the athlete information flow process diagram based on YOLO DeepSort 3DCNN is as follows:Figure 4 As shown, step S2 specifically includes:

[0053] S21, detecting the bounding box through the You Only Look Once version 8 small (YOLOv8s) model output , wherein is the upper left corner coordinate of the bounding box, is the lower right corner coordinate of the bounding box, is the player category confidence; the anchor box size is redesigned using K-Means clustering to adapt to the body shape of basketball players, that is, the average width-height ratio is 0.4:1;

[0054] S22, enhancing the position sensitivity of the bounding box through the loss function , wherein represents the classification loss, represents the weight coefficient of the classification loss, represents the bounding box regression loss, represents the weight coefficient of the bounding box regression loss, wherein the bounding box regression loss is calculated by the formula , wherein represents the Euclidean distance, represents the overlap ratio of the predicted box and the real box, c represents the diagonal line of the minimum enclosing box, v represents the aspect ratio consistency term, represents the predicted box, represents the real box, represents the weight coefficient, the bounding box is the observation result of the current frame, and the predicted box is the time sequence prediction of the previous frame trajectory;

[0055] S23, tracking the players using the DeepSort algorithm, extracting the information features and action features of the players from the preprocessed video through the trained information and action feature extraction model, and predicting the next frame state of the players through Kalman filtering , wherein F represents a state transition matrix, represents a state vector, , x’ represents the horizontal coordinate of the center point, y represents the vertical coordinate of the center point, w represents the width of the bounding box, and h’ represents the height of the bounding box, represents the horizontal velocity, represents the vertical velocity;

[0056] S24, matching the bounding box with the tracking trajectory through the Hungarian algorithm, and the cost matrix C includes the Mahalanobis distance (motion consistency) and the appearance feature cosine similarity ; the Mahalanobis distance The calculation formula of S is wherein S represents the covariance matrix of Kalman filtering, represents the trajectory, T1 represents the transpose, and the appearance feature cosine similarity The calculation formula of S is wherein is the i-th appearance feature vector, is the appearance feature mean value of the j-th trajectory;

[0057] S25, by synthesizing the cost track the athlete information, wherein is the weight of the motion cost, is the weight of the appearance cost, and by default =0.7, that is, more trust in the motion information (Mahalanobis distance); obtain the information feature, and the athlete information includes the athlete running distance, the athlete foot center point, the athlete speed and the actual speed; wherein the athlete foot center point The calculation formula of S is , ; the calculation formula of the athlete speed is ; the actual speed is converted by camera calibration , and the calculation formula is , and the unit is m / s;

[0058] S26, estimate the joint coordinates of AlphaPose based on the pose estimation in the tracking frame wherein k is the index number of the joint, the action feature is calculated: , the information feature and the action feature are fused, and the extracted motion information feature is output .

[0059] For example, NAS-HP²Net is a joint human parsing and pose estimation network based on neural architecture search. AlphaPose and NAS-HP²Net complement each other in function, making NAS-HP²Net more powerful in pose estimation and realizing the coordinated optimization of human parsing and pose estimation tasks. Its core design focuses on task-specific feature extraction and cross-task feature interaction, especially strengthening the precise extraction and semantic association of human joints. The architectural design of NAS-HP²Net revolves around task-specific feature extraction and cross-task feature interaction. It consists of two key parts: encoder-decoder search (Pose Encoder-Pose Decoder) and high-level feature fusion search (High-Level Feature FusionSearch). The input image is passed through two independent task-specific encoder-decoders (pose branch and parsing branch) to extract multi-scale features. Pose branch: The first output main feature ( ) and auxiliary features ( ), the auxiliary feature is the key point heat map after Gaussian blur. Parsing branch: the second output main feature ( ) and edge features ( ), edge features are used to assist in parsing tasks. High-level feature fusion: The pose cell fuses four types of high-level features from two branches: , the parsing cell input is The structure is symmetrical with the posture unit, which strengthens the fusion of analytical features and posture features. By stacking the posture unit and the analytical unit, multi-level interaction of features is achieved, and the fused prediction results are finally output. The flowchart of locating the athlete's joints is shown in the figure. Figure 3 As shown, represents the position feature of the second joint point, represents the position feature of the second joint point, represents the first player's position characteristics, Represents the second player's position characteristics.

[0060] Specifically, the fusion signal is input into the multi-source signal fusion network to continuously predict the athlete's fatigue level. Figure 6 As shown in the figure, a multi-source signal fusion network of a deep learning model is designed to continuously predict the fatigue level of athletes by integrating multi-dimensional information. The input of the model is the rPPG signal feature. and motion information features The two features are first processed through 3D Convolutional Neural Networks to obtain local spatiotemporal features, and then through Long Short-Term Memory (LSTM) networks to extract the temporal information of the features. The features are then enhanced and fused and input into the fatigue prediction model for prediction. Step S3 specifically includes the following steps:

[0061] S31, extracting the rPPG signal features and motion information features The multi-source signal fusion network is input into the multi-source signal fusion network, which consists of two branches. First, the local spatiotemporal information is extracted through the 3DCNN network, and then the local spatiotemporal information is input into the LSTM model to extract the temporal features. Get the first output feature , through motion information features Get the second output feature ;

[0062] S32, the first output feature Perform short-time Fourier transform (STFT) to obtain the spectrum , and then perform inverse transformation back to the time domain feature to obtain the first enhanced feature ;

[0063] S33, from the second output feature Analyze joint coordinates ,in Represent feature dimensions and calculate joint importance weights The MLP (Multi-Layer Perceptron) maps the coordinates of the joints to a high-dimensional space, capturing the semantic information of their motion or posture (such as the degree of elbow bending). Softmax normalization converts the scores output by the MLP into a probability distribution, highlighting key joints (such as the wrist when shooting), and then obtains the second enhanced feature through weighted aggregation. ;

[0064] S34, the first enhancement feature and the second enhancement feature Fusion , obtain fusion features , the fusion features The input is fed into a fatigue prediction model to obtain the related characteristics of the fatigue degree of the athlete, including signal predicted heart rate HR, heart rate variability HRV, blood oxygen saturation SpO2, exercise load and exercise characteristics and other cardiac cycle related physiological indexes, and a neuromuscular fatigue index NFI index is calculated through the related characteristics of the fatigue degree of the athlete, that is ; represents other, represents the weight of the signal predicted heart rate HR, represents the weight of the ratio of low frequency to high frequency of heart rate variability , represents the weight of the exercise load, represents the weight of the exercise characteristics, represents the weight of the blood oxygen saturation SpO2, represents the weight of other ; represents the importance of each part, and is adaptively adjusted during training; the fatigue degree of the athlete is predicted through the NFI index.

[0065] The application discloses a non-contact athlete physiological state evaluation system based on feature fusion, which is used for the non-contact athlete physiological state evaluation method based on feature fusion.

[0066] The rPPG signal module based on the 3DCNN Transform encoder is used for constructing an rPPG signal feature extraction model, and domain adaptation is adopted as an unsupervised learning scheme to train the rPPG signal feature extraction model; the trained rPPG signal feature extraction model is used to extract rPPG signal features from the preprocessed video;

[0067] The player information module based on YOLO DeepSort 3DCNN is used for extracting information features and action features of athletes from the preprocessed video through the trained information and action feature extraction model, fusing the information features and the action features, and obtaining motion information features;

[0068] The continuity body state evaluation module based on the fused signal is used for constructing a multi-source signal fusion network based on a deep learning model, taking the obtained rPPG signal features and motion information features as inputs, and continuously predicting the fatigue degree of the athlete.

[0069] Exemplarily, the present application also provides a computer readable storage medium, which stores computer readable instructions or programs, and the computer readable instructions or programs are executed by a processor to make the computer perform the following operations: the operations include the steps of the feature fusion based non-contact athlete physiological state evaluation method described above, which will not be repeated herein. The storage medium can include, for example, optical discs, hard disks, floppy disks, flash memories, magnetic tapes, etc.

[0070] The above description is only preferred embodiments of the present application, and it should be understood that the present application is not limited to the forms disclosed herein, and should not be considered as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concepts described herein by the above teachings or related art or knowledge. Any modification and change made by those skilled in the art without departing from the spirit and scope of the present application shall be within the protection scope of the claims of the present application.

Claims

1. A non-contact athlete physiological state assessment method based on feature fusion, characterized in that: The following steps are involved: S1. Collect and preprocess videos of athletes in motion; construct an rPPG signal feature extraction model and use domain adaptation as an unsupervised learning scheme to train the rPPG signal feature extraction model; use the trained rPPG signal feature extraction model to extract rPPG signal features from the preprocessed video, including extracting local spatiotemporal features through spatiotemporal feature extraction 3D-CNN and combining it with Transform modeling to model long-term temporal dependencies to obtain rPPG signal features; S2. Build and train an information and motion feature extraction model, extract information features and motion features of athletes from the preprocessed video using the trained information and motion feature extraction model, and fuse the information features and motion features to obtain motion information features; S3. Construct a multi-source signal fusion network based on a deep learning model, use the acquired rPPG signal features and motion information features as input, and make continuous predictions on the athlete's fatigue level.

2. A non-contact athlete physiological state assessment method based on feature fusion according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11. Video Where T represents the time step and H×W represents the spatial resolution. Specifically, the RGB color channels of the video are converted into YCbCr color channels. The mixing relationship of the rPPG signal in the YCbCr color channels is: ;in represents the blue difference change, represents the red difference change, Motion ( t ) represents the noise signal caused by motion, α and γ represent the pulse signal gain, and β and δ represent the motion interference coefficient; they are then input into the motion compensation to eliminate the interference of motion artifacts on the rPPG signal; the motion compensation formula is: , express t The skin area pixel intensity signal extracted from the RGB color channel video at each moment, Represents the motion displacement gradient signal calculated by the optical flow method, Represents the signal after eliminating motion artifacts; obtain the preprocessed video ; S12, spatiotemporal feature extraction 3D-CNN for preprocessed video Processing, that is, through the 3D convolution kernel After preprocessing, the video Processing, where represents the input channel, h×w represents the spatial resolution, Represents the output channel, and obtains the spatiotemporal feature extraction output : , represents the time step, Represents spatial resolution, RELU represents activation function, and BatchNorm represents batch normalization; S13, extract and output spatiotemporal features Flattened into a time series feature sequence And add position code , ;in Indicates the moment, , Indicates flattening of spatiotemporal features; S14. Capturing global dependencies through the encoder Transformer’s multi-head self-attention MSA and layer normalization LN: , ;in Represents the Transformer model The output features of the layer, FFN represents the feed-forward network, l is the layer index; obtain rPPG signal features .

3. A non-contact athlete physiological state assessment method based on feature fusion according to claim 2, characterized in that: Step S2 specifically includes: S21. Output detection box through YOLOv8s model ,in( ) is the coordinate of the upper left corner of the detection frame, ( ) is the coordinate of the lower right corner of the detection box, The confidence level of the athlete category is calculated; K-Means clustering is used to redesign the size of the anchor box to adapt to the body shape of basketball players, that is, the average aspect ratio is 0.4:1; S22. Through loss function The position sensitivity of the detection frame is enhanced, where represents the classification loss, represents the weight coefficient of classification loss, represents the bounding box regression loss, Represents the weight coefficient of the bounding box regression loss, where the bounding box regression loss The calculation formula is ,in represents the Euclidean distance, Indicates the overlap ratio between the predicted box and the true box, c represents the minimum bounding box diagonal, v represents the aspect ratio consistency term, represents the prediction box, represents the true frame, represents the weight coefficient; S23. Use DeepSort algorithm to track athletes, extract information features and motion features of athletes from pre-processed videos through trained information and motion feature extraction models, and predict the next frame state of athletes through Kalman filtering. , where F represents the state transfer matrix, Represents the state vector, x=(x',y,w,h', , ), x' represents the horizontal coordinate of the center point, y represents the vertical coordinate of the center point, w represents the width of the bounding box, and h' represents the height of the bounding box. represents the horizontal velocity, Indicates vertical speed; S24, match the detection box and tracking trajectory through the Hungarian algorithm, the cost matrix C includes the Mahalanobis distance and cosine similarity of appearance features ; Mahalanobis distance The calculation formula is , where S represents the covariance matrix of the Kalman filter, represents the trajectory, T1 represents the transposition, and the cosine similarity of the appearance feature The calculation formula is ,in is the i-th appearance feature vector, is the mean of the appearance features of the j-th trajectory; S25, through comprehensive cost Tracking of athlete information, including is the weight of the motion cost, is the weight of the appearance cost; obtain information features , the athlete information includes the athlete's running distance, the athlete's foot center point, the athlete's speed and actual speed; S26, AlphaPose joint coordinates based on the pose estimation in the tracking frame , where k is the index number of the joint point, calculate the action features: , fuse information features and action features, and output the extracted motion information features .

4. A non-contact athlete physiological state assessment method based on feature fusion according to claim 3, characterized in that: Step S3 specifically includes the following steps: S31, extracting the rPPG signal features and motion information features Input into the multi-source signal fusion network, first extract the local spatiotemporal information through the 3DCNN network, then input the local spatiotemporal information into the LSTM model to extract the temporal features, and then extract the temporal features through the rPPG signal features. Get the first output feature , through motion information features Get the second output feature ; S32, the first output feature Perform short-time Fourier transform (STFT) to obtain the spectrum Then perform inverse transformation back to the time domain feature to obtain the first enhanced feature ; S33, from the second output feature Analyze joint coordinates ,in Represent feature dimensions and calculate joint importance weights , and then the second enhanced feature is obtained by weighted aggregation ; S34, the first enhancement feature and the second enhancement feature Fusion , obtain fusion features , the fusion features Input into the fatigue prediction model to obtain the relevant features of the athlete's fatigue level, including signal predicted heart rate HR, heart rate variability HRV, blood oxygen saturation SpO2, exercise load and exercise characteristics. The neuromuscular fatigue index NFI index is calculated based on the relevant features of the athlete's fatigue level, that is, ;in Indicates other, represents the weight of the signal predicting heart rate HR, Indicates the ratio of low frequency to high frequency of heart rate variability The weight of Indicates the weight of the exercise load, represents the weight of motion features, Indicates the weight of blood oxygen saturation SpO2, Indicates other The weight of the athlete's fatigue level is predicted by the NFI index.

5. A non-contact athlete physiological state assessment system based on feature fusion, used in the non-contact athlete physiological state assessment method based on feature fusion according to any one of claims 1 to 4, characterized in that: include: The rPPG signal extraction module is based on the 3DCNNTransform encoder to build an rPPG signal feature extraction model. The rPPG signal feature extraction model is trained using domain adaptation as an unsupervised learning scheme. The trained rPPG signal feature extraction model is used to extract rPPG signal features from the preprocessed video. A player information extraction module based on YOLO DeepSort 3DCNN is used to extract the player's information features and action features from the pre-processed video using the trained information and action feature extraction model, and then fuse the information features and action features to obtain motion information features. The continuous body state assessment module based on fusion signals is used to build a multi-source signal fusion network based on a deep learning model. It uses the acquired rPPG signal features and motion information features as input to continuously predict the athlete's fatigue level.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the non-contact athlete physiological state assessment method based on feature fusion as described in any one of claims 1-4.

Images