A real-time multi-person motion analysis system based on motion perception and semantic understanding

Abstract

The present invention relates to the fields of computer vision and artificial intelligence, and in particular to a real-time multi-person motion analysis system based on motion perception and semantic understanding, comprising a motion perception module, a motion tracking module, a motion semantic understanding module, and a motion performance analysis module. The motion region segmentation unit adopts background modeling or inter-frame difference method to identify motion regions and generate motion masks; the target detection unit is connected to the central processing unit via a data bus, and is used to process video frame data and detect the position and bounding box of the moving target. The motion region segmentation unit is connected to the target detection unit to generate a motion mask. By integrating multiple modules, the present invention can comprehensively capture and analyze motion data, thereby improving the accuracy and real-time performance of motion analysis. The combination of the target detection unit and the motion region segmentation unit enables the system to accurately identify moving targets and regions, laying a solid foundation for subsequent tracking and analysis.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and artificial intelligence, and in particular to a real-time multi-person motion analysis system based on motion perception and semantic understanding. Background Art

[0002] With the rapid development of computer vision and artificial intelligence technologies, sports video analysis has shown great application potential in sports training, medical rehabilitation, human-computer interaction and other fields. However, existing technologies still have the following problems:

[0003] In multi-person scenarios, moving targets are easily obscured by other targets or the background, making it difficult for tracking algorithms to capture the target's complete trajectory. Rapid movement and occlusion between the target's limbs can cause tracking algorithms to drift or lose the target. Multiple moving targets may have similar appearances, increasing the difficulty of distinguishing and tracking different targets. Methods based on correlation filtering or twin networks often lack motion information modeling when processing complex scenarios, making it difficult to cope with challenges such as occlusion and rapid movement.

[0004] In addition, when performing semantic understanding on single-person sports videos, existing methods mainly focus on action recognition, that is, identifying the action categories occurring in the video, such as running, jumping, etc., but lack a refined understanding of the details of the movement, such as the amplitude, angle, rhythm, etc., and cannot perform quantitative analysis of the sports performance, such as evaluating the standardization of the movement, the intensity of the movement, and the efficiency of the movement. Summary of the Invention

[0005] The present invention aims to provide a real-time multi-person motion analysis system based on motion perception and semantic understanding to solve the above problems.

[0006] The present invention is achieved through the following technical solutions:

[0007] A real-time multi-person motion analysis system based on motion perception and semantic understanding includes a motion perception module, a motion tracking module, a motion semantic understanding module, and a motion performance analysis module. The motion perception module is electrically connected to the motion tracking module to achieve real-time data transmission and processing.

[0008] The motion perception module includes a target detection unit and a motion region segmentation unit. The target detection unit uses a lightweight convolutional neural network to extract image features from a video frame sequence and detect the position and bounding box of the moving target. The motion region segmentation unit uses background modeling or inter-frame difference method to identify the moving area and generate a motion mask.

[0009] The object detection unit is connected to the central processing unit via a data bus and is used to process video frame data and detect the position and bounding box of moving objects. The motion region segmentation unit is connected to the object detection unit and is used to generate motion masks.

[0010] By integrating multiple modules, the present invention can comprehensively capture and analyze motion data, improving the accuracy and real-time performance of motion analysis. The combination of the target detection unit and the motion region segmentation unit enables the system to accurately identify moving targets and regions, laying a solid foundation for subsequent tracking and analysis.

[0011] Preferably, the motion perception module further comprises an adaptive clustering segmentation unit, which uses the DBSCAN algorithm to cluster and segment the moving target to generate multiple moving target regions, and the adaptive clustering segmentation unit processes and segments the moving target regions in real time through data;

[0012] The target detection unit uses a lightweight convolutional neural network (CNN) to extract image features from a video frame sequence and detect the position and bounding box of the moving target, which is expressed by the following formula:

[0013] Feature extraction: ;

[0014] in, is the input video frame, is the extracted feature, CNN represents convolutional neural network;

[0015] Bounding Box Detection: ;

[0016] in, is the bounding box of the detected moving object, represents the detection algorithm;

[0017] According to the bounding box information, the pixel coordinates are converted into actual physical coordinates or positions in the camera coordinate system to determine the exact position of the moving target in the image;

[0018] The motion region segmentation unit uses background modeling or inter-frame difference method to identify the motion region and generate a motion mask, which is expressed by the following formula:

[0019] Motion area recognition: ;

[0020] in, is the motion mask, Represents a segmentation algorithm that uses bounding boxes and video frames To identify motion areas;

[0021] The adaptive clustering segmentation unit uses the DBSCAN algorithm to cluster and segment the moving targets, generating multiple moving target regions. It inputs the motion mask and the position information of the moving targets, outputs the clustering results, and segments the moving regions into independent moving targets. When the DBSCAN algorithm determines the core points and boundary points, the density of the point P is defined as:

[0022] ;

[0023] in, It is the set of points in the eps-neighborhood of point P in the motion mask M. If point P is not a core point in the motion mask M, but there is at least one core point in its eps-neighborhood, then P is a boundary point. In addition, if the eps-neighborhoods of two core points overlap, they belong to the same cluster, which is represented in the motion mask M as two moving targets belonging to the same motion event.

[0024] This invention adds an adaptive clustering segmentation unit that uses the DBSCAN algorithm to cluster and segment moving targets, generating multiple moving target regions and improving the accuracy of moving target segmentation. By using this adaptive clustering segmentation unit, the system can more effectively handle complex motion scenes, distinguish different moving targets, reduce misidentification and overlap issues, and enhance the system's adaptability and robustness.

[0025] Preferably, the motion tracking module includes a motion perception memory selection unit and a Kalman filter unit. The motion perception memory selection unit calculates the IoU (Intersection over Union) score between the mask predicted by the current frame and the mask in the historical frame memory, calculates the target appearance score, calculates the IoU score predicted by the Kalman filter, and selects the historical frame most relevant to the current frame as the memory information, which is specifically obtained by the following formula;

[0026] ;

[0027] Among them, Area of Overlap is the area of the overlapping area of the two masks, and Area of Union is the area of the union of the two masks;

[0028] The motion perception memory selection unit outputs the mask affinity score, the target appearance score, and the IoU score predicted by the Kalman filter. The Kalman filter unit uses the Kalman filter to model the target motion trajectory and predict the position and size of the target in the next frame.

[0029] The state vector of the Kalman filter unit is defined as:

[0030] ;

[0031] in, and Represents the coordinates of the center of the bounding box;

[0032] and Indicates the width and height of the bounding box;

[0033] 、 、 、 The velocities representing the center coordinates, width, and height of the bounding box, respectively;

[0034] The prediction and update steps of the Kalman filter are as follows:

[0035] predict:

[0036] ;

[0037] in, is the state transition matrix, is the state vector at time t, is the predicted state vector at time t+1

[0038] renew:

[0039] ;

[0040] ;

[0041] in, is the Kalman gain, which is used to adjust the predicted state vector to match the observed value;

[0042] is the prediction covariance matrix at time t;

[0043] is the observation matrix, which maps the state vector to the observation space;

[0044] is the observation noise covariance matrix;

[0045] is the observed value at time t;

[0046] is the updated state vector at time t+1;

[0047] The motion perception memory selection unit and the Kalman filter unit realize real-time data processing and generation of tracking results.

[0048] The present invention combines a motion perception memory selection unit with a Kalman filter unit to enable the system to track the target's motion trajectory in real time, and maintain tracking continuity and accuracy even when the target moves quickly or is blocked.

[0049] Preferably, the motion tracking module further includes a multi-target trajectory prediction unit, which uses an RNN (recurrent neural network) model to predict the future trajectories of multiple targets based on the prediction results of the Kalman filter to improve the stability and accuracy of tracking. The multi-target trajectory prediction unit predicts and adjusts the motion trajectory of the target in real time;

[0050] The RNN model structure includes an input layer, an RNN layer, and an output layer. The input layer receives the state vector from the Kalman filter unit, including position, size, and velocity information. The RNN layer uses LSTM (Long Short-Term Memory Network) as the core of the RNN to process sequence data and capture long-term dependencies. The output layer predicts the position and size of the target in the future frame.

[0051] During the training phase of the RNN model, historical trajectory data is used to train the RNN model. Specifically, the state vector and observation value of the Kalman filter are used as input, and the target position and size are used as output. During the testing phase, the prediction results of the Kalman filter are used as the input of the RNN. The RNN predicts the future trajectory and feeds the results back to the Kalman filter for state update. The back-propagation through time (BPTT) algorithm is then used for training to optimize the weights of the RNN to minimize the prediction error.

[0052] By adding an RNN model, the present invention enables the system to predict the future motion trajectory of the target and respond in advance, thereby improving the foresight and accuracy of tracking, especially in complex dynamic environments.

[0053] Preferably, the motion semantic understanding module includes a multimodal data fusion unit, a motion video semantic description unit and a spatiotemporal relationship understanding unit. The multimodal data fusion unit takes motion data and video data as input and adopts a unified multimodal training strategy for training. The motion data is parameterized using the SMPL model, and the video data uses a pre-trained convolutional neural network to extract image features. The multimodal data fusion unit realizes data fusion and training.

[0054] The motion video semantic description unit uses a pre-trained language model to perform semantic description on the motion video and generates a natural language description. The motion video semantic description unit is connected to the multimodal data fusion unit and outputs the semantic description result.

[0055] The spatiotemporal relationship understanding unit uses the reasoning ability of the language model to analyze the spatiotemporal relationship in the motion video and identify the action sequence, direction change, target intention, etc. The spatiotemporal relationship understanding unit is connected to the motion video semantic description unit and outputs the spatiotemporal relationship analysis results;

[0056] Through multimodal data fusion and semantic description, the present invention can generate natural language descriptions and analyze the spatiotemporal relationships in motion videos, thereby improving the system's ability to understand and describe motion scenes and providing rich semantic information for motion performance analysis.

[0057] Preferably, the motion semantic understanding module also includes a multi-perspective fusion unit, which extracts motion data from video streams of multiple cameras, fuses the multi-perspective data using a 3D convolutional neural network, and generates a more comprehensive and accurate motion semantic description. The multi-perspective fusion unit processes and fuses multi-perspective motion data in real time.

[0058] The application of the multi-view fusion unit of the present invention enables the system to capture motion information from different angles, improves the comprehensiveness and accuracy of motion semantic description, and can provide richer motion information, especially in complex scenes.

[0059] Preferably, the sports performance analysis module includes a movement normativeness evaluation unit, a movement intensity quantification unit, and a sports performance comprehensive evaluation unit. The movement normativeness evaluation unit evaluates the normativeness of the movement, such as movement amplitude, angle, rhythm, etc., based on the output of the movement semantic understanding module. The movement normativeness evaluation unit is connected to the movement semantic understanding module and outputs the evaluation result.

[0060] The exercise intensity quantification unit quantifies and analyzes exercise intensity, which includes exercise frequency, speed, and strength. The exercise intensity quantification unit is connected to the exercise tracking module to transmit the quantification results. The exercise performance comprehensive evaluation unit comprehensively evaluates exercise performance and provides improvement suggestions. The exercise performance comprehensive evaluation unit is connected to the movement standardization evaluation unit and the exercise intensity quantification unit. The exercise intensity quantification unit outputs the comprehensive evaluation results.

[0061] Through the evaluation of movement norms and quantification of exercise intensity, the system can quantitatively analyze exercise performance and provide improvement suggestions, thereby improving the scientificity and effectiveness of exercise training and evaluation.

[0062] Preferably, the motion perception module further includes a background modeling unit, which is used to generate a background model, and transmit the background model to the motion region segmentation unit, which identifies the motion region through a background modeling method;

[0063] The background modeling unit includes a Gaussian mixture model (GMM) and a background update mechanism. The background update mechanism dynamically updates the background model according to the real-time changes of the background model.

[0064] The application of the background modeling unit of the present invention enables the system to dynamically identify and distinguish the background from the motion area, thereby improving the accuracy and robustness of moving target detection, especially in an environment with complex or changing backgrounds.

[0065] Preferably, the motion tracking module further includes a multi-target tracking management unit, which is connected to the motion perception memory selection unit and the Kalman filter unit to transmit tracking management information thereto;

[0066] The multi-target tracking management unit includes a target ID allocation module and a target association module. The target ID allocation module assigns a unique ID to each target. The target association module associates the target with the ID according to the target's motion trajectory to avoid confusion between targets.

[0067] The application of the multi-target tracking management unit of the present invention enables the system to effectively manage the tracking of multiple targets, avoids confusion between targets, and improves the accuracy and reliability of tracking.

[0068] Preferably, the motion semantic understanding module further includes a motion data preprocessing unit, which is connected to the multimodal data fusion unit;

[0069] The motion data preprocessing unit includes a noise filtering module and a data alignment module. The noise filtering module removes noise from the motion data, and the data alignment module aligns data of different modes to improve the consistency and comparability of the data.

[0070] The application of the motion data preprocessing unit of the present invention improves the consistency and comparability of data, provides high-quality data input for subsequent analysis and processing, and enhances the accuracy and robustness of the system.

[0071] Beneficial effects

[0072] By integrating multiple modules, the present invention achieves comprehensive capture and in-depth analysis of motion scenes, significantly improving the accuracy and real-time performance of motion analysis. The system effectively distinguishes different moving targets through an adaptive clustering segmentation unit and the DBSCAN algorithm, enhancing adaptability and robustness. The combination of a motion-sensing memory selection unit and a Kalman filter unit enables the system to maintain tracking continuity and accuracy even when the target is rapidly moving or obscured. The multi-target trajectory prediction unit utilizes an RNN model to predict the target's future trajectory, improving tracking foresight and accuracy. The combination of a multimodal data fusion unit and a motion video semantic description unit enables the system to generate natural language descriptions and analyze spatiotemporal relationships in motion videos, enhancing the system's ability to understand and describe motion scenes. The use of a multi-view fusion unit enables the system to capture motion information from different angles, improving the comprehensiveness and accuracy of motion semantic descriptions. The combination of an action norm assessment unit and a motion intensity quantification unit enables the system to quantitatively analyze athletic performance and provide improvement suggestions, enhancing the scientific nature and effectiveness of sports training and assessment. The use of a background modeling unit enables the system to dynamically identify and distinguish between background and motion areas, improving the accuracy and robustness of moving target detection. The implementation of the multi-target tracking management unit enables the system to effectively manage the tracking of multiple targets, avoiding confusion between targets and improving tracking accuracy and reliability. The application of the motion data preprocessing unit improves data consistency and comparability, providing high-quality data input for subsequent analysis and processing, and enhancing the accuracy and robustness of the system. Overall, through the collaborative operation of these modules, the system provides a comprehensive, accurate, and efficient solution for real-time multi-person motion analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0074] Figure 1 A connection block diagram of the system of the present invention;

[0075] Figure 2 Flowchart of steps implemented for the motion tracking module;

[0076] Figure 3 Flowchart of the steps implemented for the motion semantic understanding module. DETAILED DESCRIPTION

[0077] In order to enable those skilled in the art to better understand the present invention, the following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present invention.

[0078] Example

[0079] A real-time multi-person motion analysis system based on motion perception and semantic understanding includes a motion perception module, a motion tracking module, a motion semantic understanding module, and a motion performance analysis module. The motion perception module is electrically connected to the motion tracking module to achieve real-time data transmission and processing.

[0080] The motion perception module includes a target detection unit and a motion region segmentation unit. The target detection unit uses a lightweight convolutional neural network to extract image features from a video frame sequence and detect the position and bounding box of the moving target. The motion region segmentation unit uses background modeling or inter-frame difference method to identify the moving area and generate a motion mask.

[0081] The object detection unit is connected to the central processing unit via a data bus and is used to process video frame data and detect the position and bounding box of moving objects. The motion region segmentation unit is connected to the object detection unit and is used to generate motion masks.

[0082] By integrating multiple modules, the present invention can comprehensively capture and analyze motion data, improving the accuracy and real-time performance of motion analysis. The combination of the target detection unit and the motion region segmentation unit enables the system to accurately identify moving targets and regions, laying a solid foundation for subsequent tracking and analysis.

[0083] In this embodiment, the motion perception module further includes an adaptive clustering segmentation unit, which uses the DBSCAN algorithm to cluster and segment the moving target to generate multiple moving target regions. The adaptive clustering segmentation unit processes and segments the moving target regions in real time through data.

[0084] The target detection unit uses a lightweight convolutional neural network (CNN) to extract image features from a video frame sequence and detect the position and bounding box of the moving target, which is expressed by the following formula:

[0085] Feature extraction: ;

[0086] in, is the input video frame, is the extracted feature, CNN represents convolutional neural network;

[0087] Bounding Box Detection: ;

[0088] in, is the bounding box of the detected moving object, represents the detection algorithm;

[0089] According to the bounding box information, the pixel coordinates are converted into actual physical coordinates or positions in the camera coordinate system to determine the exact position of the moving target in the image;

[0090] The motion region segmentation unit uses background modeling or inter-frame difference method to identify the motion region and generate a motion mask, which is expressed by the following formula:

[0091] Motion area recognition: ;

[0092] in, is the motion mask, Represents a segmentation algorithm that uses bounding boxes and video frames To identify motion areas;

[0093] The adaptive clustering segmentation unit uses the DBSCAN algorithm to cluster and segment the moving targets, generating multiple moving target regions. It inputs the motion mask and the position information of the moving targets, outputs the clustering results, and segments the moving regions into independent moving targets. When the DBSCAN algorithm determines the core points and boundary points, the density of the point P is defined as:

[0094] ;

[0095] in, It is the set of points in the eps-neighborhood of point P in the motion mask M. If point P is not a core point in the motion mask M, but there is at least one core point in its eps-neighborhood, then P is a boundary point. In addition, if the eps-neighborhoods of two core points overlap, they belong to the same cluster, which is represented in the motion mask M as two moving targets belonging to the same motion event.

[0096] This invention adds an adaptive clustering segmentation unit that uses the DBSCAN algorithm to cluster and segment moving targets, generating multiple moving target regions and improving the accuracy of moving target segmentation. By using this adaptive clustering segmentation unit, the system can more effectively handle complex motion scenes, distinguish different moving targets, reduce misidentification and overlap issues, and enhance the system's adaptability and robustness.

[0097] In this embodiment, the motion tracking module includes a motion perception memory selection unit and a Kalman filter unit. The motion perception memory selection unit calculates the IoU (Intersection over Union) score between the mask predicted by the current frame and the mask in the historical frame memory, calculates the target occurrence score, calculates the IoU score predicted by the Kalman filter, and selects the historical frame most relevant to the current frame as the memory information, which is specifically obtained by the following formula:

[0098] ;

[0099] Among them, Area of Overlap is the area of the overlapping area of the two masks, and Area of Union is the area of the union of the two masks;

[0100] The motion perception memory selection unit outputs the mask affinity score, the target appearance score, and the IoU score predicted by the Kalman filter. The Kalman filter unit uses the Kalman filter to model the target motion trajectory and predict the position and size of the target in the next frame.

[0101] The state vector of the Kalman filter unit is defined as:

[0102] ;

[0103] in, and Represents the coordinates of the center of the bounding box;

[0104] and Indicates the width and height of the bounding box;

[0105] 、 、 、 The velocities representing the center coordinates, width, and height of the bounding box, respectively;

[0106] The prediction and update steps of the Kalman filter are as follows:

[0107] predict:

[0108] ;

[0109] in, is the state transition matrix, is the state vector at time t, is the predicted state vector at time t+1

[0110] renew:

[0111] ;

[0112] ;

[0113] in, is the Kalman gain, which is used to adjust the predicted state vector to match the observed value;

[0114] is the prediction covariance matrix at time t;

[0115] is the observation matrix, which maps the state vector to the observation space;

[0116] is the observation noise covariance matrix;

[0117] is the observed value at time t;

[0118] is the updated state vector at time t+1;

[0119] The motion perception memory selection unit and the Kalman filter unit realize real-time data processing and generation of tracking results.

[0120] The present invention combines a motion perception memory selection unit with a Kalman filter unit to enable the system to track the target's motion trajectory in real time, and maintain tracking continuity and accuracy even when the target moves quickly or is blocked.

[0121] In this embodiment, the motion tracking module further includes a multi-target trajectory prediction unit. The multi-target trajectory prediction unit uses an RNN (recurrent neural network) model to predict the future trajectories of multiple targets based on the prediction results of the Kalman filter, thereby improving the stability and accuracy of tracking. The multi-target trajectory prediction unit predicts and adjusts the motion trajectory of the target in real time.

[0122] The RNN model structure includes an input layer, an RNN layer, and an output layer. The input layer receives the state vector from the Kalman filter unit, including position, size, and velocity information. The RNN layer uses LSTM (Long Short-Term Memory Network) as the core of the RNN to process sequence data and capture long-term dependencies. The output layer predicts the position and size of the target in the future frame.

[0123] During the training phase of the RNN model, historical trajectory data is used to train the RNN model. Specifically, the state vector and observation value of the Kalman filter are used as input, and the target position and size are used as output. During the testing phase, the prediction results of the Kalman filter are used as the input of the RNN. The RNN predicts the future trajectory and feeds the results back to the Kalman filter for state update. The back-propagation through time (BPTT) algorithm is then used for training to optimize the weights of the RNN to minimize the prediction error.

[0124] By adding an RNN model, the present invention enables the system to predict the future motion trajectory of the target and respond in advance, thereby improving the foresight and accuracy of tracking, especially in complex dynamic environments.

[0125] In this embodiment, the motion semantic understanding module includes a multimodal data fusion unit, a motion video semantic description unit and a spatiotemporal relationship understanding unit. The multimodal data fusion unit takes motion data and video data as input and adopts a unified multimodal training strategy for training. The motion data is parameterized using the SMPL model, and the video data uses a pre-trained convolutional neural network to extract image features. The multimodal data fusion unit realizes data fusion and training.

[0126] The motion video semantic description unit uses a pre-trained language model to perform semantic description on the motion video and generates a natural language description. The motion video semantic description unit is connected to the multimodal data fusion unit and outputs the semantic description result.

[0127] The spatiotemporal relationship understanding unit uses the reasoning ability of the language model to analyze the spatiotemporal relationship in the motion video and identify the action sequence, direction change, target intention, etc. The spatiotemporal relationship understanding unit is connected to the motion video semantic description unit and outputs the spatiotemporal relationship analysis results;

[0128] Through multimodal data fusion and semantic description, the present invention can generate natural language descriptions and analyze the spatiotemporal relationships in motion videos, thereby improving the system's ability to understand and describe motion scenes and providing rich semantic information for motion performance analysis.

[0129] In this embodiment, the motion semantic understanding module also includes a multi-perspective fusion unit, which extracts motion data from video streams of multiple cameras, uses a 3D convolutional neural network to fuse the multi-perspective data, and generates a more comprehensive and accurate motion semantic description. The multi-perspective fusion unit processes and fuses multi-perspective motion data in real time.

[0130] The application of the multi-view fusion unit of the present invention enables the system to capture motion information from different angles, improves the comprehensiveness and accuracy of motion semantic description, and can provide richer motion information, especially in complex scenes.

[0131] In this embodiment, the sports performance analysis module includes a movement normativeness assessment unit, a movement intensity quantification unit, and a sports performance comprehensive evaluation unit. The movement normativeness assessment unit assesses the normativeness of the movement, such as movement amplitude, angle, rhythm, etc., based on the output of the movement semantic understanding module. The movement normativeness assessment unit is connected to the movement semantic understanding module and outputs the assessment result.

[0132] The exercise intensity quantification unit quantifies and analyzes exercise intensity, which includes exercise frequency, speed, and strength. The exercise intensity quantification unit is connected to the exercise tracking module to transmit the quantification results. The exercise performance comprehensive evaluation unit comprehensively evaluates exercise performance and provides improvement suggestions. The exercise performance comprehensive evaluation unit is connected to the movement standardization evaluation unit and the exercise intensity quantification unit. The exercise intensity quantification unit outputs the comprehensive evaluation results.

[0133] Through the evaluation of movement norms and quantification of exercise intensity, the system can quantitatively analyze exercise performance and provide improvement suggestions, thereby improving the scientificity and effectiveness of exercise training and evaluation.

[0134] In this embodiment, the motion perception module further includes a background modeling unit, which is used to generate a background model. The background model is transmitted to the motion region segmentation unit, and the motion region segmentation unit identifies the motion region through the background modeling method.

[0135] The background modeling unit includes a Gaussian mixture model (GMM) and a background update mechanism. The background update mechanism dynamically updates the background model according to the real-time changes of the background model.

[0136] The application of the background modeling unit of the present invention enables the system to dynamically identify and distinguish the background from the motion area, thereby improving the accuracy and robustness of moving target detection, especially in an environment with complex or changing backgrounds.

[0137] In this embodiment, the motion tracking module further includes a multi-target tracking management unit, which is connected to the motion perception memory selection unit and the Kalman filter unit to transmit tracking management information thereto;

[0138] The multi-target tracking management unit includes a target ID allocation module and a target association module. The target ID allocation module assigns a unique ID to each target. The target association module associates the target with the ID according to the target's motion trajectory to avoid confusion between targets.

[0139] The application of the multi-target tracking management unit of the present invention enables the system to effectively manage the tracking of multiple targets, avoids confusion between targets, and improves the accuracy and reliability of tracking.

[0140] In this embodiment, the motion semantic understanding module further includes a motion data preprocessing unit, which is connected to the multimodal data fusion unit;

[0141] The motion data preprocessing unit includes a noise filtering module and a data alignment module. The noise filtering module removes noise from the motion data, and the data alignment module aligns data of different modes to improve the consistency and comparability of the data.

[0142] The application of the motion data preprocessing unit of the present invention improves the consistency and comparability of data, provides high-quality data input for subsequent analysis and processing, and enhances the accuracy and robustness of the system.

[0143] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A real-time multi-person motion analysis system based on motion perception and semantic understanding, characterized by: It includes a motion perception module, a motion tracking module, a motion semantic understanding module and a motion performance analysis module. The motion perception module is electrically connected to the motion tracking module to achieve real-time data transmission and processing; The motion perception module includes a target detection unit and a motion region segmentation unit. The target detection unit uses a lightweight convolutional neural network to extract image features from a video frame sequence and detect the position and bounding box of the moving target. The motion region segmentation unit uses background modeling or inter-frame difference method to identify the moving area and generate a motion mask. The target detection unit is connected to the central processing unit through a data bus and is used to process video frame data and detect the position and bounding box of the moving target. The motion region segmentation unit is connected to the target detection unit and is used to generate a motion mask. The motion tracking module includes a motion perception memory selection unit and a Kalman filter unit. The motion perception memory selection unit calculates the IoU (Intersection over Union) score between the mask predicted by the current frame and the mask in the historical frame memory, calculates the target occurrence score, calculates the IoU score predicted by the Kalman filter, and selects the historical frame most relevant to the current frame as the memory information, which is specifically obtained by the following formula: Among them, Area of Overlap is the area of the overlapping area of the two masks, and Area of Union is the area of the union of the two masks; The motion perception memory selection unit outputs the mask affinity score, the target appearance score, and the IoU score predicted by the Kalman filter. The Kalman filter unit uses the Kalman filter to model the target motion trajectory and predict the position and size of the target in the next frame. The state vector of the Kalman filter unit is defined as: Where x and y represent the coordinates of the center of the bounding box; w and h represent the width and height of the bounding box; The velocities representing the center coordinates, width, and height of the bounding box, respectively; The prediction and update steps of the Kalman filter are as follows: predict: Where F is the state transition matrix, x t is the state vector at time t, is the predicted state vector at time t+1 renew: K t =P t ·H T ·(H·P t ·H T +R) -1 Among them, K t is the Kalman gain, which is used to adjust the predicted state vector to match the observed value; P t is the prediction covariance matrix at time t; H is the observation matrix, which maps the state vector to the observation space; R is the observation noise covariance matrix; z t is the observed value at time t; x t+1 is the updated state vector at time t+1; The motion perception memory selection unit and the Kalman filter unit realize real-time data processing and generation of tracking results; The motion tracking module also includes a multi-target trajectory prediction unit. The multi-target trajectory prediction unit uses an RNN model to predict the future trajectories of multiple targets based on the prediction results of the Kalman filter to improve the stability and accuracy of tracking. The multi-target trajectory prediction unit predicts and adjusts the motion trajectory of the target in real time; The RNN model structure includes an input layer, an RNN layer, and an output layer. The input layer receives the state vector from the Kalman filter unit, including position, size, and velocity information. The RNN layer uses LSTM as the core of the RNN to process sequence data and capture long-term dependencies. The output layer predicts the position and size of the target in the future frame. During the training phase of the RNN model, historical trajectory data is used to train the RNN model. Specifically, the state vector and observation value of the Kalman filter are used as input, and the target position and size are used as output. During the testing phase, the prediction results of the Kalman filter are used as input to the RNN. The RNN predicts the future trajectory and feeds the results back to the Kalman filter for state update. The RNN is then trained through a time algorithm using backpropagation to optimize the RNN weights to minimize the prediction error. The motion semantic understanding module includes a multimodal data fusion unit, a motion video semantic description unit, and a spatiotemporal relationship understanding unit. The multimodal data fusion unit takes motion data and video data as input and adopts a unified multimodal training strategy for training. The motion data is parameterized using the SMPL model, and the video data uses a pre-trained convolutional neural network to extract image features. The multimodal data fusion unit realizes data fusion and training. The motion video semantic description unit uses a pre-trained language model to perform semantic description on the motion video and generates a natural language description. The motion video semantic description unit is connected to the multimodal data fusion unit and outputs the semantic description result. The spatiotemporal relationship understanding unit uses the reasoning ability of the language model to analyze the spatiotemporal relationship in the motion video and identify the action sequence, direction change, and target intention. The spatiotemporal relationship understanding unit is connected to the motion video semantic description unit, and the spatiotemporal relationship understanding unit outputs the spatiotemporal relationship analysis results.

2. A real-time multi-person motion analysis system based on motion perception and semantic understanding according to claim 1, characterized in that: The motion perception module also includes an adaptive clustering segmentation unit, which uses the DBSCAN algorithm to cluster and segment the moving target to generate multiple moving target areas. The adaptive clustering segmentation unit processes and segments the moving target areas in real time through data; The target detection unit uses a lightweight convolutional neural network to extract image features from a video frame sequence and detect the position and bounding box of the moving target, which is expressed by the following formula: Feature extraction: F(I) = CNN(I); Where I is the input video frame, F(I) is the extracted feature, and CNN represents convolutional neural network; Bounding box detection: B = Detect(F(I)); Where B is the bounding box of the detected moving object, and Detect represents the detection algorithm; According to the bounding box information, the pixel coordinates are converted into actual physical coordinates or positions in the camera coordinate system to determine the exact position of the moving target in the image; The motion region segmentation unit uses background modeling or inter-frame difference method to identify the motion region and generate a motion mask, which is expressed by the following formula: Motion region identification: M = Segment (I, B); Where M is the motion mask and Segment represents the segmentation algorithm, which identifies the motion region based on the bounding box B and the video frame I; The adaptive clustering segmentation unit uses the DBSCAN algorithm to cluster and segment the moving targets, generating multiple moving target regions. It inputs the motion mask and the position information of the moving targets, outputs the clustering results, and segments the moving regions into independent moving targets. When the DBSCAN algorithm determines the core points and boundary points, the density of the point P is defined as: Density(P)=|N ∈ (P)|; Among them, N ∈ (P) is the set of points in the eps-neighborhood of point P in the motion mask M, and if point P is not a core point in the motion mask M, but there is at least one core point in its eps-neighborhood, then P is a boundary point. In addition, if the eps-neighborhoods of two core points overlap, they belong to the same cluster, which is represented in the motion mask M as two moving targets belonging to the same motion event.

3. The real-time multi-person motion analysis system based on motion perception and semantic understanding according to claim 1 is characterized in that: The motion semantic understanding module also includes a multi-perspective fusion unit, which extracts motion data from video streams of multiple cameras and uses a 3D convolutional neural network to fuse the multi-perspective data to generate a more comprehensive and accurate motion semantic description. The multi-perspective fusion unit processes and fuses multi-perspective motion data in real time.

4. The real-time multi-person motion analysis system based on motion perception and semantic understanding according to claim 1, characterized in that: The sports performance analysis module includes a movement normativeness evaluation unit, a movement intensity quantification unit, and a movement performance comprehensive evaluation unit. The movement normativeness evaluation unit evaluates the normativeness of the movement, including movement amplitude, angle, and rhythm, based on the output of the movement semantic understanding module. The movement normativeness evaluation unit is connected to the movement semantic understanding module and outputs the evaluation result. The exercise intensity quantification unit quantifies and analyzes exercise intensity, which includes exercise frequency, speed, and strength. The exercise intensity quantification unit is connected to the exercise tracking module to transmit the quantification results. The exercise performance comprehensive evaluation unit comprehensively evaluates exercise performance and provides improvement suggestions. The exercise performance comprehensive evaluation unit is connected to the movement standardization evaluation unit and the exercise intensity quantification unit. The exercise intensity quantification unit outputs the comprehensive evaluation results.

5. The real-time multi-person motion analysis system based on motion perception and semantic understanding according to claim 1 is characterized in that: The motion perception module also includes a background modeling unit, which is used to generate a background model. The background model is transmitted to the motion region segmentation unit, and the motion region segmentation unit identifies the motion region through the background modeling method. The background modeling unit includes a Gaussian mixture model and a background updating mechanism. The background updating mechanism dynamically updates the background model according to the real-time changes of the background model.

6. The real-time multi-person motion analysis system based on motion perception and semantic understanding according to claim 1, characterized in that: The motion tracking module also includes a multi-target tracking management unit, which is connected to the motion perception memory selection unit and the Kalman filter unit to transmit tracking management information to them; The multi-target tracking management unit includes a target ID allocation module and a target association module. The target ID allocation module assigns a unique ID to each target. The target association module associates the target with the ID according to the target's motion trajectory to avoid confusion between targets.

7. The real-time multi-person motion analysis system based on motion perception and semantic understanding according to claim 1 is characterized in that: The motion semantic understanding module also includes a motion data preprocessing unit, which is connected to the multimodal data fusion unit; The motion data preprocessing unit includes a noise filtering module and a data alignment module. The noise filtering module removes noise from the motion data, and the data alignment module aligns data of different modes to improve the consistency and comparability of the data.

Images