Real-time multi-person motion analysis system based on motion perception and semantic understanding

Abstract

The invention relates to the field of computer vision and artificial intelligence, in particular to a real-time multi-person motion analysis system based on motion perception and semantic understanding, which comprises a motion perception module, a motion tracking module, a motion semantic understanding module and a motion performance analysis module. The motion area segmentation unit adopts a background modeling or inter-frame difference method to identify a motion area and generate a motion mask; the target detection unit is connected with the central processing unit through a data bus and used for processing video frame data and detecting the position and the bounding box of a moving target. The motion area segmentation unit is connected with the target detection unit and is used for generating a motion mask. By integrating a plurality of modules, the motion data can be comprehensively captured and analyzed, and the accuracy and real-time performance of motion analysis are improved. Through combination of the target detection unit and the motion area segmentation unit, the system can accurately identify a motion target and a motion area, and a solid foundation is laid for subsequent tracking and analysis.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and artificial intelligence, and in particular to a real-time multi-person motion analysis system based on motion perception and semantic understanding. Background Art

[0002] With the rapid development of computer vision and artificial intelligence technology, sports video analysis has shown great application potential in sports training, medical rehabilitation, human-computer interaction and other fields. However, the existing technology still has the following problems: In multi-person scenes, moving targets are easily blocked by other targets or backgrounds, making it difficult for the tracking algorithm to capture the target's complete motion trajectory. The fast movement of the moving target and the occlusion between its own limbs can cause the tracking algorithm to drift or lose the target. Multiple moving targets may have similar appearances, which increases the difficulty of distinguishing and tracking different targets. Methods based on correlation filtering or twin networks often lack modeling of motion information when dealing with complex scenes, making it difficult to cope with challenges such as occlusion and fast movement. In addition, when performing semantic understanding on single-person sports videos, existing methods mainly focus on action recognition, that is, identifying the action categories occurring in the video, such as running, jumping, etc., but lack a refined understanding of the details of the movement, such as the amplitude, angle, rhythm, etc., and cannot perform quantitative analysis of the sports performance, such as evaluating the standardization of movements, exercise intensity, and exercise efficiency. Summary of the invention

[0003] The present invention aims to provide a real-time multi-person motion analysis system based on motion perception and semantic understanding to solve the above problems.

[0004] The present invention is achieved through the following technical solutions: A real-time multi-person motion analysis system based on motion perception and semantic understanding includes a motion perception module, a motion tracking module, a motion semantic understanding module and a motion performance analysis module. The motion perception module is electrically connected to the motion tracking module to achieve real-time data transmission and processing; The motion perception module includes a target detection unit and a motion region segmentation unit. The target detection unit uses a lightweight convolutional neural network to extract image features from a video frame sequence and detect the position and bounding box of the moving target. The motion region segmentation unit uses background modeling or inter-frame difference method to identify the motion region and generate a motion mask. The target detection unit is connected to the central processing unit through a data bus and is used to process video frame data and detect the position and bounding box of the moving target. The motion region segmentation unit is connected to the target detection unit and is used to generate a motion mask.

[0005] The present invention can comprehensively capture and analyze motion data by integrating multiple modules, thereby improving the accuracy and real-time performance of motion analysis. The combination of the target detection unit and the motion region segmentation unit enables the system to accurately identify motion targets and regions, laying a solid foundation for subsequent tracking and analysis.

[0006] Preferably, the motion perception module further comprises an adaptive clustering segmentation unit, which uses a DBSCAN algorithm to perform clustering segmentation on the moving target to generate a plurality of moving target regions, and the adaptive clustering segmentation unit processes and segments the moving target regions in real time through data; The target detection unit uses a lightweight convolutional neural network (CNN) to extract image features from a video frame sequence and detect the position and bounding box of a moving target, which is expressed by the following formula: Feature extraction: ; in, is the input video frame, is the extracted feature, CNN stands for convolutional neural network; Bounding Box Detection: ; in, is the bounding box of the detected moving object, represents the detection algorithm; According to the bounding box information, the pixel coordinates are converted into actual physical coordinates or positions in the camera coordinate system to determine the exact position of the moving target in the image; The motion region segmentation unit uses background modeling or inter-frame difference method to identify the motion region and generate a motion mask, which is expressed by the following formula: Motion Region Identification: ; in, is the motion mask, represents the segmentation algorithm, which is based on the bounding box and video frames To identify the motion area; The adaptive clustering segmentation unit uses the DBSCAN algorithm to perform clustering segmentation on the moving target and generate multiple moving target regions. It outputs the clustering result by inputting the motion mask and the position information of the moving target, and segments the moving region into independent moving targets. When the DBSCAN algorithm determines the core points and boundary points, the density of the point P is defined as: ; in, It is the point set of point P in the eps-neighborhood of the motion mask M. If point P is not a core point in the motion mask M, but there is at least one core point in its eps-neighborhood, then P is a boundary point. In addition, if the eps-neighborhoods of two core points overlap, they belong to the same cluster, which is represented in the motion mask M as two moving targets belonging to the same motion event.

[0007] The present invention adds an adaptive clustering segmentation unit, uses the DBSCAN algorithm to cluster and segment moving targets, generates multiple moving target regions, and improves the accuracy of moving target segmentation. Through the application of the adaptive clustering segmentation unit, the system can more effectively handle complex motion scenes, distinguish different moving targets, reduce the problems of misidentification and overlap, and enhance the adaptability and robustness of the system.

[0008] Preferably, the motion tracking module includes a motion perception memory selection unit and a Kalman filter unit. The motion perception memory selection unit calculates the IoU (Intersection over Union) score between the mask predicted by the current frame and the mask in the historical frame memory, calculates the target appearance score, calculates the IoU score predicted by the Kalman filter, and selects the historical frame most relevant to the current frame as the memory information, which is specifically obtained by the following formula: ; Among them, Area of ​​Overlap is the area of ​​the overlapping area of ​​the two masks, and Area of ​​Union is the area of ​​the union of the two masks; The motion perception memory selection unit outputs the mask affinity score, the target appearance score and the IoU score predicted by the Kalman filter. The Kalman filter unit uses the Kalman filter to model the target motion trajectory and predict the position and size of the target in the next frame. The state vector of the Kalman filter unit is defined as: ; in, and Represents the coordinates of the center of the bounding box; and Indicates the width and height of the bounding box; , , , The speeds representing the center coordinates, width, and height of the bounding box, respectively; The prediction and update steps of the Kalman filter are as follows: predict: ; in, is the state transition matrix, is the state vector at time t, is the predicted state vector at time t+1 renew: ; ; in, is the Kalman gain, which is used to adjust the predicted state vector to match the observed value; is the prediction covariance matrix at time t; is the observation matrix, which maps the state vector to the observation space; is the observation noise covariance matrix; is the observed value at time t; is the updated state vector at time t+1; The motion perception memory selection unit and the Kalman filter unit realize real-time data processing and generation of tracking results.

[0009] The present invention combines the motion perception memory selection unit and the Kalman filter unit, so that the system can track the motion trajectory of the target in real time, and can maintain the continuity and accuracy of tracking even when the target moves quickly or is blocked; Preferably, the motion tracking module further includes a multi-target trajectory prediction unit, which uses an RNN (recurrent neural network) model to predict the future trajectories of multiple targets based on the prediction results of the Kalman filter to improve the stability and accuracy of tracking. The multi-target trajectory prediction unit predicts and adjusts the motion trajectory of the target in real time; The RNN model structure includes an input layer, an RNN layer and an output layer. The input layer receives the state vector from the Kalman filter unit, including the position, size and speed information. The RNN layer uses LSTM (Long Short-Term Memory Network) as the core of the RNN to process sequence data and capture long-term dependencies. The output layer predicts the position and size of the target in the future frame. During the training phase of the RNN model, historical trajectory data is used to train the RNN model. Specifically, the state vector and observation value of the Kalman filter are used as input, and the target position and size are used as output. During the testing phase, the prediction results of the Kalman filter are used as the input of the RNN. The RNN predicts the future trajectory and feeds the result back to the Kalman filter for state update. The back propagation through time (BPTT) algorithm is then used for training to optimize the weights of the RNN to minimize the prediction error.

[0010] By adding an RNN model, the present invention enables the system to predict the future motion trajectory of the target and respond in advance, thereby improving the foresight and accuracy of tracking, especially in complex dynamic environments.

[0011] Preferably, the motion semantic understanding module includes a multimodal data fusion unit, a motion video semantic description unit and a spatiotemporal relationship understanding unit. The multimodal data fusion unit takes motion data and video data as input and adopts a unified multimodal training strategy for training. The motion data is parameterized using the SMPL model, and the video data uses a pre-trained convolutional neural network to extract image features. The multimodal data fusion unit realizes data fusion and training.

[0012] The motion video semantic description unit uses a pre-trained language model to semantically describe the motion video and generates a natural language description. The motion video semantic description unit is connected to the multimodal data fusion unit, and the motion video semantic description unit outputs a semantic description result. The spatiotemporal relationship understanding unit uses the reasoning ability of the language model to analyze the spatiotemporal relationship in the motion video, identify the action sequence, direction change, target intention, etc. The spatiotemporal relationship understanding unit is connected to the motion video semantic description unit, and the spatiotemporal relationship understanding unit outputs the spatiotemporal relationship analysis results; Through multimodal data fusion and semantic description, the present invention can generate natural language descriptions and analyze the spatiotemporal relationships in motion videos, thereby improving the system's ability to understand and describe motion scenes and providing rich semantic information for motion performance analysis.

[0013] Preferably, the motion semantic understanding module also includes a multi-perspective fusion unit, which extracts motion data from video streams of multiple cameras, fuses the multi-perspective data using a 3D convolutional neural network, and generates a more comprehensive and accurate motion semantic description. The multi-perspective fusion unit processes and fuses multi-perspective motion data in real time.

[0014] The application of the multi-view fusion unit of the present invention enables the system to capture motion information from different angles, improves the comprehensiveness and accuracy of motion semantic description, and can provide richer motion information, especially in complex scenes.

[0015] Preferably, the sports performance analysis module includes an action normativeness evaluation unit, an exercise intensity quantification unit and a sports performance comprehensive evaluation unit. The action normativeness evaluation unit evaluates the normativeness of the action, such as the action amplitude, angle, rhythm, etc., based on the output of the sports semantic understanding module. The action normativeness evaluation unit is connected to the sports semantic understanding module, and the action normativeness evaluation unit outputs the evaluation result. The exercise intensity quantification unit quantitatively analyzes exercise intensity, which includes exercise frequency, speed, and strength. The exercise intensity quantification unit is connected to the exercise tracking module to transmit the quantification results. The exercise performance comprehensive evaluation unit comprehensively evaluates the exercise performance and provides improvement suggestions. The exercise performance comprehensive evaluation unit is connected to the action normative evaluation unit and the exercise intensity quantification unit. The exercise intensity quantification unit outputs the comprehensive evaluation results.

[0016] Through the action normative evaluation and exercise intensity quantification, the system of the present invention can quantitatively analyze the exercise performance and provide improvement suggestions, thereby improving the scientificity and effectiveness of exercise training and evaluation.

[0017] Preferably, the motion perception module further includes a background modeling unit, which is used to generate a background model, and the background model is transmitted to the motion region segmentation unit, and the motion region segmentation unit identifies the motion region through a background modeling method; The background modeling unit includes a Gaussian mixture model (GMM) and a background updating mechanism. The background updating mechanism dynamically updates the background model according to the real-time changes of the background model.

[0018] The application of the background modeling unit of the present invention enables the system to dynamically identify and distinguish the background from the motion area, thereby improving the accuracy and robustness of the motion target detection, especially in an environment with a complex or changing background.

[0019] Preferably, the motion tracking module further includes a multi-target tracking management unit, which is connected to the motion perception memory selection unit and the Kalman filter unit to transmit tracking management information thereto; The multi-target tracking management unit includes a target ID allocation module and a target association module. The target ID allocation module allocates a unique ID to each target. The target association module associates the target with the ID according to the target's motion trajectory to avoid confusion between targets.

[0020] The application of the multi-target tracking management unit of the present invention enables the system to effectively manage the tracking of multiple targets, avoids confusion between targets, and improves the accuracy and reliability of tracking.

[0021] Preferably, the motion semantic understanding module further includes a motion data preprocessing unit, which is connected to the multimodal data fusion unit; The motion data preprocessing unit includes a noise filtering module and a data alignment module. The noise filtering module removes noise in the motion data, and the data alignment module aligns data of different modes to improve the consistency and comparability of the data.

[0022] The application of the motion data preprocessing unit of the present invention improves the consistency and comparability of data, provides high-quality data input for subsequent analysis and processing, and enhances the accuracy and robustness of the system.

[0023] Beneficial Effects

[0024] The present invention realizes comprehensive capture and in-depth analysis of motion scenes by integrating multiple modules, and significantly improves the accuracy and real-time performance of motion analysis. The system effectively distinguishes different moving targets through an adaptive clustering segmentation unit and a DBSCAN algorithm, and enhances adaptability and robustness. The combination of a motion perception memory selection unit and a Kalman filter unit enables the system to maintain the continuity and accuracy of tracking even when the target moves quickly or is blocked. The multi-target trajectory prediction unit uses an RNN model to predict the future trajectory of the target, which improves the foresight and accuracy of tracking. The combination of a multimodal data fusion unit and a motion video semantic description unit enables the system to generate natural language descriptions and analyze the spatiotemporal relationship in motion videos, thereby improving the system's ability to understand and describe motion scenes. The application of a multi-view fusion unit enables the system to capture motion information from different angles, thereby improving the comprehensiveness and accuracy of motion semantic description. The combination of an action normativeness evaluation unit and a motion intensity quantification unit enables the system to quantitatively analyze motion performance and provide improvement suggestions, thereby improving the scientificity and effectiveness of motion training and evaluation. The application of a background modeling unit enables the system to dynamically identify and distinguish between background and motion areas, thereby improving the accuracy and robustness of motion target detection. The application of the multi-target tracking management unit enables the system to effectively manage the tracking of multiple targets, avoid confusion between targets, and improve the accuracy and reliability of tracking. The application of the motion data preprocessing unit improves the consistency and comparability of the data, provides high-quality data input for subsequent analysis and processing, and enhances the accuracy and robustness of the system. Overall, the system provides a comprehensive, accurate, and efficient solution for real-time multi-person motion analysis through the collaborative work of these modules. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings: Figure 1 is a connection block diagram of the system of the present invention; Figure 2 Flowchart of steps implemented for the motion tracking module; Figure 3 Flowchart of the steps implemented for the motion semantic understanding module. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0027] Example A real-time multi-person motion analysis system based on motion perception and semantic understanding includes a motion perception module, a motion tracking module, a motion semantic understanding module and a motion performance analysis module. The motion perception module is electrically connected to the motion tracking module to achieve real-time data transmission and processing; The motion perception module includes a target detection unit and a motion region segmentation unit. The target detection unit uses a lightweight convolutional neural network to extract image features from a video frame sequence and detect the position and bounding box of the moving target. The motion region segmentation unit uses background modeling or inter-frame difference method to identify the motion region and generate a motion mask. The target detection unit is connected to the central processing unit through a data bus and is used to process video frame data and detect the position and bounding box of the moving target. The motion region segmentation unit is connected to the target detection unit and is used to generate a motion mask.

[0028] The present invention can comprehensively capture and analyze motion data by integrating multiple modules, thereby improving the accuracy and real-time performance of motion analysis. The combination of the target detection unit and the motion region segmentation unit enables the system to accurately identify motion targets and regions, laying a solid foundation for subsequent tracking and analysis.

[0029] In this embodiment, the motion perception module further includes an adaptive clustering segmentation unit, which uses the DBSCAN algorithm to perform clustering segmentation on the moving target to generate multiple moving target regions, and the adaptive clustering segmentation unit processes and segments the moving target regions in real time through data; The target detection unit uses a lightweight convolutional neural network (CNN) to extract image features from a video frame sequence and detect the position and bounding box of a moving target, which is expressed by the following formula: Feature extraction: ; in, is the input video frame, is the extracted feature, CNN stands for convolutional neural network; Bounding Box Detection: ; in, is the bounding box of the detected moving object, represents the detection algorithm; According to the bounding box information, the pixel coordinates are converted into actual physical coordinates or positions in the camera coordinate system to determine the exact position of the moving target in the image; The motion region segmentation unit uses background modeling or inter-frame difference method to identify the motion region and generate a motion mask, which is expressed by the following formula: Motion Region Identification: ; in, is the motion mask, represents the segmentation algorithm, which is based on the bounding box and video frames To identify the motion area; The adaptive clustering segmentation unit uses the DBSCAN algorithm to perform clustering segmentation on the moving target and generate multiple moving target regions. It outputs the clustering result by inputting the motion mask and the position information of the moving target, and segments the moving region into independent moving targets. When the DBSCAN algorithm determines the core points and boundary points, the density of the point P is defined as: ; in, It is the point set of point P in the eps-neighborhood of the motion mask M. If point P is not a core point in the motion mask M, but there is at least one core point in its eps-neighborhood, then P is a boundary point. In addition, if the eps-neighborhoods of two core points overlap, they belong to the same cluster, which is represented in the motion mask M as two moving targets belonging to the same motion event.

[0030] The present invention adds an adaptive clustering segmentation unit, uses the DBSCAN algorithm to cluster and segment moving targets, generates multiple moving target regions, and improves the accuracy of moving target segmentation. Through the application of the adaptive clustering segmentation unit, the system can more effectively handle complex motion scenes, distinguish different moving targets, reduce the problems of misidentification and overlap, and enhance the adaptability and robustness of the system.

[0031] In this embodiment, the motion tracking module includes a motion perception memory selection unit and a Kalman filter unit. The motion perception memory selection unit calculates the IoU (Intersection over Union) score between the mask predicted by the current frame and the mask in the historical frame memory, calculates the target appearance score, calculates the IoU score predicted by the Kalman filter, and selects the historical frame most relevant to the current frame as the memory information, which is specifically obtained by the following formula; ; Among them, Area of ​​Overlap is the area of ​​the overlapping area of ​​the two masks, and Area of ​​Union is the area of ​​the union of the two masks; The motion perception memory selection unit outputs the mask affinity score, the target appearance score and the IoU score predicted by the Kalman filter. The Kalman filter unit uses the Kalman filter to model the target motion trajectory and predict the position and size of the target in the next frame. The state vector of the Kalman filter unit is defined as: ; in, and Represents the coordinates of the center of the bounding box; and Indicates the width and height of the bounding box; , , , The speeds representing the center coordinates, width, and height of the bounding box, respectively; The prediction and update steps of the Kalman filter are as follows: predict: ; in, is the state transition matrix, is the state vector at time t, is the predicted state vector at time t+1 renew: ; ; in, is the Kalman gain, which is used to adjust the predicted state vector to match the observed value; is the prediction covariance matrix at time t; is the observation matrix, which maps the state vector to the observation space; is the observation noise covariance matrix; is the observed value at time t; is the updated state vector at time t+1; The motion perception memory selection unit and the Kalman filter unit realize real-time data processing and generation of tracking results.

[0032] The present invention combines the motion perception memory selection unit and the Kalman filter unit, so that the system can track the motion trajectory of the target in real time, and can maintain the continuity and accuracy of tracking even when the target moves quickly or is blocked; In this embodiment, the motion tracking module further includes a multi-target trajectory prediction unit. The multi-target trajectory prediction unit uses an RNN (recurrent neural network) model to predict the future trajectories of multiple targets based on the prediction results of the Kalman filter to improve the stability and accuracy of tracking. The multi-target trajectory prediction unit predicts and adjusts the motion trajectory of the target in real time; The RNN model structure includes an input layer, an RNN layer and an output layer. The input layer receives the state vector from the Kalman filter unit, including the position, size and speed information. The RNN layer uses LSTM (Long Short-Term Memory Network) as the core of the RNN to process sequence data and capture long-term dependencies. The output layer predicts the position and size of the target in the future frame. During the training phase of the RNN model, historical trajectory data is used to train the RNN model. Specifically, the state vector and observation value of the Kalman filter are used as input, and the target position and size are used as output. During the testing phase, the prediction results of the Kalman filter are used as the input of the RNN. The RNN predicts the future trajectory and feeds the result back to the Kalman filter for state update. The back propagation through time (BPTT) algorithm is then used for training to optimize the weights of the RNN to minimize the prediction error.

[0033] By adding an RNN model, the present invention enables the system to predict the future motion trajectory of the target and respond in advance, thereby improving the foresight and accuracy of tracking, especially in complex dynamic environments.

[0034] In this embodiment, the motion semantic understanding module includes a multimodal data fusion unit, a motion video semantic description unit and a spatiotemporal relationship understanding unit. The multimodal data fusion unit takes motion data and video data as input and adopts a unified multimodal training strategy for training. The motion data is parameterized using the SMPL model, and the video data uses a pre-trained convolutional neural network to extract image features. The multimodal data fusion unit realizes data fusion and training.

[0035] The motion video semantic description unit uses a pre-trained language model to semantically describe the motion video and generates a natural language description. The motion video semantic description unit is connected to the multimodal data fusion unit, and the motion video semantic description unit outputs a semantic description result. The spatiotemporal relationship understanding unit uses the reasoning ability of the language model to analyze the spatiotemporal relationship in the motion video, identify the action sequence, direction change, target intention, etc. The spatiotemporal relationship understanding unit is connected to the motion video semantic description unit, and the spatiotemporal relationship understanding unit outputs the spatiotemporal relationship analysis results; Through multimodal data fusion and semantic description, the present invention can generate natural language descriptions and analyze the spatiotemporal relationships in motion videos, thereby improving the system's ability to understand and describe motion scenes and providing rich semantic information for motion performance analysis.

[0036] In this embodiment, the motion semantic understanding module also includes a multi-perspective fusion unit, which extracts motion data from video streams of multiple cameras, fuses the multi-perspective data using a 3D convolutional neural network, and generates a more comprehensive and accurate motion semantic description. The multi-perspective fusion unit processes and fuses multi-perspective motion data in real time.

[0037] The application of the multi-view fusion unit of the present invention enables the system to capture motion information from different angles, improves the comprehensiveness and accuracy of motion semantic description, and can provide richer motion information, especially in complex scenes.

[0038] In this embodiment, the sports performance analysis module includes an action normativeness evaluation unit, an exercise intensity quantification unit and a sports performance comprehensive evaluation unit. The action normativeness evaluation unit evaluates the normativeness of the action, such as the action amplitude, angle, rhythm, etc., based on the output of the sports semantic understanding module. The action normativeness evaluation unit is connected to the sports semantic understanding module, and the action normativeness evaluation unit outputs the evaluation result. The exercise intensity quantification unit quantitatively analyzes exercise intensity, which includes exercise frequency, speed, and strength. The exercise intensity quantification unit is connected to the exercise tracking module to transmit the quantification results. The exercise performance comprehensive evaluation unit comprehensively evaluates the exercise performance and provides improvement suggestions. The exercise performance comprehensive evaluation unit is connected to the action normative evaluation unit and the exercise intensity quantification unit. The exercise intensity quantification unit outputs the comprehensive evaluation results.

[0039] Through the action normative evaluation and exercise intensity quantification, the system of the present invention can quantitatively analyze the exercise performance and provide improvement suggestions, thereby improving the scientificity and effectiveness of exercise training and evaluation.

[0040] In this embodiment, the motion perception module further includes a background modeling unit, which is used to generate a background model. The background model is transmitted to the motion region segmentation unit, and the motion region segmentation unit identifies the motion region through a background modeling method. The background modeling unit includes a Gaussian mixture model (GMM) and a background updating mechanism. The background updating mechanism dynamically updates the background model according to the real-time changes of the background model.

[0041] The application of the background modeling unit of the present invention enables the system to dynamically identify and distinguish the background from the motion area, thereby improving the accuracy and robustness of the motion target detection, especially in an environment with a complex or changing background.

[0042] In this embodiment, the motion tracking module further includes a multi-target tracking management unit, which is connected to the motion perception memory selection unit and the Kalman filter unit to transmit tracking management information thereto; The multi-target tracking management unit includes a target ID allocation module and a target association module. The target ID allocation module allocates a unique ID to each target. The target association module associates the target with the ID according to the target's motion trajectory to avoid confusion between targets.

[0043] The application of the multi-target tracking management unit of the present invention enables the system to effectively manage the tracking of multiple targets, avoids confusion between targets, and improves the accuracy and reliability of tracking.

[0044] In this embodiment, the motion semantic understanding module further includes a motion data preprocessing unit, which is connected to the multimodal data fusion unit; The motion data preprocessing unit includes a noise filtering module and a data alignment module. The noise filtering module removes noise in the motion data, and the data alignment module aligns data of different modes to improve the consistency and comparability of the data.

[0045] The application of the motion data preprocessing unit of the present invention improves the consistency and comparability of data, provides high-quality data input for subsequent analysis and processing, and enhances the accuracy and robustness of the system.

[0046] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A real-time multi-person motion analysis system based on motion perception and semantic understanding, characterized in that: It includes a motion perception module, a motion tracking module, a motion semantic understanding module and a motion performance analysis module, wherein the motion perception module is electrically connected to the motion tracking module to realize real-time transmission and processing of data; The motion perception module includes a target detection unit and a motion region segmentation unit. The target detection unit uses a lightweight convolutional neural network to extract image features from a video frame sequence and detect the position and bounding box of the moving target. The motion region segmentation unit uses background modeling or inter-frame difference method to identify the motion region and generate a motion mask. The target detection unit is connected to the central processing unit through a data bus, and is used to process the video frame data and detect the position and bounding box of the moving target. The motion region segmentation unit is connected to the target detection unit and is used to generate a motion mask. The motion tracking module includes a motion perception memory selection unit and a Kalman filter unit. The motion perception memory selection unit calculates the IoU (Intersection over Union) score between the mask predicted by the current frame and the mask in the historical frame memory, calculates the target appearance score, calculates the IoU score predicted by the Kalman filter, and selects the historical frame most relevant to the current frame as the memory information, which is specifically obtained by the following formula; ； Among them, Area of ​​Overlap is the area of ​​the overlapping area of ​​the two masks, and Area of ​​Union is the area of ​​the union of the two masks; The motion perception memory selection unit outputs the mask affinity score, the target appearance score and the IoU score predicted by the Kalman filter. The Kalman filter unit uses the Kalman filter to model the target motion trajectory and predict the position and size of the target in the next frame. The state vector of the Kalman filter unit is defined as: ； in, and Represents the coordinates of the center of the bounding box; and Indicates the width and height of the bounding box; , , , The speeds representing the center coordinates, width, and height of the bounding box, respectively; The prediction and update steps of the Kalman filter are as follows: predict: ； in, is the state transition matrix, is the state vector at time t, is the predicted state vector at time t+1 renew: ； ； in, is the Kalman gain, which is used to adjust the predicted state vector to match the observed value; is the prediction covariance matrix at time t; is the observation matrix, which maps the state vector to the observation space; is the observation noise covariance matrix; is the observed value at time t; is the updated state vector at time t+1; The motion perception memory selection unit and Kalman filter unit realize real-time data processing and generation of tracking results; The motion tracking module also includes a multi-target trajectory prediction unit. The multi-target trajectory prediction unit uses an RNN model to predict the future trajectories of multiple targets based on the prediction results of the Kalman filter to improve the stability and accuracy of tracking. The multi-target trajectory prediction unit predicts and adjusts the motion trajectory of the target in real time; The RNN model structure includes an input layer, an RNN layer and an output layer. The input layer receives the state vector from the Kalman filter unit, including the position, size and speed information. The RNN layer uses LSTM as the core of the RNN to process sequence data and capture long-term dependencies. The output layer predicts the position and size of the target in the future frame. During the training phase of the RNN model, historical trajectory data is used to train the RNN model. Specifically, the state vector and observation value of the Kalman filter are used as input, and the target position and size are used as output. During the testing phase, the prediction results of the Kalman filter are used as the input of the RNN. The RNN predicts the future trajectory and feeds the result back to the Kalman filter for state update. After that, the back-propagation algorithm is used for training to optimize the weights of the RNN to minimize the prediction error.

2. A real-time multi-person motion analysis system based on motion perception and semantic understanding according to claim 1, characterized in that: The motion perception module also includes an adaptive clustering segmentation unit, which uses the DBSCAN algorithm to cluster and segment the moving target to generate multiple moving target regions, and the adaptive clustering segmentation unit processes and segments the moving target regions in real time through data; The target detection unit uses a lightweight convolutional neural network to extract image features from a video frame sequence and detect the position and bounding box of a moving target, which is expressed by the following formula: Feature extraction: ; in, is the input video frame, is the extracted feature, CNN stands for convolutional neural network; Bounding Box Detection: ; in, is the bounding box of the detected moving object, represents the detection algorithm; According to the bounding box information, the pixel coordinates are converted into actual physical coordinates or positions in the camera coordinate system to determine the exact position of the moving target in the image; The motion region segmentation unit uses background modeling or inter-frame difference method to identify the motion region and generate a motion mask, which is expressed by the following formula: Motion Region Identification: ; in, is the motion mask, represents the segmentation algorithm, which is based on the bounding box and video frames To identify the motion area; The adaptive clustering segmentation unit uses the DBSCAN algorithm to perform clustering segmentation on the moving target and generate multiple moving target regions. It outputs the clustering result by inputting the motion mask and the position information of the moving target, and segments the moving region into independent moving targets. When the DBSCAN algorithm determines the core points and boundary points, the density of the point P is defined as: ； in, It is the point set of point P in the eps-neighborhood of the motion mask M. If point P is not a core point in the motion mask M, but there is at least one core point in its eps-neighborhood, then P is a boundary point. In addition, if the eps-neighborhoods of two core points overlap, they belong to the same cluster, which is represented in the motion mask M as two moving targets belonging to the same motion event.

3. A real-time multi-person motion analysis system based on motion perception and semantic understanding according to claim 1, characterized in that: The motion semantic understanding module includes a multimodal data fusion unit, a motion video semantic description unit and a spatiotemporal relationship understanding unit. The multimodal data fusion unit takes motion data and video data as input and adopts a unified multimodal training strategy for training. The motion data is parameterized using the SMPL model, and the video data uses a pre-trained convolutional neural network to extract image features. The multimodal data fusion unit realizes data fusion and training. The motion video semantic description unit uses a pre-trained language model to semantically describe the motion video and generates a natural language description. The motion video semantic description unit is connected to the multimodal data fusion unit, and the motion video semantic description unit outputs a semantic description result. The spatiotemporal relationship understanding unit uses the reasoning ability of the language model to analyze the spatiotemporal relationship in the motion video and identify the action sequence, direction change, and target intention. The spatiotemporal relationship understanding unit is connected to the motion video semantic description unit and outputs the spatiotemporal relationship analysis results.

4. A real-time multi-person motion analysis system based on motion perception and semantic understanding according to claim 1, characterized in that: The motion semantic understanding module also includes a multi-view fusion unit, which extracts motion data from video streams of multiple cameras, fuses the multi-view data using a 3D convolutional neural network, and generates a more comprehensive and accurate motion semantic description. The multi-view fusion unit processes and fuses multi-view motion data in real time.

5. The real-time multi-person motion analysis system based on motion perception and semantic understanding according to claim 1, characterized in that: The sports performance analysis module includes a movement normativeness evaluation unit, a movement intensity quantification unit and a sports performance comprehensive evaluation unit. The movement normativeness evaluation unit evaluates the normativeness of the movement based on the output of the movement semantic understanding module, including the movement amplitude, angle and rhythm. The movement normativeness evaluation unit is connected to the movement semantic understanding module, and the movement normativeness evaluation unit outputs the evaluation result. The exercise intensity quantification unit quantitatively analyzes exercise intensity, which includes exercise frequency, speed, and strength. The exercise intensity quantification unit is connected to the exercise tracking module to transmit the quantification results. The exercise performance comprehensive evaluation unit comprehensively evaluates the exercise performance and provides improvement suggestions. The exercise performance comprehensive evaluation unit is connected to the action normative evaluation unit and the exercise intensity quantification unit. The exercise intensity quantification unit outputs the comprehensive evaluation results.

6. A real-time multi-person motion analysis system based on motion perception and semantic understanding according to claim 1, characterized in that: The motion perception module also includes a background modeling unit, which is used to generate a background model. The background model is transmitted to the motion region segmentation unit, and the motion region segmentation unit identifies the motion region through a background modeling method. The background modeling unit includes a Gaussian mixture model and a background updating mechanism. The background updating mechanism dynamically updates the background model according to the real-time changes of the background model.

7. The real-time multi-person motion analysis system based on motion perception and semantic understanding according to claim 1, characterized in that: The motion tracking module also includes a multi-target tracking management unit, which is connected to the motion perception memory selection unit and the Kalman filter unit to transmit tracking management information to them; The multi-target tracking management unit includes a target ID allocation module and a target association module. The target ID allocation module allocates a unique ID to each target. The target association module associates the target with the ID according to the target's motion trajectory to avoid confusion between targets.

8. The real-time multi-person motion analysis system based on motion perception and semantic understanding according to claim 1, characterized in that: The motion semantic understanding module also includes a motion data preprocessing unit, which is connected to the multimodal data fusion unit; The motion data preprocessing unit includes a noise filtering module and a data alignment module. The noise filtering module removes noise in the motion data, and the data alignment module aligns data of different modes to improve the consistency and comparability of the data.

Images