Multi-module collaborative human body posture dynamic monitoring system

Abstract

The invention provides a multi-module collaborative human body posture dynamic monitoring system, which relates to the technical field of human body posture monitoring and comprises a skeleton key point extraction module, a coordinate point conversion module, a posture library construction module, a data labeling module, an MLP model training and tuning module, a model deployment module, an Android end module and a server end reasoning module. The method has the beneficial effects that high-precision recognition of 120 classic postures is realized through cooperation of MediaPipe and MMPose, an elaborately designed rule and strict data processing and model training, the MediaPipe processing speed is high, and the average response time is less than 50ms in combination with model deployment optimization; through complex scene test optimization, the performance is good under the conditions of low illumination, partial shielding and network jitter; the Android end technology saves resources and reduces hardware requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of human posture monitoring, and in particular to a multi-module collaborative human posture dynamic monitoring system. Background Art

[0002] Accurately monitoring human posture is crucial in many areas of modern society, including sports training, rehabilitation medicine, safety monitoring, and human-computer interaction. For example, in sports training, precise posture monitoring can help athletes correct their movements, improve training effectiveness, and reduce injury risks. In rehabilitation medicine, doctors can monitor patients' daily posture to promptly assess their progress and adjust treatment plans. In safety monitoring scenarios, identifying abnormal posture can effectively prevent accidents.

[0003] However, existing human posture monitoring technology relies on complex wearable devices, which brings inconvenience to users and affects normal activities, limiting its application in daily life and some special scenarios. Although vision-based monitoring methods do not require wearable devices, they perform poorly in terms of accuracy, real-time performance, and ability to cope with complex environments. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-module collaborative human posture dynamic monitoring system to solve the problem of poor monitoring effect in complex environments.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] A multi-module collaborative human posture dynamic monitoring system, comprising:

[0007] Skeleton keypoint extraction module: This module uses the MediaPipe module to extract 33 keypoints of the human body in real time. The keypoint coordinates are normalized (x, y, z) and range from [0, 1]. When the keypoint confidence is less than 0.4, it is marked as invalid. In multi-person scenarios, the human instance with the highest confidence is selected as the primary target.

[0008] Coordinate point conversion module: Converts 33 key points of MediaPipe to 17 key points of MMPose through coordinate filtering, relative position calculation, and normalization, reducing the data volume by 48% after conversion.

[0009] Posture library construction module: Builds a posture library containing 120 classic postures. The posture library is divided into three categories based on the visible range of the human body: headshot, half-body, and full-body, and further subdivided. Data collection covers multiple viewing angles, various lighting conditions, and people of different body types.

[0010] Data Annotation Module: This module annotates 50,000 images of single-person human poses into 120 preset pose categories. The annotation process includes pre-annotation and manual annotation, with a verification mechanism for two-person annotation comparison and team leader judgment.

[0011] The MLP model training and tuning module uses the (x, y) coordinates of 17 key points as 34-dimensional basic features, combined with 6-dimensional high-order features such as the left and right elbow flexion angles as input for training. Training strategies include coordinate jittering and partial blocking.

[0012] Model deployment module: uses bilinear interpolation to reconstruct the original resolution and optimizes performance through model warm-up and dynamic batching;

[0013] Android module: Calls the Camera2API to obtain 1080 frames of images, performs black and white processing and checkerboard sampling, and transmits data to the server through a specific transmission protocol;

[0014] Server-side inference module: Receives Android data, restores image resolution, performs skeleton point detection and feature conversion, uses the MLP model for inference, and returns the posture results to the app.

[0015] As an improvement, in the coordinate point conversion module, the eyes and ears of MMPose use the facial key points of MediaPipe, the left and right of MediaPipe are based on the left and right of the person in the image, and the redundant key points of MediaPipe's hands and feet are ignored.

[0016] As an improvement, in the posture library construction module, multi-perspective coverage collects three angles for each posture: front view, side view, and top view. The lighting conditions include normal lighting, backlight, and low light (≥50lux) scenes. The body shape differences cover people with a height of 150-190cm and a BMI of 18-30.

[0017] As an improvement, in the MLP model training and tuning module, the coordinate jitter is increased by ±2% random offset, and 5% key points are randomly zeroed to partially block them.

[0018] As an improvement, in the model deployment module, model preheating is to load the model into the GPU memory when the system starts, and dynamic batching is to merge inference requests accumulated within 5ms.

[0019] As an improvement, in the Android module, black and white processing can save 75% of storage space, and checkerboard sampling only retains odd-numbered column and row pixels.

[0020] The beneficial effects of the present invention are as follows: through the collaboration of MediaPipe and MMPose, carefully designed rules, and strict data processing and model training, high-precision recognition of 120 classic postures is achieved. MediaPipe has a fast processing speed, and combined with model deployment optimization, the average response time is less than 50ms; after optimization through complex scene testing, it performs well in low light, partial occlusion, and network jitter; Android-side technology saves resources and reduces hardware requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a system flow framework diagram of a multi-module collaborative human posture dynamic monitoring system of the present invention. DETAILED DESCRIPTION

[0022] like Figure 1 As shown, a multi-module collaborative human posture dynamic monitoring system includes:

[0023] Skeleton keypoint extraction module: This module uses the MediaPipe module to extract 33 keypoints of the human body in real time. The keypoint coordinates are normalized (x, y, z) and range from [0, 1]. When the keypoint confidence is less than 0.4, it is marked as invalid. In multi-person scenarios, the human instance with the highest confidence is selected as the primary target.

[0024] Coordinate point conversion module: Converts 33 key points of MediaPipe to 17 key points of MMPose through coordinate filtering, relative position calculation, and normalization, reducing the data volume by 48% after conversion.

[0025] Posture library construction module: Builds a posture library containing 120 classic postures. The posture library is divided into three categories based on the visible range of the human body: headshot, half-body, and full-body, and further subdivided. Data collection covers multiple viewing angles, various lighting conditions, and people of different body types.

[0026] Data Annotation Module: This module annotates 50,000 images of single-person human poses into 120 preset pose categories. The annotation process includes pre-annotation and manual annotation, with a verification mechanism for two-person annotation comparison and team leader judgment.

[0027] The MLP model training and tuning module uses the (x, y) coordinates of 17 key points as 34-dimensional basic features, combined with 6-dimensional high-order features such as the left and right elbow flexion angles as input for training. Training strategies include coordinate jittering and partial blocking.

[0028] Model deployment module: uses bilinear interpolation to reconstruct the original resolution and optimizes performance through model warm-up and dynamic batching;

[0029] Android module: Calls the Camera2API to obtain 1080 frames of images, performs black and white processing and checkerboard sampling, and transmits data to the server through a specific transmission protocol;

[0030] Server-side inference module: Receives Android data, restores image resolution, performs skeleton point detection and feature conversion, uses the MLP model for inference, and returns the posture results to the app.

[0031] In the coordinate point conversion module, MMPose's eyes and ears use MediaPipe's facial key points, MediaPipe's left and right are based on the left and right of the person in the image, and MediaPipe's redundant key points of the hands and feet are ignored; in the posture library construction module, multi-perspective coverage collects three angles of view for each posture: front view, side view, and top view. The lighting conditions include normal lighting, backlight, and low light (≥50lux) scenes. The body shape differences cover people with a height of 150-190cm and a BMI of 18-30; in the MLP model training and tuning module, the coordinate jitter increases the random offset by ±2%, and partially blocks 5% of the key points randomly set to zero; in the model deployment module, model preheating is to load the model into the GPU memory when the system starts, and dynamic batching is to merge inference requests accumulated within 5ms; in the Android module, black and white processing can save 75% of storage space, and checkerboard sampling only retains odd-numbered column and row pixels.

[0032] When in use, first start the system, the skeleton key point extraction module starts working immediately, the MediaPipe module processes the input image or video stream frame by frame, quickly detects 33 high-precision human key points, and assigns confidence to each key point. The system monitors the confidence of the key points in real time. Once a key point with a confidence of <0.4 is found, it is immediately marked as invalid. In a multi-person scenario, the system automatically filters out the human body instance with the highest confidence and uses it as the main target for subsequent processing to ensure the accuracy and pertinence of data processing. After receiving the coordinates of the 33 key points extracted by MediaPipe, the coordinate point conversion module converts them into the MMPose17 point format according to the preset mapping rules. First, the coordinates are filtered, and only The (x, y) coordinates of the 17 key points of the target are retained, and the z-axis coordinates are discarded. Then, the offset of each point is calculated with the center point of the eye (the average of the left and right eye coordinates) as the origin. The influence of the different positions of the characters in the picture is eliminated through relative position calculation. Finally, the calculated coordinates are normalized and scaled to the [-1, 1] interval to further eliminate the interference of body shape differences on subsequent calculations. After the conversion is completed, the conversion effect is evaluated and verified through precision loss test and visual verification to ensure the accuracy of the conversion. In the process of constructing the posture library, a large amount of image data of different postures is collected according to the pre-set classification standards. For each posture, it is shot from multiple perspectives (front view, side view, and top view), covering normal lighting, We tested a variety of lighting conditions such as backlight and low light (≥50lux), and selected people of different heights (150-190cm) and BMI (18-30) as the collection objects to ensure the diversity and representativeness of the posture library. After the collection was completed, the data was sorted and classified to build a posture library containing 120 classic postures. The data annotation work was carried out after the posture library was built. First, we entered the pre-annotation stage, and used MediaPipe to automatically generate 33 key points. According to the mapping rules, they were converted into 17 MMPose key points. Then, manual annotation personnel carefully annotated the pictures with key points and divided them into three categories and 120 subcategories. In the annotation process, the inspection mechanism was strictly implemented, and two annotation personnel were responsible for the same A picture is annotated. If the annotation results are different, they will be submitted to the annotation team leader for arbitration to ensure the accuracy and consistency of the annotation data. During the MLP model training phase, the annotated data is divided into training set, validation set and test set according to a certain ratio. The model is trained with the designed 34-dimensional basic features and 6-dimensional high-order features as input. During the training process, coordinate jittering and partial blocking strategies are adopted. Coordinate jittering simulates detection errors by adding ±2% random offsets, and partial blocking randomly sets 5% of key points to zero, so that the model is exposed to various possible interference situations during training, thereby improving the robustness and generalization ability of the model. By continuously adjusting the model parameters, the model performance is evaluated using the validation set, and finally tested on the test set.To ensure the model achieves optimal performance, when deploying the model, the low-resolution image processed by the Android end is first restored using the bilinear interpolation algorithm to reconstruct the original resolution. When the system starts, the trained model is loaded into the GPU memory to preheat the model and speed up the startup of the model. During operation, dynamic batch processing technology is used to merge requests received within 5ms for inference, thereby improving the processing efficiency and response speed of the system. On Android devices, the system calls the Camera2API to obtain 1080 frames of image data. In order to reduce the amount of data and reduce the transmission pressure, the obtained color image is converted to black and white. This operation can save 75% of storage space. Next, a checkerboard sampling method is used to retain only pixels in odd columns and rows, further compressing the data. The processed image data is transmitted to the server via the network according to the defined transmission protocol. After receiving the image data transmitted by the Android client, the server first uses an image restoration algorithm to restore the image to its original resolution. Then, MediaPipe is called to perform full 33-point skeleton point detection, and the detected key points are converted to MMPose17-point format. Finally, the processed data is input into the trained MLP model for inference. The model returns the TOP3 categories and confidence results, and returns the results to the app to achieve human posture recognition and feedback.

[0033] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A multi-module collaborative human posture dynamic monitoring system, characterized in that: include: Skeleton keypoint extraction module: This module uses the MediaPipe module to extract 33 keypoints of the human body in real time. The keypoint coordinates are normalized (x, y, z) and range from [0, 1]. When the keypoint confidence is less than 0.4, it is marked as invalid. In multi-person scenarios, the human instance with the highest confidence is selected as the primary target. Coordinate point conversion module: Converts 33 key points of MediaPipe to 17 key points of MMPose through coordinate filtering, relative position calculation, and normalization, reducing the data volume by 48% after conversion. Posture library construction module: Builds a posture library containing 120 classic postures. The posture library is divided into three categories based on the visible range of the human body: headshot, half-body, and full-body, and further subdivided. Data collection covers multiple viewing angles, various lighting conditions, and people of different body types. Data Annotation Module: This module annotates 50,000 images of single-person human poses into 120 preset pose categories. The annotation process includes pre-annotation and manual annotation, with a verification mechanism for two-person annotation comparison and team leader judgment. The MLP model training and tuning module uses the (x, y) coordinates of 17 key points as 34-dimensional basic features, combined with 6-dimensional high-order features such as the left and right elbow flexion angles as input for training. Training strategies include coordinate jittering and partial blocking. Model deployment module: uses bilinear interpolation to reconstruct the original resolution and optimizes performance through model warm-up and dynamic batching; Android module: Calls the Camera2API to obtain 1080 frames of images, performs black and white processing and checkerboard sampling, and transmits data to the server through a specific transmission protocol; Server-side inference module: Receives Android data, restores image resolution, performs skeleton point detection and feature conversion, uses the MLP model for inference, and returns the posture results to the app.

2. A multi-module collaborative human posture dynamic monitoring system according to claim 1, characterized in that: In the coordinate point conversion module, the eyes and ears of MMPose use the facial key points of MediaPipe, the left and right of MediaPipe are based on the left and right of the person in the image, and the redundant key points of MediaPipe's hands and feet are ignored.

3. The multi-module collaborative human posture dynamic monitoring system according to claim 1, characterized in that: In the posture library construction module, multi-perspective coverage collects three angles for each posture: front view, side view, and top view. The lighting conditions include normal lighting, backlight, and low light (≥50lux) scenes. The body shape differences cover people with a height of 150-190cm and a BMI of 18-30.

4. The multi-module collaborative human posture dynamic monitoring system according to claim 1, characterized in that: In the MLP model training and tuning module, coordinate jittering adds ±2% random offset and partially blocks 5% of key points by random zeroing.

5. The multi-module collaborative human posture dynamic monitoring system according to claim 1, characterized in that: In the model deployment module, model preheating is to load the model into the GPU memory when the system starts, and dynamic batching is to merge inference requests accumulated within 5ms.

6. The multi-module collaborative human posture dynamic monitoring system according to claim 1, characterized in that: In the Android module, black and white processing can save 75% of storage space, and checkerboard sampling only retains odd-numbered column and row pixels.

Images