Human Action Recognition Method and System Based on Computer Vision

Abstract

This application relates to the field of computer vision technology, and in particular to a computer vision-based human action recognition method and system. The method includes real-time acquisition of user action videos, decomposing them into continuous video frames; deploying sensors on key human feature points to acquire angle data of all key human feature points in each video frame, including roll angle, yaw angle, pitch angle, angular velocity, and angular acceleration; performing preliminary correction on the image of each video frame based on the roll angle, yaw angle, and pitch angle from the angle data of all key human feature points in each video frame to generate a first corrected image; performing secondary correction on the first corrected image based on the angular velocity and angular acceleration from the angle data of all key human feature points in each video frame to generate a second corrected image; performing action recognition on the second corrected image, and sequentially combining the actions of single frames to output the user's complete action. This application effectively solves the problem of decreased recognition accuracy caused by user posture deviations and dynamic interference during movement, improving the stability and robustness of action recognition.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and system for human motion recognition based on computer vision. Background Technology

[0002] With the continuous maturation of virtual reality technology and computer vision methods, image and video-based behavior understanding has become a core module in interactive systems. Action recognition is a crucial research direction in virtual reality technology and computer vision, widely applied in education and training, medical rehabilitation, and interactive entertainment. In these fields, real-time capture and analysis of user actions not only enhances the immersive experience but also provides quantitative data for teaching assessment and rehabilitation progress monitoring.

[0003] Existing human motion recognition solutions mostly rely on the combination of camera equipment and visual algorithms. Specifically, firstly, a camera captures video of the user's actions, breaks the video down into several frames, and then extracts features from each frame. Using these features, computer vision algorithms can perform motion recognition and classification. In this process, visual information from the image is primarily used to reconstruct and analyze the user's actions, thereby enabling the judgment and recognition of the action state.

[0004] However, existing technologies may fail to accurately reflect a user's actual actions when recognizing user movements, especially when the user's head or body is offset relative to the camera. When the user deviates from the camera's direct line of sight, the captured image will show an angular shift, causing the motion trajectory and posture changes in the image to be inconsistent with reality. This offset makes action recognition based on visual information in the image prone to errors, thus affecting the accuracy of action recognition. Summary of the Invention

[0005] This application provides a computer vision-based human action recognition method and system, which effectively solves the problem of decreased recognition accuracy caused by user posture deviation and dynamic interference during movement, and improves the stability and robustness of action recognition. This application provides the following technical solution:

[0006] In a first aspect, this application provides a human action recognition method based on computer vision, the method comprising:

[0007] The system collects user action videos in real time and decomposes the action videos into several consecutive video frames.

[0008] Sensors are deployed on the user's key human feature points to acquire angle data of all key human feature points in each video frame. The angle data includes roll angle, yaw angle, pitch angle, angular velocity, and angular acceleration.

[0009] Based on the roll angle, yaw angle and pitch angle of all key human feature points in each video frame, the image of each video frame is initially corrected to generate the first corrected image.

[0010] The first corrected image is then corrected a second time based on the angular velocity and angular acceleration of all key human feature points in each video frame to generate the second corrected image.

[0011] Perform single-frame human motion recognition on the second corrected image of each video frame, combine the single-frame motions of all video frames in time sequence, and output the user's complete motion.

[0012] In one specific implementation, the preliminary correction of the image of each video frame based on the roll angle, yaw angle, and pitch angle from the angle data of all key human feature points in each video frame, to generate a first corrected image, includes:

[0013] The sensors at all key human body feature points output a set of three-dimensional Euler angles (Roll). i ,Yaw i Pitch i ), representing the roll, pitch, and yaw angles of the i-th key human feature point in the current frame, respectively. Weights are assigned based on the importance of the human body structure, and the fused Euler angles are obtained through weighted averaging.

[0014]

[0015] Among them, w i Let be the weight of the i-th key human feature point, and n be the number of key human feature points. The average Euler angles, after fusion, represent the posture of the entire human body;

[0016] To compensate for system errors introduced by sensor installation deviations and long-term drift, the pre-obtained offsets ΔRoll, ΔYaw, and ΔPitch are subtracted from the fused Euler angles, as shown below:

[0017]

[0018] in, These are the corrected Euler angles;

[0019] The global correction matrix R is generated using the corrected Euler angles in the order of Tait–Bryan angles. corr ;

[0020] For each pixel x in each video frame, inverse projection and remapping are performed using the inverse of the camera intrinsic matrix and the correction matrix:

[0021]

[0022] Where x′ is the position of the corrected pixel in the output image, and K is the camera intrinsic parameter matrix. The first corrected image is obtained by resampling pixels through bilinear or nearest neighbor interpolation to obtain the inverse of the correction matrix.

[0023] In a specific feasible implementation, the pre-obtained offsets ΔRoll, ΔYaw, and ΔPitch are calculated as follows:

[0024] ΔRoll, ΔYaw, and ΔPitch are calculated based on the angular velocities and angular accelerations collected from the key feature points corresponding to each video frame. Taking ΔRoll as an example, the calculation formula is as follows:

[0025]

[0026] Where N is the sliding window length, i.e., the number of historical frames involved in the calculation. Roll is the roll angle after fusion in frame k. ref For the ideal reference roll angle, These are the average angular velocity and the average angular acceleration corresponding to the k-th frame, respectively.

[0027] In one specific implementation, the global correction matrix R is generated using the corrected Euler angles in the Tait–Bryan angle order. corr include:

[0028]

[0029] In one specific implementation, the step of performing secondary correction on the first corrected image based on the angular velocity and angular acceleration in the angle data of all key human feature points in each video frame to generate a second corrected image includes:

[0030] The full-body fusion dynamic features are obtained by weighting all key human feature points in each video frame according to their importance:

[0031]

[0032] Among them, v k,i a k,i Let w be the angular velocity and angular acceleration of the i-th person's key feature point in the k-th frame, in the form of a three-dimensional vector. i It is the weight of the key feature point of the i-th human body, v k a k These are the fused whole-body angular velocity and angular acceleration vectors, respectively.

[0033] In one specific implementation, the step of performing secondary correction on the first corrected image based on the angular velocity and angular acceleration in the angle data of all key human feature points in each video frame to generate the second corrected image further includes:

[0034] The residual rotation vector θ is calculated using classical kinematic formulas and by introducing a dynamic weighting factor. k :

[0035]

[0036] Where Δt is the time interval between two adjacent frames, and γ k For dynamic equilibrium coefficients, ||a k || represents the norm of the fusion angular acceleration;

[0037] The residual rotation vector θ k Mapped to a 3×3 rotation matrix R using Rodrigues' formula. res,k :

[0038]

[0039] Among them, [θ k ] is the residual rotation vector θ k The antisymmetric matrix, where I is a 3×3 identity matrix;

[0040] Apply residual rotation to the homogeneous coordinate x′ of each pixel in the first corrected image:

[0041]

[0042] Where x″ represents the remapped pixel coordinates. Let R be the rotation matrix. res,k The inverse matrix of the first corrected image is used to resample the pixel values ​​of the first corrected image by performing bilinear interpolation resampling on all remapped pixel coordinates, thereby generating the second corrected image.

[0043] In one specific implementation, the step of performing single-frame human motion recognition on the second corrected image of each video frame, and sequentially combining the single-frame actions of all video frames to output the user's complete actions includes:

[0044] For the second corrected image of each video frame, a pose-aware convolutional neural network is used to perform single-frame human motion recognition.

[0045] The single-frame motion of all video frames is input into a temporal modeling network based on bidirectional gated recurrent units to extract the dynamic evolution relationship between continuous motions;

[0046] Output the user's complete actions.

[0047] Secondly, this application provides a human motion recognition system based on computer vision, which adopts the following technical solution:

[0048] A human motion recognition system based on computer vision includes:

[0049] Thirdly, this application provides an electronic device, the device including a processor and a memory; the memory stores a program, the program being loaded and executed by the processor to implement a computer vision-based human action recognition method as described in the first aspect.

[0050] Fourthly, this application provides a computer-readable storage medium storing a program that, when executed by a processor, is used to implement a computer vision-based human action recognition method as described in the first aspect.

[0051] In summary, the beneficial effects of this application include at least the following:

[0052] (1) Angle sensors are deployed at multiple key human feature points of the user to acquire attitude data such as roll angle, yaw angle, and pitch angle, and this information is used for preliminary correction of video frame images. The overall fused Euler angle is calculated by weighted fusion of angle data from different parts, and then a correction matrix is ​​generated by combining the camera intrinsic parameter matrix. The original video frame is then inversely projected and remapped to generate the first corrected image under a standardized viewpoint. This method can effectively solve the image distortion problem caused by the user's head or body rotation in traditional single-view shooting based on the camera. Compared with methods based solely on head posture correction, this application considers the comprehensive influence of posture changes in multiple parts, which can accurately restore the user's action form under a standard viewpoint, thereby providing a unified and stable input image for the action recognition algorithm and significantly improving the accuracy and robustness of the recognition model.

[0053] (2) To address the subtle jitter that may still exist after the initial correction due to motion inertia, sensor delay, or short-term rotation, this application further utilizes the angular velocity and angular acceleration data of key user parts to perform secondary dynamic correction on the image, generating a second corrected image. By fusing the angular velocity and angular acceleration of key parts, a residual rotation vector is constructed, and it is converted into a rotation matrix using the Rodrigues rotation formula, thereby performing dynamic remapping compensation on each pixel. In particular, the correction process introduces an adjustable dynamic balance factor, which automatically adjusts the weights of angular velocity and acceleration on the correction result according to the actual acceleration, so that the compensation mechanism can adapt to rapid adjustments under violent movements and maintain a smooth transition in slow movements. This pixel-level dynamic compensation mechanism makes the human body structure in the image more stable and natural, effectively eliminating problems such as motion blur, jitter, and tilt, providing high-quality, low-noise image input for subsequent recognition, and is particularly suitable for real-time interactive and behavior analysis scenarios with extremely high stability requirements.

[0054] (3) After obtaining the second corrected image to eliminate the effects of perspective distortion and motion jitter, this application utilizes a pose-aware convolutional neural network to perform single-frame human action recognition on each frame of the image, and strengthens the extraction of key features based on skeleton structure priors or attention mechanisms, thereby recognizing fine-grained actions such as "raising a hand" and "turning around". Furthermore, to restore the user's complete action flow, the system introduces a temporal modeling network based on bidirectional gated recurrent units (Bi-GRU) to learn the dynamic evolution relationship between video frames. The Bi-GRU network can simultaneously capture forward and backward dependencies in the time series, maintaining sensitivity to the logic before and after the action while keeping the structure simple, enabling the system to accurately recognize complex compound actions such as "jumping-landing" and "turning around-bending". Compared with traditional recognition methods based on static frames, this application constructs a complete recognition process from image correction to feature extraction to temporal modeling, which not only significantly improves the accuracy of single-frame action recognition, but also makes the recognition of the entire continuous action more coherent and accurate, and can be widely applied to various application scenarios such as intelligent monitoring, rehabilitation assessment, and human-computer interaction.

[0055] By deploying angle sensors on key parts of the user's body, the system collects roll angle, yaw angle, pitch angle, angular velocity, and angular acceleration for each frame. Combined with video frame images captured by a camera, the system performs viewpoint and dynamic correction in two stages to generate a standardized image sequence free from viewpoint distortion and dynamic jitter. Subsequently, a pose-aware convolutional neural network is used for single-frame action recognition, and frame-level features are integrated through temporal modeling networks such as Bi-GRU to achieve accurate recognition of complete actions. This solution effectively solves the problem of decreased recognition accuracy caused by user posture deviations and dynamic interference during movement, improving the stability and robustness of action recognition. It is particularly suitable for applications requiring high recognition accuracy and response speed, such as real-time interaction and human-computer collaboration.

[0056] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, the preferred embodiments of this application are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating the human motion recognition method based on computer vision in an embodiment of this application.

[0058] Figure 2 This is a schematic diagram of the overall process of the human motion recognition method based on computer vision in the embodiments of this application.

[0059] Figure 3 This is a structural block diagram of a human motion recognition system based on computer vision in an embodiment of this application.

[0060] Figure 4 This is a block diagram of an electronic device for human motion recognition based on computer vision, as described in an embodiment of this application. Detailed Implementation

[0061] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application.

[0062] Optionally, this application uses the computer vision-based human motion recognition method provided in various embodiments as an example for description in electronic devices. The electronic device is a terminal or server. The terminal can be a mobile phone, computer, tablet computer, etc. This embodiment does not limit the type of electronic device.

[0063] Reference Figure 1 This is a flowchart illustrating a computer vision-based human action recognition method according to an embodiment of this application. The method includes at least the following steps:

[0064] Step S101: Real-time acquisition of user action video, and decomposition of the action video into several consecutive video frames.

[0065] In step S101, the user's actions are captured in real time using a high frame rate camera, and the resulting video stream is decomposed into several consecutive video frames in chronological order. Each video frame also carries precise timestamp information to facilitate subsequent synchronization and processing with other sensor data.

[0066] Step S102: Deploy sensors on the user's key human feature points to acquire angle data of all key human feature points in each video frame. The angle data includes roll angle, yaw angle, pitch angle, angular velocity, and angular acceleration.

[0067] In step S102, angle sensors are deployed at multiple key human feature points of the user to acquire posture information of various parts of the body for each video frame. Specifically, in this application, the key human feature points are the head, left and right shoulders, left and right elbows, left and right wrists, torso, and left and right knees. Each sensor has a built-in inertial measurement unit (IMU) that can simultaneously acquire data from a three-axis gyroscope and a three-axis accelerometer. The built-in algorithm, such as existing complementary filtering or extended Kalman filtering, fuses the data to calculate the roll angle, yaw angle, and pitch angle of that part in real time. Furthermore, the built-in algorithm can directly output angular velocity and angular acceleration data based on changes between consecutive frames to describe the dynamic changes of each key part during movement. All sensor data is strictly synchronized with the timestamps of the video frames, ensuring that each video frame corresponds to a complete and accurate set of angle data, thus providing a solid foundation for subsequent data fusion, viewpoint correction, and motion recognition.

[0068] Step S103: Based on the roll angle, yaw angle, and pitch angle of all human key feature points in each video frame, perform preliminary correction on the image of each video frame to generate the first corrected image.

[0069] In step S103, roll angle, yaw angle, and pitch angle data collected from key feature points corresponding to each video frame are used to perform preliminary viewpoint correction on the image to eliminate viewpoint distortion caused by the user's posture deviating from the camera's direct line of sight. Specifically, the sensors output a set of three-dimensional Euler angles (Roll angles) for all key human feature points. i ,Yaw i Pitch i ), representing the roll, pitch, and yaw angles of the i-th key human body feature point in the current frame, respectively. Then, weights are assigned according to the importance of human body structure, and the fused Euler angles are obtained by weighted averaging as follows:

[0070]

[0071] Among them, w i Let be the weight of the i-th key human feature point, and n be the number of key human feature points. The average Euler angles, after fusion, represent the posture of the entire human body.

[0072] Subsequently, to compensate for system errors introduced by sensor installation deviations and long-term drift, the pre-calibrated or online adaptively estimated offsets ΔRoll, ΔYaw, and ΔPitch are subtracted from the fused Euler angles, as shown below:

[0073]

[0074] in, The corrected Euler angles more accurately reflect the user's deviation from the ideal viewing direction. ΔRoll, ΔYaw, and ΔPitch are calibration offsets for different axes, which can be obtained through static calibration or online least-squares optimization.

[0075] In addition, as a preferred method, besides obtaining ΔRoll, ΔYaw, and ΔPitch through static calibration or online least squares optimization, ΔRoll, ΔYaw, and ΔPitch can also be calculated based on the angular velocity and angular acceleration collected from each key feature point corresponding to each video frame. Taking ΔRoll as an example, the calculation formula is as follows:

[0076]

[0077] Where N is the sliding window length, i.e., the number of historical frames involved in the calculation. Roll is the roll angle after fusion in frame k. ref The ideal reference roll angle is usually set to 0 degrees, but the specific setting depends on the camera's installation angle. These represent the average angular velocity and average angular acceleration of the k-th frame, respectively, which are the average angular velocities or angular accelerations of all key human feature points in the k-th frame. α and β are the angular velocity suppression coefficient and angular acceleration suppression coefficient, respectively. In the design of the above formula, the term in the denominator dynamically increases according to the rotation rate and acceleration of the current frame, so that the deviation of that frame is correspondingly reduced when the user moves violently or shakes, avoiding misleading the overall offset estimation due to a single large movement or shaking. Averaging over all frames within the window smooths the impact of slow drift (such as temperature drift) on the offset, ensuring that ΔRoll does not fluctuate violently due to single-frame anomalies, while retaining the ability to track slow changing trends. Combining static Euler angle deviation with dynamic features (velocity, acceleration) for weighted suppression allows the offset estimation to smooth slow drift while automatically ignoring noise from violent movements. Compared to conventional mean or exponential smoothing methods based solely on Euler angles, this formula has stronger robustness in violent motion scenarios, while being more computationally simpler and easier to implement and debug.

[0078] Then, using the corrected Euler angles, the global correction matrix is ​​generated in the order of Tait–Bryan angles:

[0079]

[0080]

[0081] Finally, for each pixel x in each video frame, inverse projection and remapping are performed using the inverse of the camera intrinsic matrix and the correction matrix:

[0082]

[0083] Where x′ is the position of the corrected pixel in the output image, and K is the camera intrinsic parameter matrix, including focal length and principal point offset. This is the inverse of the correction matrix, used to transform the corrected world coordinate system back to the camera coordinate system. The first corrected image is obtained by resampling pixels using bilinear or nearest-neighbor interpolation.

[0084] In the above design, when traditional single-camera video frames are directly fed into the recognition algorithm, even slight deviations in the user's head or body from the viewing direction can cause perspective and angular distortions in the image, affecting the accuracy of feature extraction. By multi-point weighted fusion of Euler angles from key components and constructing an overall correction matrix for inverse mapping of each frame, the viewpoint of all frames can be standardized to a position parallel to the camera's optical axis, significantly eliminating viewpoint distortion caused by posture deviations and ensuring that subsequent motion analysis is based on a consistent viewpoint. During long-term use of inertial sensors, small deviations in installation angle and time drift can cause gradual distortion in posture estimation. By introducing a calibration offset and updating it online, the system can correct sensor errors in real time, ensuring that the first corrected image maintains high accuracy even after long-term operation.

[0085] After completing the above steps, a corresponding first corrected image is generated for each frame of video. The viewpoint of this image has been standardized to match the ideal position of the camera facing the user. Compared to the original frame, the first corrected image effectively eliminates viewpoint distortion caused by the user's head or body deviating from its normal position. This allows subsequent motion analysis based on visual features to be performed from a unified viewpoint, significantly improving recognition accuracy and robustness.

[0086] Step S104: Perform secondary correction on the first corrected image based on the angular velocity and angular acceleration in the angle data of all human key feature points in each video frame to generate the second corrected image.

[0087] In step S104, after obtaining the first corrected image of each video frame, the first corrected image is further dynamically corrected by using the angular velocity and angular acceleration of each human key feature point acquired synchronously, so as to eliminate residual motion jitter and small rotation errors.

[0088] Specifically, firstly, a weighted average is calculated for all key human feature points in each video frame according to their importance, resulting in the following full-body fusion dynamic features:

[0089]

[0090] Among them, v k,i a k,i Let w represent the angular velocity and angular acceleration of the i-th key feature point of the human body in the k-th frame, respectively, both in three-dimensional vector form. i It is the weight of the key feature point of the i-th human body, consistent with that in step S103, v k a k These are the fused whole-body angular velocity and angular acceleration vectors, respectively.

[0091] Then, by applying classical kinematic formulas and introducing a dynamic weighting factor to balance the effects of velocity and acceleration, the residual rotation vector θ is obtained. k:

[0092]

[0093] Where Δt is the time interval between two adjacent frames, and γ k The dynamic balance coefficient is automatically adjusted based on instantaneous acceleration, ensuring that corrections during violent movements rely more on the second-order term of acceleration, while angular velocity is used preferentially for less intense movements; ||a k ‖ represents the norm of the fusion angular acceleration.

[0094] Then the residual rotation vector θ k Mapped to a 3×3 rotation matrix R using Rodrigues' formula. res,k :

[0095]

[0096] Among them, [θ k ] is the residual rotation vector θ k The antisymmetric matrix, I is a 3×3 identity matrix.

[0097] Finally, residual rotation is applied to the homogeneous coordinate x′ of each pixel in the first corrected image:

[0098]

[0099] Where x″ represents the remapped pixel coordinates. Let R be the rotation matrix. res,k The inverse matrix of the first corrected image is used to resample the pixel values ​​of the first corrected image by performing bilinear interpolation resampling on all remapped pixel coordinates, ultimately generating the second corrected image. Because each pixel undergoes pixel-level dynamic compensation based on the current residual rotation trend, the image is more stable and natural, reducing issues such as motion blur and tilt deviation.

[0100] In practice, compared to static correction methods using only angles, this application can identify and compensate for dynamic rotation trends, eliminating residual errors caused by motion inertia or device delays. The proposed residual rotation correction based on angular velocity and angular acceleration modeling can be embedded into existing image processing pipelines, has low computational cost, and is suitable for real-time processing. The introduction of an acceleration term into the residual rotation vector modeling is an innovative extension of the classic attitude calculation model, improving the ability to predict future trends, and belongs to an image correction mechanism that integrates dynamic attitude estimation.

[0101] Step S105: Perform single-frame human motion recognition on the second corrected image of each video frame, combine the single-frame motions of all video frames in sequence, and output the user's complete motion.

[0102] In step S105, after completing the aforementioned angle correction and residual dynamic compensation, a second corrected image corresponding to each video frame has been obtained, which significantly reduces image distortion caused by viewpoint distortion and dynamic jitter, making the human body structural features in the image clearer and more stable.

[0103] In implementation, this step is based on recognizing human movements from the second corrected image, and specifically includes the following two stages:

[0104] First, for the second corrected image of each video frame, a single-frame human action recognition is performed using an existing pose-aware convolutional neural network. This type of network, by introducing prior knowledge of human structure, such as skeleton topology and keypoint-guided attention mechanisms, on top of standard CNNs, enhances the network's response to sensitive areas of human action, thereby improving action recognition accuracy. Preferably, techniques such as Action-CNN or PA-CNN (Pose Attention CNN) can be used. After extracting features through convolution, these features are combined with keypoint heatmaps as guidance to achieve accurate recognition of micro-movements such as "raising a hand," "turning around," and "turning sideways."

[0105] Secondly, to achieve the final recognition of the complete action, the single-frame action feature encoding sequences of all video frames are input into an existing temporal modeling network based on Bidirectional Gated Recurrent Units (Bi-GRU) to extract the dynamic evolutionary relationships between continuous actions. Bi-GRU, as an improved recurrent neural network, can simultaneously capture forward and backward temporal dependencies, making it suitable for analyzing the complex sequential continuity in human movements. Compared to traditional LSTM or TCN structures, Bi-GRU has advantages such as simple structure, high learning efficiency, and strong modeling capabilities for short-term and medium-term action patterns, accurately recognizing complex actions with obvious staged characteristics, such as "jump-land" and "extend-turn-bend."

[0106] Through the two stages described above, not only can key action nodes be accurately identified at the image level, but the complete action sequence can also be reconstructed in the time dimension, ultimately outputting the complete action category or label of the human body, such as "walking," "jumping," and "turning and patting," providing high-quality input for subsequent human-computer interaction, intelligent analysis, and other scenarios. It should be noted that the pose-aware convolutional neural network and the temporal modeling network based on bidirectional gated recurrent units are both existing network models, and will not be described in detail in this application.

[0107] In summary, combining Figure 2This method involves deploying angle sensors on key parts of the user's body to collect roll angle, yaw angle, pitch angle, angular velocity, and angular acceleration for each frame. Combined with video frame images captured by a camera, the images undergo two-stage perspective and dynamic correction to generate a standardized image sequence free from perspective distortion and dynamic jitter. Subsequently, a pose-aware convolutional neural network is used for single-frame action recognition, and frame-level features are integrated through temporal modeling networks such as Bi-GRU to achieve accurate recognition of complete actions. This solution effectively solves the problem of decreased recognition accuracy caused by user posture deviations and dynamic interference during movement, improving the stability and robustness of action recognition. It is particularly suitable for applications requiring high recognition accuracy and response speed, such as real-time interaction and human-computer collaboration.

[0108] Figure 3 This is a structural block diagram of a computer vision-based human motion recognition system provided in one embodiment of this application. The system includes at least the following modules:

[0109] The video capture module is used to capture the user's action video in real time and decompose the action video into several consecutive video frames;

[0110] The angle data acquisition module is used to deploy sensors on the user's key human feature points to acquire angle data of all key human feature points in each video frame. The angle data includes roll angle, yaw angle, pitch angle, angular velocity and angular acceleration.

[0111] The first correction module is used to perform preliminary correction on the image of each video frame based on the roll angle, yaw angle and pitch angle in the angle data of all key human feature points in each video frame, and generate the first corrected image.

[0112] The second correction module is used to perform secondary correction on the first correction image based on the angular velocity and angular acceleration in the angle data of all key human feature points in each video frame, and generate the second correction image.

[0113] The motion recognition module is used to perform single-frame human motion recognition on the second corrected image of each video frame, and to combine the single-frame motions of all video frames in sequence to output the user's complete motion.

[0114] For relevant details, please refer to the above method implementation examples.

[0115] Figure 4 This is a block diagram of an electronic device provided in one embodiment of this application. The device includes at least a processor 401 and a memory 402.

[0116] Processor 401 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 401 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 401 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 401 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 401 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0117] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 402 are used to store at least one instruction, which is executed by the processor 401 to implement the computer vision-based human action recognition method provided in the method embodiments of this application.

[0118] In some embodiments, the electronic device may also optionally include: a peripheral device interface and at least one peripheral device. The processor 401, memory 402, and peripheral device interface can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface via a bus, signal line, or circuit board. Indicatively, peripheral devices include, but are not limited to: radio frequency circuitry, a touch display screen, audio circuitry, and a power supply.

[0119] Of course, electronic devices may also include fewer or more components, and this embodiment does not limit this.

[0120] Optionally, this application also provides a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the computer vision-based human action recognition method of the above-described method embodiments.

[0121] Optionally, this application also provides a computer product including a computer-readable storage medium storing a program, which is loaded and executed by a processor to implement the computer vision-based human motion recognition method of the above method embodiments.

[0122] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A computer vision-based human motion recognition method, characterized by, The method comprises: real-time acquisition of user action video, decomposing the action video into a plurality of continuous video frames; deploying sensors on the key feature points of the user's body to obtain angle data of all key feature points of each video frame, the angle data including roll angle, yaw angle, pitch angle, angular velocity and angular acceleration; based on the roll angle, yaw angle and pitch angle in the angle data of all key feature points of each video frame, performing preliminary correction on the image of each video frame to generate a first corrected image, including: The sensor outputs a set of three-dimensional Euler angles of all human key feature points , respectively representing the roll, pitch, and yaw angles of the personal key feature points in the current frame The personal key feature points in the current frame are weighted and averaged to obtain the fused Euler angles according to the importance of the human body structure wherein, is the first is the second is the weight of the i-th key feature point of the human body, is the number of key feature points of the human body, is the average Euler angle representing the whole human body pose after fusion; is the offset obtained in advance to compensate for the system error introduced by the sensor installation deviation and long-term drift, which is subtracted from the fusion Euler angle , as follows: wherein, , , are the corrected Euler angles; Generating an overall correction matrix in Tait-Bryan angle order using the corrected Euler angles ; For each pixel point in each video frame inverse projection and remapping are performed by using the camera intrinsic matrix and the inverse of the correction matrix: wherein, is a camera intrinsic matrix, is a camera intrinsic matrix, is an inverse matrix of the rectification matrix, pixel resampling is completed by bilinear or nearest neighbor interpolation to obtain a first rectified image; based on the angular velocity and angular acceleration in the angle data of all key feature points of each video frame, performing secondary correction on the first corrected image to generate a second corrected image, including: weighting and averaging all key feature points of each video frame according to importance weight to obtain whole-body fusion dynamic features as follows: wherein, , are the first and second frame of the first and second body key feature points respectively, are the first and second frame of the first and second body key feature points respectively, are the angular velocity and angular acceleration of the first and second body key feature points respectively in three-dimensional vector form, is the weight of the first body key feature point, is the weight of the second body key feature point, , are the fused angular velocity and angular acceleration vectors respectively; the residual rotation vector is calculated using classical kinematic formulas and introducing dynamic weighting factors : 其 In, is the time interval between two adjacent frames, is the dynamic balance coefficient, is the norm of the fused angular acceleration; the residual rotation vector is mapped to : wherein is the skew-symmetric matrix of the residual rotation vector , is the unit matrix of . for each pixel in the first corrected image Apply residual rotation: ； wherein, is the inverse matrix of is by bilinearly interpolating and resampling the pixel values of the first rectified image for all the remapped pixel coordinates, a second rectified image is generated. performing single-frame human action recognition on the second corrected image of each video frame, and time-series combining single-frame actions of all video frames to output complete action of the user. 2.The computer vision-based human motion recognition method of claim 1, wherein, The pre-obtained offset , and The calculation method is as follows: The angular velocity and angular acceleration are calculated based on the key feature points corresponding to each video frame. , , ,by For example, the calculation formula is as follows: in, The sliding window length is the number of historical frames used in the calculation. For the first Roll angle after frame fusion For the ideal reference roll angle, , The first The average value of the angular velocity and the average value of the angular acceleration corresponding to the frame. 3.The computer vision-based human motion recognition method of claim 1, wherein, The overall correction matrix is generated in Tait-Bryan angle order using the corrected Euler angles comprising: 。 4.The computer vision-based human motion recognition method of claim 1, wherein, The single-frame human action recognition on the second corrected image of each video frame and the time-series combining of single-frame actions of all video frames to output complete action of the user comprises: for the second corrected image of each video frame, using a posture perception type convolutional neural network to perform single-frame human action recognition; inputting single-frame actions of all video frames into a time-series modeling network based on a bidirectional gated recurrent unit to extract dynamic evolution relationship between continuous actions; outputting complete action of the user.

5. A computer vision based human action recognition system, characterized in that, It comprises: a video acquisition module for real-time acquisition of user action video, decomposing the action video into a plurality of continuous video frames; an angle data acquisition module for deploying sensors on the key feature points of the user's body to obtain angle data of all key feature points of each video frame, the angle data including roll angle, yaw angle, pitch angle, angular velocity and angular acceleration; a first correction module for performing preliminary correction on the image of each video frame based on the roll angle, yaw angle and pitch angle in the angle data of all key feature points of each video frame to generate a first corrected image, including: The sensor outputs a set of three-dimensional Euler angles for all the key feature points of the human body , respectively representing the roll, pitch, and yaw angles of the key feature points of the human body in the current frame The roll, pitch, and yaw angles of the key feature points of the human body in the current frame are weighted and averaged to obtain the fused Euler angles wherein, is the first is the weight of the personal body key feature point, is the number of the personal body key feature points, , , is the average Euler angle representing the whole body pose after fusion; is the offset obtained in advance to compensate for the system error introduced by the sensor installation deviation and long-term drift, which is subtracted from the fusion Euler angle , and as follows: wherein, , , are the corrected Euler angles; Generating an overall correction matrix in Tait-Bryan angle order using the corrected Euler angles ; For each pixel point in each video frame inverse projection and remapping are performed by using the camera intrinsic matrix and the inverse of the correction matrix: wherein, is a camera intrinsic matrix, is a camera intrinsic matrix, is an inverse matrix of the rectification matrix, and pixel resampling is completed by bilinear or nearest neighbor interpolation to obtain a first rectified image; a second correction module for performing secondary correction on the first corrected image based on the angular velocity and angular acceleration in the angle data of all key feature points of each video frame to generate a second corrected image, including: weighting and averaging all key feature points of each video frame according to importance weight to obtain whole-body fusion dynamic features as follows: wherein, , are the first frame the second angular velocity and angular acceleration of the key feature points of the individual in the form of three-dimensional vectors, are the first weight of the key feature points of the individual, , are the fused angular velocity and angular acceleration vectors of the whole body, respectively; the residual rotation vector is calculated using classical kinematic formulas and introducing a dynamic weighting factor wherein, is the time interval between two adjacent frames, is the dynamic balance coefficient, is the norm of the fused angular acceleration; the residual rotation vector is mapped to : wherein is the skew-symmetric matrix of the residual rotation vector , is the identity matrix of . for each pixel in the first corrected image Apply residual rotation: ； wherein, is the inverse matrix of is by bilinearly interpolating and resampling the pixel values of the first rectified image for all the remapped pixel coordinates, a second rectified image is generated. an action recognition module for performing single-frame human action recognition on the second corrected image of each video frame, and time-series combining single-frame actions of all video frames to output complete action of the user.

6. An electronic device, comprising: The device comprises a processor and a memory; the memory stores a program, which is loaded and executed by the processor to implement the computer vision-based human action recognition method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The storage medium stores a program, which is executed by the processor to implement the computer vision-based human action recognition method according to any one of claims 1 to 4.

Images