A neural network-based human pose recognition and tracking method

Abstract

This invention belongs to the field of information technology, specifically disclosing a human posture recognition and tracking method based on neural networks. The invention first sets preset scene parameters, then acquires video image data containing human movements through a camera or video file. The video image data is processed using the BlazePose convolutional neural network framework to extract key point data of the human posture. A custom scene algorithm is used to determine whether the result data conforms to the preset scene. The key point data is filtered using a scene parameter-based algorithm to exclude data that does not conform to the preset scene, thus stabilizing the human posture recognition result. Savitzky Golay is used to smooth the output result, smoothing jittery skeletal joint data. The processed human posture data can smoothly and seamlessly drive a virtual character model.

Description

Technical Field

[0001] This invention belongs to the field of information technology, and specifically relates to a human posture recognition and tracking method based on neural networks. Background Technology

[0002] Human pose recognition is a widely used application in scenarios such as virtual character live streaming, wearable motion capture, animation production, and robot driving. By using neural network-based algorithms to analyze and process video footage in real time, it can identify the poses of people in the scene, making human pose recognition easily applicable to the aforementioned scenarios.

[0003] Typically, neural network-based human pose recognition algorithms often output jitter and positional errors in the key skeletal joint data. If this pose recognition data is used directly to drive a human model, it will cause the model to jitter incongruously and even make illogical movements. Therefore, when implementing the algorithm in engineering, it is necessary to correct these jitter errors, which reduces the smoothness of the virtual model's movements. Therefore, we propose a neural network-based human pose recognition and tracking method to solve the above problems, so that the processed human pose data can smoothly and seamlessly drive the virtual human model. Summary of the Invention

[0004] The purpose of this invention is to provide a human posture recognition and tracking method based on neural networks. By using a custom scene algorithm to determine whether the result data conforms to the scene preset, the recognition result of human posture is stabilized by excluding data that does not conform to the preset scene. Then, Savitzky Golay filtering is used to smooth the jittery skeletal joint data. The processed human posture data can smoothly drive the virtual character model, thereby solving the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A human pose recognition and tracking method based on neural networks includes the following steps:

[0007] S1. During system initialization, preset scene parameters are set, including the number of characters, the movement speed of the skeleton key points, the movement range of the key points, and the distance ratio of each key point.

[0008] S2. Acquire video image data containing human movements through a camera or video file;

[0009] S3. Use the BlazePose convolutional neural network framework to calculate and extract human pose key point data from the video image data;

[0010] S4. Filter the key point data using a scene parameter-based algorithm. This mainly includes calculating whether the range of the key point in the image exceeds the preset range, whether the movement speed of the key point exceeds the preset range, and whether the position of the key point relative to other key points is logical.

[0011] S5. Use Savitzky Golay to smooth and filter the output results for subsequent driving of the virtual character model.

[0012] Preferably, the preset scene parameters in step S1 are set by the scene parameter setting module. The scene parameter setting module sets a set of scene parameters for posture key point filtering according to different business needs. If the output result is used to drive a single virtual character model, the number of characters in the scene parameters is 1, and the range of motion speed limits for motion scenes, dancing scenes, and key point motion is high.

[0013] Preferably, the video image data in step S2 is output through the BlazePose neural network framework, which outputs thirty-three pose key points.

[0014] Preferably, the human posture key point data in step S3 is extracted by a human posture recognition module. The human posture recognition module is based on the BlazePose lightweight convolutional neural network framework, which can run on mobile devices and easily recognize and track human posture in real time.

[0015] Preferably, the scene parameter algorithm in step S4 is to filter the output results by the scene algorithm processing module based on preset scene parameters, and exclude results that do not meet the preset scene conditions.

[0016] Preferably, the specific processing flow of the scene algorithm processing module is as follows:

[0017] A1. During module initialization, obtain preset scene parameters, including the number of characters, the movement speed of the skeleton key points, the movement range of the key points, and the distance ratio of each key point.

[0018] A2. Obtain the key point data output by the posture recognition module;

[0019] A3. Calculate whether the position of each key point exceeds the preset range of scene parameters;

[0020] A4. Calculate the movement speed of each key point based on its previous and current positions, and determine whether it exceeds the preset range of scene parameters.

[0021] A5. Calculate the distance between each key point and other key points, and determine whether the ratio of each distance is within the preset range of scene parameters;

[0022] A6. Eliminate results that do not meet the scene parameters and output the results that meet the parameters to the subsequent smoothing filter module.

[0023] Preferably, the scene algorithm processing module calculates the key points of the human skeleton output for each frame of image, filters the key points with reference to preset scene parameters, and excludes key point data that does not meet the preset scene conditions. If the key point data of the current frame is determined to not meet the scene parameters, the data of this frame is discarded, and the position of the target human skeleton is kept as the position of the previous frame.

[0024] Preferably, the method for judging whether the position of the key point is logically consistent with other key points is to calculate whether the x and y coordinates of each key point are within the set range. Assuming the preset range is (rx0, ry0, rx1, ry1), the method is to check whether the key point coordinates (px, py) satisfy (rx0 <= px <= rx1) && (ry0 <= py <= ry1). Before calculating the x and y coordinates of each key point, the center of the left and right shoulders must be defined as the center point of the entire human skeleton.

[0025] Whether the range of key points in the image exceeds the preset range is determined by the distance between the left and right shoulders. The ratio of the distance from each key point to the center point to the distance between the key points on the left and right shoulders is calculated and defined as the key point distance ratio. It is then determined whether the ratio exceeds the preset maximum value.

[0026] To determine whether the movement speed of a keypoint exceeds a preset range, the distance between the skeleton center points of the previous N frames is calculated. The ratio of the positional change of each keypoint across the previous N frames to the unit is then calculated and defined as the keypoint movement speed. This speed is then used to determine whether it exceeds a preset value.

[0027] Preferably, in step S5, the smoothing filter is performed by a smoothing filter module. The smoothing filter module uses a Savitzky Golay low-pass filter to filter high-frequency jitter while preserving output accuracy. It also has good real-time filtering performance and no output lag.

[0028] The human pose recognition and tracking method based on neural networks proposed in this invention has the following advantages compared with existing technologies:

[0029] 1. This invention first sets preset scene parameters, then acquires video image data containing human movements through a camera or video file, uses the BlazePose convolutional neural network framework to calculate the video image data, extracts key point data of human posture, uses a custom scene algorithm to determine whether the result data conforms to the preset scene, uses a scene parameter-based algorithm to filter the key point data, excludes data that does not conform to the preset scene to stabilize the human posture recognition result, and uses Savitzky Golay to smooth the output result. Savitzky Golay filtering smooths the jittery skeletal joint data. The processed human posture data can smoothly and seamlessly drive the virtual character model. Attached Figure Description

[0030] Figure 1 This is a flowchart of the present invention;

[0031] Figure 2 This is a flowchart of the scene algorithm processing module of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The specific embodiments described herein are merely used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] This invention provides, for example Figure 1-2 The human pose recognition and tracking method based on neural networks shown includes the following steps:

[0034] S1. During system initialization, preset scene parameters are set, including the number of characters, the movement speed of the skeleton key points, the movement range of the key points, and the distance ratio of each key point.

[0035] The preset scene parameters are set through the scene parameter setting module. The scene parameter setting module sets a set of scene parameters for posture key point filtering based on different business needs. If the output result is used to drive a single virtual character model, the number of characters in the scene parameters is 1, and the speed limit range for movement scenes, dancing scenes, and key point movement is high.

[0036] S2. Acquire video image data containing human movements through a camera or video file;

[0037] The video image data is output through the BlazePose neural network framework, which outputs thirty-three pose key points.

[0038] S3. Use the BlazePose convolutional neural network framework to calculate and extract human pose key point data from the video image data;

[0039] The human posture key point data is extracted by the human posture recognition module, which is based on the BlazePose lightweight convolutional neural network framework and can run on mobile devices to easily recognize and track human posture in real time.

[0040] S4. Filter the key point data using a scene parameter-based algorithm. This mainly includes calculating whether the range of the key point in the image exceeds the preset range, whether the movement speed of the key point exceeds the preset range, and whether the position of the key point relative to other key points is logical.

[0041] The scene parameter algorithm filters the output results based on preset scene parameters through a scene algorithm processing module, excluding results that do not meet the preset scene conditions.

[0042] like Figure 2 As shown, the specific processing flow of the scene algorithm processing module is as follows:

[0043] A1. During module initialization, obtain preset scene parameters, including the number of characters, the movement speed of the skeleton key points, the movement range of the key points, and the distance ratio of each key point.

[0044] A2. Obtain the key point data output by the posture recognition module;

[0045] A3. Calculate whether the position of each key point exceeds the preset range of scene parameters;

[0046] A4. Calculate the movement speed of each key point based on its previous and current positions, and determine whether it exceeds the preset range of scene parameters.

[0047] A5. Calculate the distance between each key point and other key points, and determine whether the ratio of each distance is within the preset range of scene parameters;

[0048] A6. Eliminate results that do not meet the scene parameters and output the results that meet the parameters to the subsequent smoothing filter module.

[0049] The scene algorithm processing module calculates the key points of the human skeleton output for each frame of image, filters the key points with reference to preset scene parameters, and excludes key point data that does not meet the preset scene conditions. If the key point data of the current frame is determined to not meet the scene parameters, the data of this frame is discarded, and the position of the target human skeleton is kept as the position of the previous frame.

[0050] The method for determining whether the position of a key point relative to other key points conforms to logic is to calculate whether the x and y coordinates of each key point are within a set range. Assuming the preset range is (rx0, ry0, rx1, ry1), the method checks whether the key point coordinates (px, py) satisfy (rx0 <= px <= rx1) && (ry0 <= py <= ry1). This parameter is used to limit the stage area, etc. Before calculating the x and y coordinates of each key point, the exact center of the left and right shoulders must be defined as the center point of the entire human skeleton.

[0051] Whether the range of key points in the image exceeds the preset range is determined by the distance between the left and right shoulders. The ratio of the distance from each key point to the center point to the distance between the key points on the left and right shoulders is calculated and defined as the key point distance ratio. It is then determined whether the ratio exceeds the preset maximum value, where the preset maximum value is the maximum value of the preset ratio when the human body is stretched out.

[0052] Whether the movement speed of the key point exceeds the preset range is determined by calculating the change in distance between the center point of the skeleton in N frames before and after, and using this as the unit to calculate the ratio of the position change of each key point in N frames before and after to the unit, which is defined as the key point movement speed. It is then determined whether this speed exceeds the preset value, where the preset value is the maximum movement speed of each part of the human body that is preset for measurement.

[0053] S5. Use Savitzky Golay to smooth and filter the output results for subsequent driving of the virtual character model. The smoothing and filtering is performed by a smoothing filter module. The smoothing filter module uses Savitzky Golay low-pass filter to filter high-frequency jitter while preserving output accuracy. It also has good real-time filtering performance and no lag in the output results.

[0054] In summary, during system initialization, preset scene parameters are first set. Then, video image data containing human movements is acquired through a camera or video file. The video image data is processed using the BlazePose convolutional neural network framework to extract key point data of human posture. A custom scene algorithm is used to determine whether the result data conforms to the preset scene. The key point data is filtered using a scene parameter-based algorithm to exclude data that does not conform to the preset scene, thereby stabilizing the human posture recognition results. Savitzky Golay is used to smooth the output results. Savitzky Golay filtering smooths the jittery skeletal joint data. The processed human posture data can smoothly and seamlessly drive the virtual character model.

[0055] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A human pose recognition tracking method based on a neural network, characterized in that: It comprises the following steps: S1, when the system is initialized, set the preset scene parameters, the scene parameters include the number of people, the motion speed of the skeleton key points, the motion range of the key points, and the distance proportion of each key point; S2, obtain video image data containing human body motion through a camera or a video file; S3, calculate the video image data using the BlazePose convolutional neural network framework to extract human body posture key point data; S4, filter the key point data using the scene parameter algorithm, mainly including calculating whether the range of the key points in the image exceeds the preset range, whether the motion speed of the key points exceeds the preset range, and whether the position of the key points relative to other key points conforms to the logic; The scene parameter algorithm is to filter the output results according to the preset scene parameters through the scene algorithm processing module, and to exclude results that do not meet the preset scene conditions. The specific processing process of the scene algorithm processing module is as follows: A1, when the module is initialized, obtain the preset scene parameters, including the number of people, the motion speed of the skeleton key points, the motion range of the key points, and the distance proportion of each key point; A2, obtain the key point data output by the posture recognition module; A3, calculate whether the position of each key point exceeds the preset range of the scene parameters; A4, calculate the moving speed of each key point according to the previous position and the current position of the key point, and judge whether it exceeds the preset range of the scene parameters; A5, calculate the distance between each key point and other key points, and judge whether the proportion of each distance is within the preset range of the scene parameters; A6, exclude results that do not meet the scene parameters, and output the conforming results to the subsequent smoothing filtering module; The scene algorithm processing module calculates the human body skeleton key points output by each frame of image, filters the key points according to the preset scene parameters, and excludes key point data that does not meet the preset scene conditions. For example, if the key point data of the current frame is judged as not meeting the scene parameters, the frame data is discarded, and the position of the target human body skeleton is maintained as the position of the previous frame; The method for judging whether the position of the key point relative to other key points conforms to the logic is to calculate whether the x and y coordinates of each key point are within the set range. Assuming that the preset range is (rx0, ry0, rx1, ry1), and the detected key point coordinates are (px, py), whether they satisfy (rx0<=px<=rx1)&&(ry0<=py<=ry1), wherein the x and y coordinates of each key point are calculated by first defining the center of the left and right shoulders as the center point of the entire human body skeleton; Whether the range of the key point in the image exceeds the preset range is to calculate the distance ratio of each key point to the center point to the distance between the left and right shoulder key points, define the distance ratio as the key point distance ratio, and judge whether the ratio exceeds the preset maximum value; Whether the motion speed of the key point exceeds the preset range is to calculate the distance change of the skeleton center point between the previous and subsequent N frames, and to calculate the position change of each key point between the previous and subsequent N frames in units of the distance change, define the speed as the key point moving speed, and judge whether the speed exceeds the preset value; S5, use Savitzky Golay to smooth and filter the output results for subsequent driving of the virtual human model. 2.The human pose recognition and tracking method based on neural network according to claim 1, characterized in that: The preset scene parameter in step S1 is set by a scene parameter setting module. The scene parameter setting module sets a set of posture key point filtering scene parameters according to different service requirements. If the output result is used to drive a single virtual person model, the number of persons in the scene parameter is 1, and the motion scene, dance scene and key point motion speed limit range are high. 3.The human pose recognition and tracking method based on neural network according to claim 1, characterized in that: The video image data in step S2 is output by a BlazePose neural network framework to obtain thirty-three posture key points. 4.The human pose recognition and tracking method based on neural network according to claim 1, characterized in that: The human posture key point data in step S3 is extracted by a human posture recognition module. The human posture recognition module is based on a BlazePose light weight convolutional neural network framework and can be run on a mobile device to easily and timely recognize and track human postures.

5. The human pose recognition and tracking method based on neural network according to claim 1, characterized in that: In step S5, the smoothing filtering is performed by a smoothing filter module. The smoothing filter module filters high frequency jitter by using a SavitzkyGolay low pass filter, can retain output precision, has good filtering real-time performance, and the result output has no delay.

Images