A single-person pose estimation system based on key point regression and a working method thereof

Abstract

The application belongs to the field of computer artificial intelligence vision application, and particularly discloses a single-person posture estimation system based on key point regression, which comprises an image preprocessing module, a key point regression module, a feature extraction module and a posture estimation module; the image preprocessing module, the key point regression module, the feature extraction module and the posture estimation module are sequentially connected. The single-person posture estimation system based on key point regression has the beneficial effect that the key point position of the human body part in the image is predicted through the pre-trained convolutional neural network key point regression model, and then the effective features are extracted, and the behavior and posture of the person are recognized, so that the behavior safety of the employees in the security field can be ensured.

Description

Technical Field

[0001] This invention belongs to the field of computer artificial intelligence vision applications, specifically involving a single-person pose estimation system based on keypoint regression and its working method. Background Technology

[0002] The rapid development of deep learning technology has driven the continuous improvement of computer vision technology.

[0003] Image-based human behavior recognition is a hot research topic in the field of computer vision. It utilizes image recognition technology to process and analyze the human body in images, and then identifies human postures under various backgrounds. By judging and recognizing human postures, a wealth of useful information can be obtained, with wide-ranging applications. For example, in smart security systems, human posture estimation technology can be used to analyze the behavior of on-site workers, providing safety warnings. In smart homes, recognizing the postures of elderly people can promptly detect falls and alert family members or even contact doctors, effectively mitigating the consequences of accidental falls.

[0004] Currently, human pose estimation has been initially applied to the above-mentioned scenarios to solve the problem of judging the behavior of people in the scenario. However, the high computational complexity of traditional methods is due to factors such as many preprocessing steps and complex postprocessing procedures, which limits the application of human pose estimation.

[0005] Therefore, based on the above problems, this invention provides a single-person pose estimation system based on keypoint regression and its working method. Summary of the Invention

[0006] Purpose of the invention: The purpose of this invention is to provide a single-person pose estimation system and its working method based on keypoint regression, which solves the problems existing in traditional methods in the background technology, and realizes single-person pose estimation with a simpler and more reasonable processing flow and lightweight features.

[0007] Technical solution: The present invention provides a single-person pose estimation system based on keypoint regression, including an image preprocessing module, a keypoint regression module, a feature extraction module, and a pose estimation module; the image preprocessing module, keypoint regression module, feature extraction module, and pose estimation module are connected in sequence.

[0008] In this technical solution, the image preprocessing module performs data preprocessing on the acquired single-person image, scaling the image to a uniform size of 512*512; the keypoint regression module uses a convolutional neural network regression model to perform human keypoint regression on the preprocessed image, obtaining the positions of 11 key points on various parts of the human body; the feature extraction module calculates the Euclidean distance between the 11 key points based on the keypoint positions of various parts of the human body, constructing a 21-dimensional human posture feature; and the posture estimation module uses a shallow neural network classification to classify the extracted human keypoint features, achieving the recognition of three postures in the single-person image.

[0009] In this technical solution, scaling the acquired human posture image and adjusting the image resolution to a uniform size of 512*512 includes scaling the image to a preset width and height, maintaining the original image proportions, filling any insufficient proportions with black, and normalizing the scaled image.

[0010] The present invention provides a working method for a single-person pose estimation system based on keypoint regression, comprising the following steps: (1) The image preprocessing module acquires the human pose image to be identified and preprocesses the image; (2) The keypoint regression module inputs the preprocessed image data into a pre-trained convolutional neural network keypoint regression model to predict the key points of various parts of the human body in the image; (3) The feature extraction module calculates the distance between the key points based on the predicted key points on the human body to construct human pose features; (4) The pose estimation module uses a neural network classifier to classify the constructed human pose features and identify the human pose.

[0011] In this technical solution, the convolutional neural network keypoint regression model is an end-to-end deep neural network model. It is a network model based on the VGG16 network, with the output layer replaced. Specifically, the VGG16 network has 13 convolutional layers, 5 max-pooling layers, 3 fully connected layers, and 1 output layer. The input image passes through 2 convolutional layers, 1 max-pooling layer, then 2 more convolutional layers, 1 more max-pooling layer, then 3 more convolutional layers, 1 more max-pooling layer, and this process is repeated twice more, passing through 3 more convolutional layers, 1 more max-pooling layer, 3 fully connected layers, and finally 1 output layer. The convolutional layers... The convolution process uses a convolution kernel to continuously scan the numerical matrix at each stride. Each scanned value is multiplied by the corresponding value in the convolution kernel, and then the results are summed to generate a new numerical matrix. The max pooling layer is equivalent to a dimensionality reduction operation. After the convolution operation, the extracted feature information will have similar features in adjacent regions, which can be substituted for each other. If all these feature information is retained, there will be information redundancy. The pooling layer continuously reduces the spatial size of the data, and the number of parameters and computational cost will decrease accordingly, which controls overfitting to a certain extent.

[0012] In this technical solution, the output layer provides the result of the convolutional neural network model. The output result is a numerical matrix of size 11*1*2, where 11 is used to predict 11 key points on the human body, and 1*2 is the coordinate (x,y) of the predicted key points on the image.

[0013] In this technical solution, the 11 key points on the human body are the position coordinates of various parts of the human body and are arranged in a certain order. The 1 to 11 key points are the center of the eyebrows, right shoulder, left shoulder, right elbow, left elbow, right hip, left hip, right knee, left knee, right ankle and left ankle.

[0014] In this technical solution, the human posture characteristics are calculated based on 11 key points on the human body. To calculate the distances between these key points, 21 distances need to be calculated: center of the eyebrows to the right shoulder, center of the eyebrows to the left shoulder, center of the eyebrows to the right elbow, center of the eyebrows to the left elbow, center of the eyebrows to the right hip, center of the eyebrows to the left hip, center of the eyebrows to the right knee, center of the eyebrows to the left knee, center of the eyebrows to the right ankle, center of the eyebrows to the left ankle, right elbow to the left elbow, right knee to the left knee, right ankle to the left ankle, right elbow to the right knee, right elbow to the right ankle, right elbow to the left knee, right knee to the left ankle, left elbow to the right knee, left elbow to the right ankle, left elbow to the left knee, and left elbow to the left ankle.

[0015] In this technical solution, the distance between the key points is the Euclidean distance calculated based on the coordinates of the two key points. Taking the distance between the center of the eyebrows and the right shoulder as an example, assuming the coordinates of the center of the eyebrows are ( , The coordinates of the right shoulder are ( , The formula for calculating the distance dist between these two key points is as follows:

[0016] .

[0017] In this technical solution, the human posture classification and recognition constructs a 21-dimensional column vector by calculating the distances of 11 key points, and feeds it into a shallow neural network for human posture recognition; the human posture classification includes standing posture, sitting posture and falling posture.

[0018] Compared with the prior art, the beneficial effect of the single-person pose estimation system based on keypoint regression of the present invention is that: through the pre-trained convolutional neural network keypoint regression model, the key point positions of human body parts in the image are predicted, and then effective features are extracted and human behavior and posture are identified. It can be used in the security field to ensure the safety of practitioners. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the structure of a single-person pose estimation system based on keypoint regression according to the present invention;

[0020] Figure 2 This is a schematic diagram of the process structure of a single-person pose estimation system based on keypoint regression according to the present invention.

[0021] Figure 3 This is a schematic diagram of human key points in a single-person pose estimation system based on key point regression according to the present invention.

[0022] The numbers in the diagram are as follows: 100 - Image preprocessing module, 200 - Keypoint regression module, 300 - Feature extraction module, 400 - Pose estimation module. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] like Figure 1 , Figure 2 and Figure 3 The single-person pose estimation system based on keypoint regression shown includes an image preprocessing module 100, a keypoint regression module 200, a feature extraction module 300, and a pose estimation module 400.

[0025] The image preprocessing module 100, the key point regression module 200, the feature extraction module 300, and the pose estimation module 400 are connected in sequence.

[0026] The image preprocessing module 100 performs data preprocessing on the acquired single-person image and scales the image to a uniform size of 512*512.

[0027] The key point regression module 200 uses a convolutional neural network regression model to perform human key point regression on the preprocessed image, and obtains the positions of 11 key points on various parts of the human body.

[0028] The feature extraction module 300 calculates the Euclidean distance between 11 key points based on the key point locations of various parts of the human body, and constructs 21-dimensional human posture features.

[0029] The pose estimation module 400 uses a shallow neural network to classify the extracted human key point features, thereby recognizing three poses in a single person image.

[0030] The process involves scaling the acquired human pose image to a uniform size of 512*512, including scaling the image to a preset width and height while maintaining the original aspect ratio. Any areas with insufficient aspect ratio are filled with black, and the scaled image is then normalized.

[0031] This invention provides a working method for a single-person pose estimation system based on keypoint regression, comprising the following steps:

[0032] Step (1) The image preprocessing module 100 acquires the human pose image to be identified and preprocesses the image;

[0033] Step (2) The key point regression module 200 inputs the preprocessed image data into the pre-trained convolutional neural network key point regression model to predict the key points of various parts of the human body in the image.

[0034] Step (3) The feature extraction module 300 calculates the distance between the key points predicted on the human body to construct human pose features;

[0035] Step (4) The pose estimation module 400 uses a neural network classifier to classify the constructed human pose features and identify the human pose.

[0036] Among them, the convolutional neural network keypoint regression model is an end-to-end deep neural network model. It is a network model based on the VGG16 network, with the output layer replaced. Specifically, the VGG16 network has 13 convolutional layers, 5 max pooling layers, 3 fully connected layers and 1 output layer. The input image goes through 2 convolutional layers, 1 max pooling layer, then 2 convolutional layers, 1 max pooling layer, then 3 convolutional layers, 1 max pooling layer, then repeats 2 times with 3 convolutional layers, 1 max pooling layer, then 3 fully connected layers, and finally 1 output layer.

[0037] The convolution process of a convolutional layer is to use a convolutional kernel to continuously scan the numerical matrix of each layer with a stride. Each scanned value is multiplied by the number at the corresponding position in the convolutional kernel, and then the results are summed to generate a new numerical matrix.

[0038] Max pooling is equivalent to dimensionality reduction. After convolution, the extracted features are similar in adjacent regions and can be substituted for each other. If all these features are retained, there will be information redundancy. Pooling layers continuously reduce the size of the data space, and the number of parameters and computational cost will decrease accordingly, which controls overfitting to a certain extent.

[0039] The output layer provides the results of the convolutional neural network model. The output result is a numerical matrix of size 11*1*2, where 11 is used to predict 11 key points on the human body, and 1*2 is the coordinates (x,y) of the predicted key points on the image.

[0040] Among them, the 11 key points on the human body are the coordinates of various parts of the human body and are arranged in a certain order. The 1 to 11 key points are the center of the eyebrows, right shoulder, left shoulder, right elbow, left elbow, right hip, left hip, right knee, left knee, right ankle and left ankle.

[0041] Among them, the human posture characteristics are calculated based on 11 key points on the human body. To calculate the distance between the key points, 21 distances need to be calculated: center of the eyebrows to the right shoulder, center of the eyebrows to the left shoulder, center of the eyebrows to the right elbow, center of the eyebrows to the left elbow, center of the eyebrows to the right hip, center of the eyebrows to the left hip, center of the eyebrows to the right knee, center of the eyebrows to the left knee, center of the eyebrows to the right ankle, center of the eyebrows to the left ankle, right elbow to the left elbow, right knee to the left knee, right ankle to the left ankle, right elbow to the right knee, right elbow to the right ankle, right elbow to the left knee, right knee to the left ankle, left elbow to the right knee, left elbow to the right ankle, left elbow to the left knee, and left elbow to the left ankle.

[0042] The distance between key points is calculated based on the Euclidean distance of the coordinates of the two key points. Taking the distance between the center of the eyebrows and the right shoulder as an example, let's assume the coordinates of the center of the eyebrows are ( , The coordinates of the right shoulder are ( , The formula for calculating the distance dist between these two key points is as follows:

[0043] .

[0044] Among them, the human posture classification and recognition constructs a 21-dimensional column vector by calculating the distances of 11 key points, and feeds it into a shallow neural network to recognize human postures; the human posture classification includes standing posture, sitting posture and falling posture. Example

[0045] like Figure 1 , Figure 2 and Figure 3 The present invention illustrates a single-person pose estimation system based on keypoint regression and its working method, which specifically includes the following steps:

[0046] Obtain the human pose image to be identified;

[0047] The acquired human pose image is scaled up to a uniform size of 512*512. The scaling method involves expanding the image's margins and scaling it to a preset width and height. This expansion and scaling will maintain the original image's proportions. If the image's proportions are insufficient, the missing parts are filled with black (0, 0, 0). The scaled image is then normalized.

[0048] The preprocessed image is input into a convolutional neural network regression model, which is derived from the VGG16 network model with a modified output layer. The model outputs a numerical matrix of size 11*1*2, where 11 represents the 11 key points regressed on the human body, and 1*2 records the coordinates (x, y) of these key points on the image. Specifically, the model outputs the position coordinates of key points for various parts of the human body, arranged in a specific order. Key points 1 to 11 are shown in the attached image. Figure 1 The coordinates of each key point are represented by two-dimensional data: the center of the forehead, right shoulder, left shoulder, right elbow, left elbow, right hip, left hip, right knee, left knee, right ankle, and left ankle.

[0049] Based on the coordinates of 11 key points on the human body output by the model, the distances between these key points are calculated. This involves calculating 21 distances: center of the forehead to the right shoulder, center of the forehead to the left shoulder, center of the forehead to the right elbow, center of the forehead to the left elbow, center of the forehead to the right hip, center of the forehead to the left hip, center of the forehead to the right knee, center of the forehead to the left knee, center of the forehead to the right ankle, center of the forehead to the left ankle, right elbow to the left elbow, right knee to the left knee, right ankle to the left ankle, right elbow to the right knee, right elbow to the right ankle, right elbow to the left knee, right knee to the left ankle, left elbow to the right knee, left elbow to the right ankle, left elbow to the left knee, and left elbow to the left ankle. The distances between key points are Euclidean distances calculated based on the coordinates of two key points. Taking the distance between the center of the forehead and the right shoulder as an example, assuming the coordinates of the center of the forehead are (… , The coordinates of the right shoulder are ( , The formula for calculating the distance dist between these two key points is as follows:

[0050] .

[0051] The 21 distances between keypoints calculated using the Euclidean distance formula are used to construct a 1*21 dimensional human pose feature.

[0052] A shallow neural network classifier is used to classify the constructed human posture features and identify the human posture, which includes standing, sitting and falling postures.

[0053] The input to a shallow neural network classifier is a 1*21 dimensional column vector. After multiple layers of nonlinear mapping and combination in the classifier, the output is a 1*3 vector. This vector represents the confidence value of the input vector belonging to the standing, sitting, and falling postures. The category with the highest confidence value is the final posture category.

[0054] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements can be made without departing from the principle of the present invention, and these improvements should also be considered within the scope of protection of the present invention.

Claims

1. A single-person pose estimation system based on keypoint regression, characterized in that: It includes an image preprocessing module (100), a keypoint regression module (200), a feature extraction module (300), and a pose estimation module (400); the image preprocessing module (100), the keypoint regression module (200), the feature extraction module (300), and the pose estimation module (400) are connected in sequence; The working method of a single-person pose estimation system based on keypoint regression includes the following steps: Step (1) The image preprocessing module (100) acquires the human pose image to be identified and preprocesses the image; Step (2) The key point regression module (200) inputs the preprocessed image data into the pre-trained convolutional neural network key point regression model to predict the key points of various parts of the human body in the image; Step (3) Feature extraction module (300) constructs human pose features by calculating the distance between key points predicted on the human body; Step (4) The pose estimation module (400) uses a neural network classifier to classify the constructed human pose features and identify the human pose. The convolutional neural network keypoint regression model is an end-to-end deep neural network model. It is a network model based on the VGG16 network, with the output layer replaced. Specifically, the VGG16 network has 13 convolutional layers, 5 max-pooling layers, 3 fully connected layers, and 1 output layer. The input image passes through 2 convolutional layers, 1 max-pooling layer, then 2 more convolutional layers, 1 more max-pooling layer, then 3 more convolutional layers, 1 more max-pooling layer, and this process is repeated twice more, passing through 3 more convolutional layers, 1 more max-pooling layer, 3 fully connected layers, and finally 1 output layer. The convolution process of the convolutional layers... It uses a convolution kernel to continuously scan the numerical matrix of each layer with a stride. Each scanned value is multiplied by the corresponding number in the convolution kernel, and then the results are summed to generate a new numerical matrix. The operation of the max pooling layer is equivalent to a dimensionality reduction operation. After the convolution operation, the extracted feature information will have similar feature information in adjacent regions, which can be substituted for each other. If all these feature information is retained, there will be information redundancy. The pooling layer continuously reduces the spatial size of the data, and the number of parameters and the amount of computation will decrease accordingly, which controls overfitting to a certain extent. The image preprocessing module (100) preprocesses the acquired single-person image, scaling the image to a uniform size of 512*512. The keypoint regression module (200) uses a convolutional neural network regression model to perform human keypoint regression on the preprocessed image, obtaining the positions of 11 key points on various parts of the human body. The feature extraction module (300) calculates the Euclidean distance between the 11 key points based on the keypoint positions of various parts of the human body, constructing a 21-dimensional human pose feature. The pose estimation module (400) uses a shallow neural network classification to classify the extracted human keypoint features, realizing the recognition of three poses in the single-person image. The acquired human pose image is scaled, and the image resolution is adjusted to a uniform size of 512*512, including scaling the image to a preset size. Width and height are adjusted, and scaling maintains the original image proportions. Areas with insufficient proportions are filled with black, and the scaled image is normalized. For human posture classification and recognition, the distances calculated from 11 key points are used to construct a 21-dimensional column vector, which is then fed into a shallow neural network for human posture recognition. Human posture classification includes standing, sitting, and falling postures. A shallow neural network classifier is used to classify the constructed human posture features, identifying standing, sitting, and falling postures. The input to the shallow neural network classifier is a 1*21-dimensional column vector. After multiple layers of nonlinear mapping and combination within the classifier, the output is a 1*3 vector. This vector represents the confidence value of the input vector belonging to the standing, sitting, or falling posture, with the category with the highest confidence value being the final posture category.

2. The working method of a single-person pose estimation system based on keypoint regression according to claim 1, characterized in that: The output layer provides the result of the convolutional neural network model, which is a numerical matrix of size 11*1*2, where 11 is used to predict 11 key points on the human body, and 1*2 is the coordinates (x,y) of the predicted key points on the image.

3. The working method of a single-person pose estimation system based on keypoint regression according to claim 2, characterized in that: The 11 key points on the human body are the coordinates of various parts of the body and are arranged in a certain order. The 1 to 11 key points are the center of the eyebrows, right shoulder, left shoulder, right elbow, left elbow, right hip, left hip, right knee, left knee, right ankle, and left ankle.

4. The working method of a single-person pose estimation system based on keypoint regression according to claim 1, characterized in that: The human posture features are calculated based on 11 key points on the human body. To calculate the distances between these key points, 21 distances need to be calculated: center of the eyebrows to the right shoulder, center of the eyebrows to the left shoulder, center of the eyebrows to the right elbow, center of the eyebrows to the left elbow, center of the eyebrows to the right hip, center of the eyebrows to the left hip, center of the eyebrows to the right knee, center of the eyebrows to the left knee, center of the eyebrows to the right ankle, center of the eyebrows to the left ankle, right elbow to the left elbow, right knee to the left knee, right ankle to the left ankle, right elbow to the right knee, right elbow to the right ankle, right elbow to the left knee, right knee to the left ankle, left elbow to the right knee, left elbow to the right ankle, left elbow to the left knee, and left elbow to the left ankle.

5. The working method of a single-person pose estimation system based on keypoint regression according to claim 4, characterized in that: The distance between keypoints is calculated based on the Euclidean distance of the coordinates of the two keypoints. Taking the distance between the center of the forehead and the right shoulder as an example, assuming the coordinates of the center of the forehead are (x1, y1) and the coordinates of the right shoulder are (x2, y2), the formula for calculating the distance *dist* between these two keypoints is as follows.

Images