A low-cost real-time skeleton key point recognition method and device

Abstract

The invention discloses a low-cost real-time skeleton key point recognition device, which comprises an image acquisition module, a core computing unit, a lightweight neural network algorithm module, a neural network acceleration engine and a skeleton key point output module, wherein the image acquisition module acquires images and sends the acquired image information to the core computing unit; the core computing unit performs image processing on the acquired images, the lightweight neural network algorithm module adopts an improved SqueezeNet as a basic backbone network, combines a feature pyramid network to perform multi-scale feature extraction on the image, accelerates the network through the neural network acceleration module, and finally outputs the skeleton key points through the skeleton key point output module, the image acquisition module adopts any monocular camera, and the core computing unit adopts a mobile terminal CPU.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a low-cost real-time skeleton key point recognition method and device. Background Art

[0002] Skeleton key point recognition technology is one of the basic technologies of computer vision. This technology detects the joints and facial features of the human body in image / video data through sensors (cameras, infrared devices, etc.) and describes the human skeleton information through key points.

[0003] However, most of the existing skeleton key point recognition algorithms based on deep learning are difficult to run in real time on low-cost hardware platforms, and high-cost hardware (such as GPU or high-end camera) needs to be matched to achieve real-time. The current invention is based on a series of software and hardware optimization technologies, which can be implemented on a low-cost hardware platform to complete the real-time recognition of skeleton key points. Under the premise of ensuring accuracy, compared with the recognition delay of about 50 milliseconds introduced by the mainstream method, the current invention technology can achieve a minimum recognition delay of 3 milliseconds.

[0004] Traditional skeleton key point algorithms are based on template matching based on geometric priors, and have poor accuracy. However, due to hardware performance limitations, existing skeleton key point recognition algorithms based on deep learning have slow recognition speeds on low-cost hardware platforms (such as mobile phones and tablets). Algorithm linkage applications can cause application freezes and frame drops, which greatly affects user experience. Summary of the invention

[0005] Based on the above shortcomings, the present invention is mainly aimed at the recognition of skeletal key points of mobile / embedded devices, adopts a lightweight deep learning algorithm and applies neural network acceleration technology for optimization. The hardware only needs to use a CPU and a monocular camera to complete low-cost real-time recognition of skeletal key points, without the need for a GPU or a high-end camera (kinect).

[0006] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention discloses a low-cost real-time skeleton key point recognition device, the recognition device includes an image acquisition module, a core computing unit, a lightweight neural network algorithm module, a neural network acceleration engine and a skeleton key point output module, wherein the image acquisition module acquires images and sends the acquired image information to the core computing unit; the core computing unit performs image processing on the acquired images, the lightweight neural network algorithm module uses an improved SqueezeNet as a basic backbone network, combines a feature pyramid network to perform multi-scale feature extraction on the image, accelerates the network through the neural network acceleration module, and finally outputs the skeleton key points through the skeleton key point output module, the image acquisition module uses any monocular camera, and the core computing unit uses a mobile terminal CPU.

[0007] Furthermore, the neural network acceleration module further includes: the input image first enters the improved SqueezeNet backbone network for calculation, the SqueezeNet backbone network consists of two convolutional layers, eight Fire layers and three pooling layers, wherein the convolutional layer conv1 passes through 96 groups of 7x7 convolution kernels, and the convolutional layer conv10 passes through 1000 groups of 1x1 convolution kernels; the pooling layers maxpool1, maxpool2, and maxpool3 are all 3x3 in size; the fire layer structure is unified, wherein the Squeeze part is composed of continuous 1x1 convolutions, and the Expand part is composed of continuous 1x1 convolutions and 3x3 convolutions.

[0008] Furthermore, if the number of channels in the squeeze part is recorded as S1x1, and the number of 1x1 convolutions and 3x3 convolutions in the expand part are recorded as e1x1 and e3x3 respectively, then the relationship between the above three satisfies the following formula: S1×1 <e1×1+e3×3。

[0009] Furthermore, the neural network acceleration module also adds network shortcuts between fire3, fire5, fire7 and fire9 layers to improve robustness during training and prevent overfitting. The improved SqueezeNet backbone network outputs a series of convolutional feature maps to the feature pyramid network for the next step of calculation.

[0010] Furthermore, the four scales of fire2, fire6, fire5 and conv10 in the backbone network SqueezeNet are extracted to construct a feature pyramid network. After being processed by the feature pyramid network, the predicted results are output at each scale.

[0011] Furthermore, the feature pyramid network further includes: the predicted result includes two parts: heatmap and vector field (paf), and the model finally obtains the output result of the skeleton key point by integrating the prediction results of heatmap and paf at multiple scales. If there are n scales of prediction results in the end, the final prediction results of heatmap and paf are Fheatmap and Fpaf respectively expressed by the following formulas:

[0012]

[0013]

[0014] Among them, vh_i and vp_i are the heatmap and paf values ​​output at each scale respectively.

[0015] Furthermore, the neural network acceleration module further includes: a quantization and acceleration engine, wherein the quantization is to convert the data in FP32 format into INT8 format during the inference process for the data in FP32 format used in the model training process, and the acceleration engine is to accelerate the model inference process based on the OpenCL framework to further improve the real-time performance of the skeleton key point recognition process.

[0016] The present application also discloses a low-cost real-time skeletal key point recognition method, which uses the above-mentioned device to collect image data, and performs image processing on the acquired image information. By using an improved SqueezeNet as the basic backbone network and combining it with a feature pyramid network, multi-scale feature extraction is performed on the image. After processing by the feature pyramid network, the predicted results are output at each scale respectively. By quantizing and accelerating the data during the reasoning process, the skeletal key point information of the collected image is finally output in real time.

[0017] Compared with the prior art, the present invention has the following beneficial effects: the invention is mainly aimed at the recognition of skeleton key points of mobile / embedded devices, adopts a lightweight deep learning algorithm and applies neural network acceleration technology for optimization, and the hardware only needs to use a CPU and a monocular camera to complete low-cost real-time recognition of skeleton key points, without the need for a GPU or a high-end camera. It uses an acceleration engine of a quantization and reasoning network, and under the premise of ensuring accuracy, compared with the recognition delay of about 50 milliseconds introduced by the mainstream method, the current invention technology can achieve a minimum recognition delay of 3 milliseconds. While reducing the hardware cost, the real-time nature of the recognition speed is guaranteed by the above technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present invention can be further understood from the following description in conjunction with the accompanying drawings. The components in the figures are not necessarily drawn to scale, but the emphasis is placed on illustrating the principles of the embodiments. In the figures, the same reference numerals designate corresponding parts in different views.

[0019] Figure 1 It is a module diagram of the core module of the real-time skeleton recognition method and device of the present invention;

[0020] Figure 2 is a structural diagram of the overall structure of a lightweight network in one embodiment of the present invention;

[0021] Figure 3 is a structural diagram of a SqueezeNet backbone network in one embodiment of the present invention;

[0022] Figure 4 is a structural diagram of a fire layer structure in one embodiment of the present invention;

[0023] Figure 5 It is a structural diagram of a feature pyramid network structure in one embodiment of the present invention. DETAILED DESCRIPTION

[0024] Embodiment 1

[0025] The core modules of the low-cost real-time skeleton recognition method and device provided by the present invention are as follows: Figure 1 As shown, it includes an image acquisition module, a core computing unit, a lightweight neural network algorithm module, a neural network acceleration engine and a skeleton key point output module. The image acquisition module uses any monocular camera, and the core computing unit uses a mobile terminal CPU. The core design of the present invention is a lightweight neural network algorithm module and a neural network acceleration module, which are used to ensure the real-time performance of the system on low-cost hardware.

[0026] Among them, lightweight neural network algorithm module:

[0027] The lightweight neural network algorithm module uses the improved SqueezeNet as the basic backbone network and combines the Feature Pyramid Networks (FPN) for multi-scale feature extraction to improve accuracy and speed. The overall structure of the lightweight network is as follows: Figure 2 shown.

[0028] The input image first enters the improved SqueezeNet backbone network for calculation. The SqueezeNet backbone network consists of two convolutional layers, eight Fire layers, and three pooling layers, as shown in Figure 3As shown in the figure, the convolution layer conv1 passes through 96 groups of 7x7 convolution kernels (with a step size of 2), and the convolution layer conv10 passes through 1000 groups of 1x1 convolution kernels (with a step size of 1); the pooling layers maxpool1, maxpool2, and maxpool3 are all 3x3 in size with a step size of 2; the fire layer structure is unified, and its structure is as follows Figure 4 As shown in Figure 1, the Squeeze part consists of continuous 1x1 convolutions, while the Expand part consists of continuous 1x1 convolutions and 3x3 convolutions. If the number of channels in the Squeeze part is denoted by S 1x1 , and the number of 1x1 convolutions and 3x3 convolutions in the Expand part are recorded as e 1x1 and e 3x3 , then the relationship between the above three must satisfy the following formula:

[0029] S 1×1 <e 1×1 +e 3×3

[0030] In addition, the improved SqueezeNet backbone network also adds network shortcuts between fire3, fire5, fire7 and fire9 layers to improve robustness during training and prevent overfitting. Finally, the backbone network outputs a series of convolutional feature maps to the feature pyramid network for the next step of calculation.

[0031] The feature pyramid network structure is as follows Figure 5 As shown in the figure, this method extracts four scales of fire2, fire6, fire5 and conv10 from the backbone network SqueezeNet to construct a feature pyramid network. After being processed by the feature pyramid network, the predicted results are output at each scale. The predicted results include two parts: heatmap and vector field (paf). Combining the prediction results of heatmap and paf at multiple scales, the model finally obtains the output results of the skeleton key points. If there are n scales of prediction results in the end, the final prediction results of heatmap and paf F heatmap and F paf They are expressed as follows:

[0032]

[0033]

[0034] Among them, vh_i and vp_i are the heatmap and paf values ​​output at each scale respectively.

[0035] Among them, the neural network acceleration module:

[0036] In this recognition method, a neural network acceleration module is used to accelerate the network. This module is divided into two parts: quantization and the use of an acceleration engine. Quantization means that the data is in FP32 format during model training, and the FP32 format data is converted to INT8 format during the inference process, which greatly improves the model's inference speed while basically ensuring the model's accuracy, in order to obtain better real-time performance. The use of an acceleration engine means that the model inference process of this method is accelerated based on the OpenCL framework, further improving the real-time performance of the skeleton key point recognition process.

[0037] Embodiment 2

[0038] The present embodiment provides a low-cost real-time skeleton key point recognition device, which includes an image acquisition module, a core computing unit, a lightweight neural network algorithm module, a neural network acceleration engine and a skeleton key point output module, wherein the image acquisition module acquires images and sends the acquired image information to the core computing unit; the core computing unit performs image processing on the acquired images, the lightweight neural network algorithm module adopts an improved SqueezeNet as a basic backbone network, combines a feature pyramid network to perform multi-scale feature extraction on the image, accelerates the network through the neural network acceleration module, and finally outputs the skeleton key points through the skeleton key point output module, the image acquisition module adopts any monocular camera, and the core computing unit adopts a mobile terminal CPU.

[0039] Furthermore, the neural network acceleration module further includes: the input image first enters the improved SqueezeNet backbone network for calculation, the SqueezeNet backbone network consists of two convolutional layers, eight Fire layers and three pooling layers, wherein the convolutional layer conv1 passes through 96 groups of 7x7 convolution kernels, and the convolutional layer conv10 passes through 1000 groups of 1x1 convolution kernels; the pooling layers maxpool1, maxpool2, and maxpool3 are all 3x3 in size; the fire layer structure is unified, wherein the Squeeze part is composed of continuous 1x1 convolutions, and the Expand part is composed of continuous 1x1 convolutions and 3x3 convolutions.

[0040] Furthermore, if the number of channels in the squeeze part is recorded as S1x1, and the number of 1x1 convolutions and 3x3 convolutions in the expand part are recorded as e1x1 and e3x3 respectively, then the relationship between the above three satisfies the following formula: S1×1 <e1×1+e3×3。

[0041] Furthermore, the neural network acceleration module also adds network shortcuts between fire3, fire5, fire7 and fire9 layers to improve robustness during training and prevent overfitting. The improved SqueezeNet backbone network outputs a series of convolutional feature maps to the feature pyramid network for the next step of calculation.

[0042] Furthermore, the four scales of fire2, fire6, fire5 and conv10 in the backbone network SqueezeNet are extracted to construct a feature pyramid network. After being processed by the feature pyramid network, the predicted results are output at each scale.

[0043] Furthermore, the feature pyramid network further includes: the predicted result includes two parts: heatmap and vector field (paf), and the model finally obtains the output result of the skeleton key point by integrating the prediction results of heatmap and paf at multiple scales. If there are n scales of prediction results in the end, the final prediction results of heatmap and paf are Fheatmap and Fpaf respectively expressed by the following formulas:

[0044]

[0045]

[0046] Among them, vh_i and vp_i are the heatmap and paf values ​​output at each scale respectively.

[0047] Furthermore, the neural network acceleration module further includes: a quantization and acceleration engine, wherein the quantization is to convert the data in FP32 format into INT8 format during the inference process for the data in FP32 format used in the model training process, and the acceleration engine is to accelerate the model inference process based on the OpenCL framework to further improve the real-time performance of the skeleton key point recognition process.

[0048] The present application also discloses a low-cost real-time skeletal key point recognition method, which uses the above-mentioned device to collect image data, and performs image processing on the acquired image information. By using an improved SqueezeNet as the basic backbone network and combining it with a feature pyramid network, multi-scale feature extraction is performed on the image. After processing by the feature pyramid network, the predicted results are output at each scale respectively. By quantizing and accelerating the data during the reasoning process, the skeletal key point information of the collected image is finally output in real time.

[0049] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0050] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0051] Although the present invention has been described above with reference to various embodiments, it should be understood that many changes and modifications can be made without departing from the scope of the present invention. Therefore, it is intended that the above detailed description is considered to be illustrative rather than restrictive, and it should be understood that the following claims (including all equivalents) are intended to limit the spirit and scope of the present invention. The above embodiments should be understood to be only used to illustrate the present invention and not to limit the scope of protection of the present invention. After reading the content of the record of the present invention, the technician can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall into the scope defined by the claims of the present invention.

Claims

1. A low-cost real-time skeleton key point recognition device, characterized in that: The recognition device includes an image acquisition module, a core computing unit, a lightweight neural network algorithm module, a neural network acceleration module, and a skeletal key point output module. Among them, the image acquisition module acquires an image and sends the obtained image information to the core computing unit; the core computing unit performs image processing on the acquired image. The lightweight neural network algorithm module uses an improved SqueezeNet as the basic backbone network, combines a feature pyramid network to perform multi-scale feature extraction on the image, accelerates the network through the neural network acceleration module, and finally outputs skeletal key points through the skeletal key point output module. The image acquisition module uses any monocular camera, and the core computing unit uses a mobile CPU. Among them, the neural network acceleration module further includes: The input image first enters the improved SqueezeNet backbone network for calculation. The improved SqueezeNet backbone network consists of two convolutional layers, eight Fire layers, and three pooling layers. The two convolutional layers are conv1 and conv10, the eight Fire layers are fire2, fire3, fire4, fire5, fire6, fire7, fire8, fire9, and the three pooling layers are maxpool1, maxpool2, maxpool3. Among them, the conv1 layer of the convolutional layer passes through 96 groups of 7x7 convolutional kernels, and the conv10 layer of the convolutional layer passes through 1000 groups of 1x1 convolutional kernels; the sizes of the pooling layers maxpool1, maxpool2, and maxpool3 are all 3x3; the structure of the Fire layer is unified. Among them, the Squeeze part consists of consecutive 1x1 convolutions, and the Expand part consists of consecutive 1x1 convolutions and 3x3 convolutions connected. If the number of channels in the Squeeze part is denoted as S1x1, and the number of 1x1 convolutions and 3x3 convolutions in the Expand part are denoted as e1x1 and e3x3 respectively, then the relationship among the above three satisfies the following formula: S1×1 < e1×1 + e3×3. The neural network acceleration module also adds network shortcuts between the fire3, fire5, fire7, and fire9 layers to improve robustness during training and prevent overfitting. The improved SqueezeNet backbone network outputs a series of convolutional feature maps to the feature pyramid network for the next step of calculation; extracts four scales of fire2, fire6, fire5, and conv10 in the backbone network SqueezeNet to construct a feature pyramid network. After being processed by the feature pyramid network, the predicted results are output respectively at each scale; the feature pyramid network further includes: The predicted results include two parts: heatmap and vector field paf. By synthesizing the predicted results of heatmap and paf at multiple scales, the model finally obtains the output result of skeletal key points;If there are n scale prediction results in the end, the final prediction results of heatmap and paf are expressed by the following formulas: Among them, vh_i and vp_i are the heatmap and paf values ​​output at each scale respectively.

2. A low-cost real-time skeleton key point recognition device as claimed in claim 1, characterized in that: The neural network acceleration module further includes: a quantization and acceleration engine, wherein the quantization is to convert the data in FP32 format into INT8 format during the inference process for the data in FP32 format used in the model training process, and the acceleration engine is to accelerate the model inference process based on the OpenCL framework to further improve the real-time performance of the skeleton key point recognition process.

3. A low-cost real-time skeleton key point recognition method, using the device according to any one of claims 1-2, characterized in that: The image data is collected and the acquired image information is processed. By using the improved SqueezeNet as the basic backbone network and combining it with the feature pyramid network, multi-scale feature extraction is performed on the image. After processing by the feature pyramid network, the prediction results are output at each scale respectively. By quantizing and accelerating the data in the reasoning process, the skeleton key point information of the collected image is finally output in real time.

Images