A human body part detection method, device and system based on images

Abstract

The application discloses a human body part detection method, device and system based on images. When an image is obtained by a mobile terminal and human body part tracking and identification on the image is needed, a first convolution module group is used to down-sample the image, so as to reduce the image size and improve the image processing speed, thereby ensuring the real-time performance of image tracking. A second convolution module group is used to extract features from the image without down-sampling, thereby ensuring the accuracy of the image after down-sampling by the first convolution module group, so as to avoid the problem of low human body part tracking and identification accuracy caused by low image accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image detection, in particular to a human body part detection method, device and system based on images. BACKGROUND

[0002] For the face tracking task on the mobile terminal, the common method is to use a face detector and a set of tracking logic to achieve together.

[0003] However, the tracking algorithm on the mobile terminal usually has limited computing power, which reduces the real-time performance and accuracy of face tracking. SUMMARY

[0004] Therefore, the present application provides a human body part detection method, device and system based on images, and the specific solutions are as follows:

[0005] A human body part detection method based on images, comprising:

[0006] obtaining an image, wherein the image includes at least one human body part;

[0007] performing feature extraction on the image by a convolution module group to obtain a feature map of the image, wherein the convolution module group includes at least a first convolution module group and a second convolution module group, the first convolution module group is used for performing feature extraction on the image by downsampling, and the second convolution module group is used for performing feature extraction on the image by non-downsampling after the downsampling;

[0008] determining the region and category of the at least one human body part in the image based on the feature map.

[0009] Further, the determination of the region and category of the at least one human body part in the image based on the feature map comprises:

[0010] performing multi-scale target detection based on the feature map to obtain the region and category of the at least one human body part.

[0011] Further, the feature extraction on the image by the convolution module group comprises:

[0012] performing grouped convolution on the image by a third convolution module group, and the third convolution module group is arranged in the first convolution module group.

[0013] Further, the first convolution module group is used for performing feature extraction on the image by downsampling, comprising:

[0014] performing continuous downsampling on the image by not less than three first convolution modules;

[0015] extracting features of the down-sampled image without down-sampling by at least one second convolution module;

[0016] The first convolution module group at least includes the first convolution module and the second convolution module, and the at least one second convolution module is arranged between the at least three first convolution modules.

[0017] Further, the system further comprises:

[0018] The features of the output channels of different layers of the corresponding arranged convolution module are interactively transmitted by an interactive convolution module;

[0019] The convolution module group at least includes the convolution module and the interactive convolution module, and the interactive convolution module is arranged correspondingly to the convolution module.

[0020] Further, the determining the region and the category of the at least one human body part in the image based on the feature map comprises:

[0021] Detecting human body parts of different sizes in the image based on not less than one fourth convolution module;

[0022] The features of the output channels of different layers of the corresponding arranged fourth convolution module are interactively transmitted by not less than one interactive convolution module;

[0023] The region of the at least one human body part in the image is determined based on the features output by the not less than one interactive convolution module;

[0024] The convolution module group includes a fourth convolution module group for group convolution of the image, and the fourth convolution module group includes the fourth convolution module and the interactive convolution module arranged correspondingly to the fourth convolution module.

[0025] Further, the determining the region and the category of the at least one human body part in the image based on the feature map comprises:

[0026] The at least one human body part in the feature map is classified by a classification convolution module, and different types of human body parts are output by different output channels in the classification convolution module.

[0027] A human body part detection system based on an image, comprising:

[0028] A first obtaining unit is configured to obtain an image, and the image includes at least one human body part;

[0029] a second obtaining unit configured to perform feature extraction on the image by a convolution module group to obtain a feature map of the image, wherein the convolution module group comprises at least a first convolution module group and a second convolution module group, the first convolution module group is configured to perform feature extraction on the image by downsampling, and the second convolution module group is configured to perform feature extraction on the image by non-downsampling after the downsampling;

[0030] a determining unit configured to determine a region and a category of the at least one human body part in the image based on the feature map.

[0031] An image-based human body part detection device, comprising:

[0032] a processor configured to obtain an image comprising at least one human body part, perform feature extraction on the image by a convolution module group to obtain a feature map of the image, wherein the convolution module group comprises at least a first convolution module group and a second convolution module group, the first convolution module group is configured to perform feature extraction on the image by downsampling, and the second convolution module group is configured to perform feature extraction on the image by non-downsampling after the downsampling, and determine a region and a category of the at least one human body part in the image based on the feature map.

[0033] a memory configured to store a program for the processor to execute the above-mentioned processing procedure.

[0034] A readable storage medium configured to store at least a set of instruction sets.

[0035] The instruction sets are configured to be invoked and at least perform the image-based human body part detection method of any one of the above.

[0036] As can be seen from the above technical solutions, the image-based human body part detection method, device and system disclosed by the present application obtain an image comprising at least one human body part, perform feature extraction on the image by a convolution module group to obtain a feature map of the image, wherein the convolution module group comprises at least a first convolution module group and a second convolution module group, the first convolution module group is configured to perform feature extraction on the image by downsampling, and the second convolution module group is configured to perform feature extraction on the image by non-downsampling after the downsampling, and determine a region and a category of the at least one human body part in the image based on the feature map. When the image is obtained by a mobile terminal and the human body part on the image needs to be tracked and recognized, the first convolution module group is used to perform downsampling on the image to reduce the image size and improve the image processing speed, thereby ensuring the real-time performance of image tracking, and the second convolution module group is used to perform feature extraction on the image by non-downsampling, thereby ensuring the accuracy of the image after the downsampling by the first convolution module group, so as to avoid the problem of low human body part tracking and recognition accuracy due to low image accuracy. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of an image-based human body part detection method disclosed in an embodiment of this application;

[0039] Figure 2 This is a flowchart of an image-based human body part detection method disclosed in an embodiment of this application;

[0040] Figure 3 This is a structural diagram of a detection model for a human body part disclosed in an embodiment of this application;

[0041] Figure 4 This is a schematic diagram of the structure of an image-based human body part detection system disclosed in an embodiment of this application;

[0042] Figure 5 This is a schematic diagram of the structure of an image-based human body part detection device disclosed in an embodiment of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] This application discloses an image-based method for detecting human body parts, the flowchart of which is shown below. Figure 1 As shown, it includes:

[0045] Step S11: Obtain an image, the image including at least one human body part;

[0046] Step S12: Extract features from the image using a convolutional module group to obtain the image feature map. The convolutional module group includes at least a first convolutional module group and a second convolutional module group. The first convolutional module group is used for feature extraction by downsampling the image, and the second convolutional module group is used for feature extraction without downsampling the downsampled image.

[0047] Step S13, determining the region and category of at least one human body part in the image based on the feature map.

[0048] The face tracking task based on the mobile terminal is usually realized by a face detector and a set of face tracking logic, wherein the face detector directly determines whether the face tracking algorithm is reliable. The tracking algorithm on the mobile terminal has two characteristics, one of which is that the mobile terminal, especially the low-end processor, has limited computing power and cannot use a model with a large enough parameter quantity to improve the accuracy of the face detector, and the other is that the tracking algorithm itself has high requirements for the real-time performance and accuracy of face detection. In the current general scheme, due to the high computing power requirement of the face detector, it is difficult to achieve real-time performance on a low-cost mobile terminal processor, and the accuracy of the face detector is low.

[0049] Therefore, the present scheme adopts a convolutional neural network model to detect and track human body parts in an image.

[0050] Specifically, after obtaining the image, the image is feature-extracted by a convolution module group, and the convolution module group at least includes a first convolution module group and a second convolution module group.

[0051] The first convolution module group is used to down-sample the image, that is, to reduce the size of the image. While the size of the image is reduced, the accuracy of the image is also reduced, which increases the processing speed of the image and ensures the real-time performance of tracking human body parts in the image.

[0052] For example, the step length of the convolution module in the first convolution module group is 2. At this time, the image size is reduced by a certain ratio. For example, the input image size is 512*512, and after passing through a convolution module with a step length of 2, the output image size of the convolution module is 256*256. That is, the size of the image is reduced after passing through the convolution module, and the calculation speed is increased.

[0053] If the step length of the convolution module is 1, the size of the image will not change. For example, the input image size is 512*512, and after passing through a convolution module with a step length of 1, the output image size of the convolution module is 512*512.

[0054] Therefore, the step length can be used as the multiple of the down-sampling. When the first convolution module group includes multiple convolution modules, continuous down-sampling can be performed to quickly reduce the size of the image and quickly reduce the accuracy of the image. For example, the first convolution module group includes three consecutive convolution modules with a step length of 2.

[0055] In addition, the first convolution module group can also include a third convolution module group, which performs grouped convolution on the image to improve the speed of feature extraction of the image.

[0056] The convolution module is selected from multiple convolution modules for image downsampling, and a grouped convolution form is used for feature extraction for image downsampling, so as to improve the image processing speed.

[0057] For example, in the four convolution modules for continuous downsampling, the second convolution module and the third convolution module are selected to perform the processing task in the form of grouped convolution, and the number of groups is the same as the number of output channels, for example, the output channel of conv2 is 4, and the number of groups is 4; the output channel of conv3 is 16, and the number of groups is 16, so that each group can correspond to each output channel.

[0058] The second convolution module group is used for feature extraction without downsampling of the image after downsampling, that is, the image only needs to be downsampled when passing through the first convolution module group, and the convolution module for downsampling is not included in the second convolution module group, that is, the step length of each convolution module in the second convolution module group is 1.

[0059] When the image passes through the second convolution module group, since it has been downsampled by the first convolution module group, the size and accuracy of the image have been reduced to a certain extent, which can meet the real-time requirement of image processing, so that in the second convolution module group, there is no need to set a convolution module with a downsampling function, and only feature extraction is needed, so that the feature map obtained after feature extraction is an image with a certain accuracy requirement, so that the process of image processing can meet the requirements of accuracy and speed at the same time, and the balance between accuracy and speed is guaranteed.

[0060] Further, determining the region and category of at least one body part in the image based on the feature map can be: performing multi-scale target detection based on the feature map to obtain the region and category of at least one body part, or directly performing target detection based on the feature map to obtain the region and category of the body part.

[0061] That is, when performing feature extraction, the number of channels of different scales can be set at different levels in the form of a feature pyramid, so as to obtain body part regions with different downsampling multiples, for example, the number of channels of the first convolution module in the feature pyramid layer is 64, the number of channels of the second convolution module is 128, and the number of channels of the third convolution module is 96, that is, different convolution modules use different numbers of channels to realize the acquisition of feature maps with different downsampling multiples.

[0062] Alternatively, the feature pyramid can not be used, but the classification of the region of the body part can be directly performed based on the feature map obtained by the above skeleton network layer.

[0063] That is, in the feature extraction process, a skeleton network and a feature pyramid layer can be used to achieve feature extraction of the image, and output a feature image. Both the skeleton network and the feature pyramid layer use a 3*3 convolution kernel.

[0064] Each convolution module includes a convolution layer, a normalization layer, and an activation layer. For example, conv1, where conv represents a standard module in a convolutional neural network (CNN) that includes a convolution layer, followed by a normalization layer, and then an activation layer. The subscript of conv1 is (3*3*4, s2), where the convolution kernel size is 3*3, the output channel is 4, and the step size is 2.

[0065] The output of each convolution module is a feature map, and the size and number of channels of the feature map affect the operation speed of the model. The larger the feature map and the more the number of channels, the slower the model calculation speed.

[0066] After feature extraction, the output is a feature map of multiple human body part regions. Further classification and regression layers are used to classify the extracted multiple human body part regions to classify different parts in the multiple human body part regions, such as: human head detection region, human face detection region, and background region, etc., so that only the human head detection region and the human face detection region are output, and the background region does not need to be output.

[0067] After outputting the human head detection region and the human face detection region, a loss function layer is used to determine the loss data in the model training process, where the regression loss function is giouloss, and the classification loss function is focal loss, so as to optimize the model based on the loss function, so that the human body part region and the category detected based on the model are more accurate.

[0068] The embodiment discloses a human body part detection method based on an image, obtains an image, the image includes at least one human body part, carries out feature extraction to the image through a convolution module group to obtain the feature map of the image, the convolution module group includes at least a first convolution module group and a second convolution module group, the first convolution module group is used to carry out the feature extraction of the image of downsampling, the second convolution module group is used to carry out the feature extraction of the image of no downsampling after downsampling, and the region and category of at least one human body part in the image are determined based on the feature map.The scheme is obtained through the image of mobile terminal, and when the human body part on the image needs to be tracked and identified, the image is downsampling through the first convolution module group, so that the size of the image is reduced, and the purpose of improving the image processing speed is achieved, so that the real-time performance of image tracking is guaranteed;And the second convolution module group is used for the feature extraction of the image of no downsampling, so that the accuracy of the image after the downsampling of the first convolution module group is guaranteed, so as to avoid the problem of low human body part tracking and identification accuracy caused by low image accuracy.

[0069] The embodiment discloses a human body part detection method based on an image, and a flow chart thereof is as shown in Figure 2 The embodiment discloses a human body part detection method based on an image, and a flow chart thereof is as shown in

[0070] Step S21, obtaining an image, the image includes at least one human body part;

[0071] Step S22, carrying out feature extraction to the image through a convolution module group to obtain the feature map of the image, wherein the convolution module group includes at least a first convolution module group and a second convolution module group, the first convolution module group is used to carry out the feature extraction of the image of downsampling, and the second convolution module group is used to carry out the feature extraction of the image of no downsampling after downsampling;

[0072] Step S23, the features of the output channels of different layers in the convolution module corresponding to the interactive convolution module are interactively transmitted through the interactive convolution module, wherein the convolution module group includes at least a convolution module and an interactive convolution module, and the interactive convolution module is correspondingly arranged with the convolution module;

[0073] Step S24, determining the region and category of at least one human body part in the image based on the feature map.

[0074] A plurality of convolution modules are arranged in the convolution module group, in addition, the convolution module group also includes a plurality of interactive convolution modules, the interactive convolution modules are correspondingly arranged with the convolution modules, one interactive convolution module can be arranged after each convolution module, or only some interactive convolution modules can be arranged after the convolution modules, so as to interactively transmit the features between the output channels of different layers in the convolution module, ensure that the information of each layer of the convolution module can be interacted between various channels, and mutual information is realized.

[0075] The convolution kernel of the interaction convolution module can be 1*1 to avoid the problem of serious information loss on the original image caused by continuous downsampling in the convolution module and serious compression of the number of output channels.

[0076] As shown in the structural diagram of the human body part detection model, the model includes a skeleton network layer, a feature pyramid layer, a classification regression layer, and a loss function layer. Figure 3 As shown in the structural diagram of the human body part detection model, the model includes a skeleton network layer, a feature pyramid layer, a classification regression layer, and a loss function layer.

[0077] The skeleton network layer includes nine convolution modules, namely, conv1-conv9. Conv1-conv5 are convolution modules in the first convolution module group, which can perform continuous downsampling. The convolution kernel of each convolution module is 3*3, except that the output channel and the step length are different.

[0078] For example, the output channel of conv1 is 4 and the step length is 2; the output channel of conv2 is 3 and the step length is 2; the output channel of conv3 is 16 and the step length is 2; the output channel of conv4 is 32 and the step length is 1; and the output channel of conv5 is 32 and the step length is 2. Four convolution modules with a step length of 2 are used, and a small number of output channels are used to quickly reduce the size of the feature map to speed up the calculation of the subsequent layer.

[0079] The image is continuously downsampled by no less than three first convolution modules, and the downsampled image is extracted by at least one second convolution module without downsampling. The first convolution module group includes at least a first convolution module and a second convolution module, and at least one second convolution module is arranged between the at least three first convolution modules.

[0080] The second convolution module without downsampling in the first convolution module group is located in the middle position of the several first convolution modules with downsampling to avoid multiple convolution modules performing multiple continuous downsampling on the image.

[0081] As shown in the structural diagram of the human body part detection model, the model includes a skeleton network layer, a feature pyramid layer, a classification regression layer, and a loss function layer. Figure 3 In the first convolution module group, the step length of all the convolution modules is 2 except that the step length of conv4 is 1. This is to ensure that one convolution module in the convolution module group does not perform multiple continuous downsampling, thereby ensuring the processing speed and avoiding the problem of rapid reduction of image accuracy caused by multiple continuous downsampling.

[0082] For example, if the input image size is 512*512*3, after the down-sampling of conv1-conv5, the output feature map is 32*32*32. After the image is down-sampled by conv1-conv5, the purpose of reducing the image size and reducing the image precision is achieved, which greatly reduces the calculation amount of the subsequent convolution module, so as to improve the processing speed of the image.

[0083] Figure 3 The conv6-conv9 are convolution modules in the second convolution module group, which can perform feature extraction without down-sampling. In the conv6-conv9, the step length of each convolution module is 1. After the image is processed by any one of the convolution modules in the conv6-conv9, the precision of the image will not be reduced, but only the feature learning will be performed.

[0084] An interaction convolution module is arranged after at least part of the convolution modules in the skeleton network layer. The interaction convolution module is only used for information interaction between different output channel layers of the corresponding convolution module. The convolution kernel of the interaction convolution module can be set to 1*1. Further, the interaction convolution module can not be arranged after the conv1.

[0085] In addition, the interaction convolution module can also be arranged in the feature pyramid layer. Specifically, a fourth convolution module group is arranged in the feature pyramid layer, and the fourth convolution module group includes a fourth convolution module and an interaction convolution module corresponding to the fourth convolution module. At least one fourth convolution module is used to detect different sizes of human body parts in the image, and the features of the output channels of the corresponding fourth convolution module are interactively transmitted through at least one interaction convolution module. At least one interaction convolution module is used to determine the region of at least one human body part in the image based on the output features of the fourth convolution module.

[0086] The feature pyramid layer includes four convolution modules, which are determined as the fourth convolution modules, i.e., conv10-conv13. The output channel of the conv10 is 64, and the step length is 2. The output channel of the conv11 is 64, and the step length is 2. The output channel of the conv12 is 128, and the step length is 1. The output channel of the conv13 is 96, and the step length is 2.

[0087] In order to ensure the calculation speed of the model, the FPN and other complex operations are not used in the feature pyramid layer, but the interaction convolution module is directly used.

[0088] Specifically, an interaction convolution module is arranged after each convolution module in the feature pyramid layer, for example, Figure 3As shown in the figure, after the interactive convolution module arranged behind conv12, a 1*1 interactive convolution module is arranged in front of conv13, which is used to compress the number of channels to achieve the purpose of acceleration.

[0089] In actual calculation, the feature pyramid composed of the three 1*1 interactive convolution modules arranged after conv10, conv12 and conv13 is directly used, and the output downsampling multiples of the original image are 32, 64 and 128 respectively. Since the interactive convolution module arranged after conv11 is used for learning and training, the feature extraction can be omitted.

[0090] Specifically, the size of the interactive convolution module is determined according to the distribution of the size of the human body part in the input image, for example, conv10, conv12 and conv13 are used to detect the human body part in a small block area with a size of about 32*32, a human body part in an area with a size of about 64*64, and a human body part in a large block area with a size of about 128*128. Among them, the sizes of the human body parts detected by conv10, conv12 and conv13 are different, that is, each convolution module is only used to detect the detection frame of a human body part with a fixed size, that is, the detection area.

[0091] Therefore, the classification layer and the regression layer in the classification regression layer need to obtain the feature maps in the three 1*1 interactive convolution modules after conv10, conv12 and conv13 to perform classification calculation on the detection areas in the feature maps.

[0092] Specifically, the classification layer classifies at least one human body part in the feature image through a classification convolution module including three output channels, so as to ensure that different types of human body parts are output through different output channels in the classification convolution module.

[0093] Among them, the classification layer is composed of only one convolution layer, the size of the convolution kernel is 3*3, the step is 1, and the number of output channels is the same as the number of categories, wherein the number of categories is 3, i.e. background, head and face, so the number of output channels is 3.

[0094] The regression layer is also composed of one convolution layer, the size of the convolution kernel is 3*3, the step is 1, and the number of output channels is the product of the number of categories and 4, i.e. 3*4=12, so the number of output channels is 12, which are the x, y, w, h of the background frame, the x, y, w, h of the face frame and the x, y, w, h of the head frame.

[0095] Among them, x is the x-axis coordinate of the top left corner of the detection frame, y is the y-axis coordinate of the top left corner of the detection frame, w is the width of the detection frame, and h is the height of the detection frame.

[0096] Further, the output channel can further include a confidence of the detection frame and a class of the detection frame.

[0097] The data output by the classification regression layer can be an n*6 matrix, n being the total number of detected human heads and faces, and the 6 dimensions of each row of the matrix being respectively an x-axis coordinate of a top-left corner of the detection frame, a y-axis coordinate of the top-left corner of the detection frame, a width w of the detection frame, a height h of the detection frame, a confidence score of the detection frame, and a class of the detection frame.

[0098] Through the skeleton network layer, the feature pyramid layer, the classification regression layer and the loss function layer in the scheme, detection and classification of human body parts in an image are realized, certain accuracy is ensured on the basis of real-time, and the scheme can run on a mobile terminal processor such as 3399, and can reach a detection speed of 12 fps in the case of an input resolution of 640*480; in addition, the scheme can detect a human head while detecting a human face in an image, which avoids the problem of tracking interruption caused by estimation of the position of the human face through the detection frame of the human head when the human face is invisible due to posture rotation or occlusion, and improves the stability in the process of human face tracking.

[0099] The embodiment discloses a human body part detection method based on an image, obtains an image, the image including at least one human body part, extracts features of the image through a convolution module group to obtain a feature map of the image, the convolution module group including at least a first convolution module group and a second convolution module group, the first convolution module group being used for feature extraction of the image with downsampling, the second convolution module group being used for feature extraction of the image without downsampling after the downsampling, and determining a region and a class of the at least one human body part in the image based on the feature map. When the image is obtained on a mobile terminal and human body part tracking and recognition on the image is needed, the first convolution module group is used for downsampling of the image to reduce the size of the image and improve the image processing speed, so that the real-time of image tracking is ensured; and the second convolution module group is used for feature extraction of the image without downsampling, so that the accuracy of the image after the downsampling by the first convolution module group is ensured, to avoid the problem of too low human body part tracking and recognition accuracy caused by too low image accuracy.

[0100] The embodiment discloses a human body part detection system based on an image, a structure diagram of which is shown in Figure 4 The embodiment discloses a human body part detection system based on an image, a structure diagram of which is shown in

[0101] The first obtaining unit 41, the second obtaining unit 42 and the determining unit 43.

[0102] The first obtaining unit 41 is used for obtaining an image, the image including at least one human body part;

[0103] The second obtaining unit 42 is configured to perform feature extraction on the image by a convolution module group to obtain a feature map of the image, wherein the convolution module group at least includes a first convolution module group and a second convolution module group, the first convolution module group is configured to perform feature extraction on the image by downsampling, and the second convolution module group is configured to perform feature extraction on the image by non-downsampling after the downsampling.

[0104] The determining unit 43 is configured to determine a region and a category of at least one human body part in the image based on the feature map.

[0105] The face tracking task based on a mobile terminal is usually realized by a face detector and a set of face tracking logic, wherein the face detector directly determines whether the face tracking algorithm is reliable. The tracking algorithm on the mobile terminal has two characteristics, one of which is that the mobile terminal, especially the low-end processor, has limited computing power, and cannot use a model with a large enough parameter quantity to improve the accuracy of the face detector, and the other is that the tracking algorithm itself has high requirements for the real-time performance and accuracy of the face detection. In the current general scheme, due to the high computing power requirement of the face detector, it is difficult to achieve real-time performance on a low-cost mobile terminal processor, and in addition, the accuracy of the face detector is low.

[0106] Therefore, the present scheme adopts a convolutional neural network model to detect and track the human body part in the image.

[0107] Specifically, after obtaining the image, the image is subjected to feature extraction by a convolution module group, and the convolution module group at least includes a first convolution module group and a second convolution module group.

[0108] The first convolution module group is configured to perform downsampling on the image, that is, to reduce the size of the image, and at the same time, the image accuracy is reduced, which makes the processing speed of the image increased, thereby ensuring the real-time performance of tracking the human body part in the image.

[0109] For example, the step length of the convolution module in the first convolution module group is 2, at this time, the image size is reduced by a multiple, such as: the input image size is 512*512, and after passing through a convolution module with a step length of 2, the output image size of the convolution module is 256*256, that is, the size of the image is reduced after passing through the convolution module, and the calculation speed is increased.

[0110] If the step length of the convolution module is 1, the image size will not change, such as: the input image size is 512*512, and after passing through a convolution module with a step length of 1, the output image size of the convolution module is 512*512.

[0111] Therefore, the step size can be used as a downsampling multiple. When multiple convolution modules are included in the first convolution module group, continuous downsampling can be performed to quickly reduce the size of the image and quickly reduce the accuracy of the image. For example, the first convolution module group includes 3 consecutive convolution modules with a step size of 2.

[0112] In addition, the first convolution module group can further include a third convolution module group for performing grouped convolution on the image to improve the speed of feature extraction on the image.

[0113] In the multiple convolution modules for downsampling the image, the convolution modules are selected to perform feature extraction by grouped convolution to improve the image processing speed.

[0114] For example, in the 4 consecutive convolution modules for downsampling, the second and third convolution modules are selected to perform processing tasks in the form of grouped convolution, where the number of groups is the same as the number of output channels. For example, the output channel of conv2 is 4, and the number of groups is 4; the output channel of conv3 is 16, and the number of groups is 16, so that each group can correspond to each output channel.

[0115] The second convolution module group is used for non-downsampling feature extraction on the image after downsampling. That is, the image only needs to be downsampling when passing through the first convolution module group, and the second convolution module group does not include a convolution module for downsampling the image. That is, the step size of each convolution module in the second convolution module group is 1.

[0116] When the image passes through the second convolution module group, it has been downsampling by the first convolution module group, and its size and accuracy have been reduced to a certain extent, which can meet the real-time requirement of image processing. Therefore, in the second convolution module group, there is no need to set a convolution module with a downsampling function, only feature extraction is needed, so as to ensure that the feature map after feature extraction is an image with a certain accuracy requirement, so that the image processing process can meet the requirements of accuracy and speed at the same time, and the balance between accuracy and speed is ensured.

[0117] Further, determining the region and category of at least one body part in the image based on the feature map can be: performing multi-scale target detection based on the feature map to obtain the region and category of at least one body part, or directly performing target detection based on the feature map to obtain the region and category of the body part.

[0118] That is, when feature extraction is performed, different numbers of channels at different scales can be set at different levels in the form of a feature pyramid, so as to obtain human body part regions with different downsampling multiples, such as: the number of channels used in the first convolution module of the feature pyramid layer is 64, the number of channels used in the second convolution module is 128, and the number of channels used in the third convolution module is 96, that is, different convolution modules use different numbers of channels to realize the acquisition of feature maps with different downsampling multiples.

[0119] Alternatively, the feature pyramid can not be used, but the features obtained based on the above skeleton network layer can be directly used for classification of the human body part region.

[0120] That is, during feature extraction, a skeleton network and a feature pyramid layer can be used to realize feature extraction, and both the skeleton network and the feature pyramid layer are used to extract features from the image and output a feature image. Both the skeleton network and the feature pyramid layer use a 3*3 convolution kernel.

[0121] Each convolution module includes a convolution layer, a normalization layer, and an activation layer. Taking conv1 as an example, where conv is used to represent a standard module in a convolutional neural network CNN, which includes a convolution layer, followed by a normalization layer, and then an activation layer. The subscript of conv1 is (3*3*4, s2), where the convolution kernel size is 3*3, the output channel is 4, and the step is 2.

[0122] The output of each convolution module is a feature map, and the size and number of channels of the feature map will affect the operation speed of the model. The larger the feature map and the more the number of channels, the slower the model calculation speed.

[0123] After feature extraction, the output is a feature image of multiple human body part regions, which is further classified by a classification regression layer to classify the multiple human body part regions, such as: human head detection region, human face detection region, and background region, so that only the human head detection region and the human face detection region are output, and the background region does not need to be output.

[0124] After outputting the human head detection region and the human face detection region, the loss data in the model training process is determined by a loss function layer, where the regression loss function is giouloss, and the classification loss function is focal loss, so as to optimize the model based on the loss function, so that the human body part region and the category detected based on the model are more accurate.

[0125] Further, the human body part detection system based on images disclosed in the embodiment can further include a transmission module.

[0126] The transmission module is configured to interactively transmit the features of the output channels of different layers in the convolution module corresponding to the interaction convolution module through the interaction convolution module.

[0127] The convolution module group comprises a plurality of convolution modules and a plurality of interaction convolution modules, and the interaction convolution modules are arranged correspondingly to the convolution modules.

[0128] The convolution kernel of the interaction convolution module can be 1x1, so as to avoid the problem of serious loss of information on the original image caused by continuous downsampling in the convolution module and serious compression of the number of output channels.

[0129] The human body part detection model shown in the structural diagram is taken as an example for description, which comprises a skeleton network layer, a feature pyramid layer, a classification regression layer and a loss function layer. Figure 3

[0130] The skeleton network layer comprises nine convolution modules, i.e. conv1-conv9, wherein conv1-conv5 are the convolution modules in the first convolution module group, which can perform continuous downsampling, and the convolution kernel of each convolution module is 3x3, except for the output channel and the step.

[0131] For example, the output channel of conv1 is 4 and the step is 2; the output channel of conv2 is 3 and the step is 2; the output channel of conv3 is 16 and the step is 2; the output channel of conv4 is 32 and the step is 1; and the output channel of conv5 is 32 and the step is 2. Four convolution modules with a step of 2 are used, and a small number of output channels are used to quickly reduce the size of the feature map for calculation acceleration of the subsequent layer.

[0132] The image is continuously downsampled by the first convolution module group, and the downsampled image is extracted by the second convolution module without downsampling, wherein the first convolution module group comprises at least one first convolution module and at least one second convolution module, and the at least one second convolution module is arranged between the at least three first convolution modules.

[0133] The second convolution module without downsampling in the first convolution module group is arranged at the middle position of the first convolution modules for downsampling, so as to avoid multiple convolution modules for multiple continuous downsampling of the image. ​

[0134] As Figure 3 In the first convolution module group, the step length of conv4 is 1, and the step length of the other convolution modules is 2, so as to ensure that one convolution module in the continuous downsampling process can make the convolution module group not be continuously downsampled multiple times, thereby ensuring the processing speed and avoiding the problem of rapid reduction of image accuracy caused by multiple continuous downsampling.

[0135] For example, if the input image size is 512*512*3, after the downsampling of conv1-conv5, the output feature map is 32*32*32. After the image is downsampled by conv1-conv5, the purpose of reducing the image size and reducing the image accuracy is achieved, which greatly reduces the calculation amount of the subsequent convolution module, so as to improve the processing speed of the image.

[0136] Figure 3 In the second convolution module group, conv6-conv9 are convolution modules capable of feature extraction without downsampling. In the second convolution module group, the step length of each convolution module is 1, and after the image is processed by any one of the convolution modules in the second convolution module group, the image accuracy is not reduced, but only the feature learning is performed.

[0137] An interaction convolution module is arranged after at least part of the convolution modules in the skeleton network layer. The interaction convolution module is only used for information interaction between different output channel layers of the corresponding convolution module. The convolution kernel of the interaction convolution module can be set to 1*1. Further, the interaction convolution module can not be arranged after conv1.

[0138] In addition, the interaction convolution module can also be arranged in the feature pyramid layer. Specifically, a fourth convolution module group is arranged in the feature pyramid layer, and the fourth convolution module group includes a fourth convolution module and an interaction convolution module corresponding to the fourth convolution module. At least one fourth convolution module detects different sizes of human body parts in the image, and at least one interaction convolution module transmits the features of different layers of the corresponding fourth convolution module. At least one interaction convolution module outputs the features of the different layers of the corresponding fourth convolution module, and determines the region of at least one human body part in the image based on the features.

[0139] The feature pyramid layer includes four convolution modules, and the four convolution modules are determined as the fourth convolution modules, i.e., conv10-conv13. The output channel of conv10 is 64, and the step length is 2. The output channel of conv11 is 64, and the step length is 2. The output channel of conv12 is 128, and the step length is 1. The output channel of conv13 is 96, and the step length is 2.

[0140] And, in order to ensure the calculation speed of the model, the FPN and other complex operations are not used in the feature pyramid layer, and an interactive convolution module is directly used.

[0141] Specifically, an interactive convolution module is arranged after each convolution module in the feature pyramid layer, as shown in the figure. Figure 3 After the interactive convolution module arranged behind the conv12, a 1*1 interactive convolution module is arranged in front of the conv13, which is used to compress the number of channels to achieve the purpose of acceleration.

[0142] In the actual calculation process, the feature pyramid composed of the three 1*1 interactive convolution modules arranged after the conv10, the conv12 and the conv13 is directly used, and the output of the feature pyramid is down-sampled by 32, 64 and 128 times relative to the original image. Since the interactive convolution module arranged after the conv11 is used for learning and training, the feature extraction of the interactive convolution module can be omitted.

[0143] Specifically, the size of the interactive convolution module is determined according to the distribution of the size of the human body parts in the input image, for example, the conv10, the conv12 and the conv13 are used to detect the human body parts in the small block area with a size of about 32*32, the human body parts in the area with a size of about 64*64, and the human body parts in the large block area with a size of about 128*128. The sizes of the human body parts detected by the conv10, the conv12 and the conv13 are different, that is, each convolution module is used to detect the detection frame of a fixed size of human body part, that is, the detection area.

[0144] Therefore, the classification layer and the regression layer in the classification and regression layer need to obtain the feature maps in the three 1*1 interactive convolution modules after the conv10, the conv12 and the conv13, and perform classification calculation on the detection areas in the feature maps.

[0145] Specifically, the classification layer classifies at least one human body part in the feature image through a classification convolution module including three output channels, so as to ensure that different types of human body parts are output through different output channels in the classification convolution module.

[0146] The classification layer is composed of only one convolution layer, the size of the convolution kernel is 3*3, the step is 1, and the number of output channels is the same as the number of categories, wherein the number of categories is 3, that is, background, head and face, and therefore the number of output channels is 3.

[0147] The regression layer is also composed of a convolution layer, the size of the convolution kernel is 3*3, the step is 1, and the number of output channels is the product of the number of categories and 4, that is, 3*4=12, the number of output channels is 12, which are respectively the x, y, w, h of the background frame, the x, y, w, h of the face frame and the x, y, w, h of the head frame.

[0148] Wherein, x is the x-axis coordinate of the top left corner of the detection frame, y is the y-axis coordinate of the top left corner of the detection frame, w is the width of the detection frame, and h is the height of the detection frame.

[0149] Further, the output channel can also include the confidence of the detection frame and the category of the detection frame.

[0150] The data output by the classification regression layer can be an n*6 matrix, n is the total number of detected head and face, and the 6 dimensions of each row of the matrix are respectively: the x-axis coordinate of the top left corner of the detection frame, the y-axis coordinate of the top left corner of the detection frame, the width w of the detection frame, the height h of the detection frame, the confidence score of the detection frame and the category class of the detection frame.

[0151] Through the skeleton network layer, the feature pyramid layer, the classification regression layer and the loss function layer in the scheme, the detection and classification of human body parts in the image are realized, the accuracy is guaranteed on the basis of real-time, and the scheme can run on mobile processors such as 3399, and the detection speed can reach 12fps under the condition that the input resolution is 640*480; in addition, the scheme can detect the head while detecting the face in the image, which avoids the problem that when the face is invisible due to posture rotation or occlusion, the position of the face is estimated through the detection frame of the head to prevent tracking interruption, and improves the stability in the face tracking process.

[0152] The embodiment discloses a human body part detection system based on images, obtains an image, the image includes at least one human body part, feature extraction is carried out on the image through a convolution module group to obtain a feature map of the image, the convolution module group includes at least a first convolution module group and a second convolution module group, the first convolution module group is used for carrying out feature extraction on the image by downsampling, the second convolution module group is used for carrying out feature extraction on the image by non-downsampling after downsampling, and the region and category of at least one human body part in the image are determined based on the feature map. When the image is obtained in the mobile terminal and the human body part on the image needs to be tracked and identified, the first convolution module group is used for carrying out feature extraction on the image by downsampling, so that the size of the image is reduced, the image processing speed is improved, and the real-time performance of image tracking is ensured. And the second convolution module group is used for carrying out feature extraction on the image by non-downsampling, so that the accuracy of the image after downsampling by the first convolution module group is ensured, so as to avoid the problem that the tracking and identification accuracy of the human body part is too low due to the too low accuracy of the image.

[0153] The embodiment discloses a human body part detection device based on images, and a structural schematic diagram thereof is as shown in Figure 5 The embodiment discloses a human body part detection device based on images, and a structural schematic diagram thereof is as shown in

[0154] The processor 51 and the memory 52.

[0155] The processor is used for obtaining an image, the image includes at least one human body part; feature extraction is carried out on the image through a convolution module group to obtain a feature map of the image, wherein the convolution module group includes at least a first convolution module group and a second convolution module group, the first convolution module group is used for carrying out feature extraction on the image by downsampling, the second convolution module group is used for carrying out feature extraction on the image by non-downsampling after downsampling; and the region and category of at least one human body part in the image are determined based on the feature map.

[0156] The memory is used for storing the program of the processor for executing the above processing process.

[0157] The human body part detection device based on images disclosed by the embodiment is realized based on the human body part detection method based on images disclosed by the above embodiment, and details are not repeated here.

[0158] The embodiment discloses a human body part detection device based on an image, obtains an image, the image includes at least one human body part, feature extraction is carried out on the image by a convolution module group, to obtain the feature map of the image, the convolution module group includes at least a first convolution module group and a second convolution module group, the first convolution module group is used to carry out feature extraction on the image by downsampling, the second convolution module group is used to carry out feature extraction on the image by no downsampling after downsampling, and the region and the category of at least one human body part in the image are determined based on the feature map.The scheme is when the image is obtained in the mobile terminal and the human body part on the image needs to be tracked and identified, the image is downsampled by the first convolution module group, so that the size of the image is reduced, the image processing speed is improved, and the real-time performance of image tracking is ensured, and the accuracy of the image after downsampling by the first convolution module group is ensured by the second convolution module group for feature extraction without downsampling, so that the problem of low human body part tracking and identification accuracy caused by too low image accuracy is avoided

[0159] The embodiment of the application further provides a readable storage medium, which has a computer program stored thereon, the computer program is loaded and executed by a processor, and each step of the above-mentioned human body part detection method based on an image is realized, and the specific implementation process can be referred to the description of the corresponding part of the above-mentioned embodiment, and the embodiment will not be repeated.

[0160] The application further provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the method provided in various optional implementation manners of the above-mentioned human body part detection method based on an image, and the specific implementation process can be referred to the description of the corresponding embodiment, and will not be repeated.

[0161] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts of each embodiment can be referred to each other.For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the related parts can be referred to the method part.

[0162] Those skilled in the art will further appreciate that the units and algorithms described in connection with the examples disclosed herein can be embodied directly in hardware, in software, or in a combination of the two. For the sake of brevity, descriptions of a method or an algorithm described in the preceding description will not be repeated in the following description of the examples. For the same reason, not all components and algorithms described in the examples will be repeated. It is to be understood that the above description is intended to be illustrative and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reading the above description. The scope of the application should, therefore, be determined not with reference to the above description, but should instead be determined with reference to the appended claims, along with their full scope of equivalents.

[0163] The steps of a method or algorithm described in connection with the examples disclosed herein can be embodied directly in hardware, in software, or in a combination of the two. A software module can reside in Random Access Memory (RAM), non-volatile memory (e.g., Flash memory, ROM, EEPROM, EPROM, programmable ROM, etc.), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an application-specific integrated circuit (ASIC).

[0164] The above description is intended to be illustrative and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reading the above description. The scope of the application should, therefore, be determined not with reference to the above description, but should instead be determined with reference to the appended claims, along with their full scope of equivalents. The disclosure of all articles and references are incorporated by reference in their entirety.

Claims

1. A method for detecting human body parts based on images, characterized in that, include: Obtain an image, the image including at least one human body part; Feature extraction is performed on the image using a group of convolutional modules to obtain a feature map of the image. The group of convolutional modules includes at least a first group of convolutional modules and a second group of convolutional modules. The first group of convolutional modules is used for feature extraction by downsampling the image, and the second group of convolutional modules is used for feature extraction without downsampling the downsampled image. The first group of convolutional modules is used for feature extraction by downsampling the image, including: continuous downsampling of the image using at least three first convolutional modules; and feature extraction without downsampling the downsampled image using at least one second convolutional module. The first group of convolutional modules includes at least a first convolutional module and a second convolutional module, with the at least one second convolutional module positioned between the at least three first convolutional modules. Based on the feature map, determine the region and category of at least one human body part in the image; Also includes: The features of the output channels of different layers of the corresponding convolutional modules are transmitted interactively through the convolutional module; wherein, the convolutional module group includes at least: a convolutional module and an interactive convolutional module, the interactive convolutional module being set in correspondence with the convolutional module; at least some convolutional modules are followed by an interactive convolutional module, which is only used for information exchange between different output channel layers of its corresponding convolutional module.

2. The method according to claim 1, characterized in that, Determining the region and category of at least one human body part in the image based on the feature map includes: Multi-scale target detection based on the feature map yields the region and category of at least one human body part.

3. The method according to claim 1, characterized in that, The feature extraction of the image through a group of convolutional modules includes: The image is grouped and convolved using a third convolution module group, which is set within the first convolution module group.

4. The method according to claim 1, characterized in that, Determining the region and category of at least one human body part in the image based on the feature map includes: The detection of human body parts of different sizes in the image is performed based on at least one fourth convolutional module; Features of the output channels of different layers of the corresponding fourth convolutional module are transmitted interactively through at least one interactive convolutional module. The region of at least one human body part in the image is determined based on the features output by the not less than one interactive convolution module; The convolution module group includes a third convolution module group for performing grouped convolution on the image, and the fourth convolution module group includes a fourth convolution module and an interactive convolution module corresponding to the fourth convolution module.

5. The method according to claim 1, characterized in that, Determining the region and category of at least one human body part in the image based on the feature map includes: At least one human body part in the feature map is classified by the classification convolution module, and different types of human body parts are output through different output channels in the classification convolution module.

6. An image-based human body part detection system, characterized in that, include: The first obtaining unit is used to obtain an image, the image including at least one human body part; The second obtaining unit is configured to extract features from the image using a group of convolutional modules to obtain a feature map of the image. The group of convolutional modules includes at least a first group of convolutional modules and a second group of convolutional modules. The first group of convolutional modules is used for feature extraction by downsampling the image, and the second group of convolutional modules is used for feature extraction without downsampling the downsampled image. The first group of convolutional modules is used for feature extraction by downsampling the image, comprising: continuously downsampling the image using at least three first convolutional modules; and performing feature extraction without downsampling the downsampled image using at least one second convolutional module. The first group of convolutional modules includes at least a first convolutional module and a second convolutional module, with the at least one second convolutional module disposed between the at least three first convolutional modules. A determining unit is configured to determine the region and category of at least one human body part in the image based on a feature map; The image-based human body part detection system is also used for: The features of the output channels of different layers of the corresponding convolutional modules are transmitted interactively through the convolutional module; wherein, the convolutional module group includes at least: a convolutional module and an interactive convolutional module, the interactive convolutional module being set in correspondence with the convolutional module; at least some convolutional modules are followed by an interactive convolutional module, which is only used for information exchange between different output channel layers of its corresponding convolutional module.

7. An image-based human body part detection device, characterized in that, include: A processor is configured to acquire an image, the image including at least one human body part; extract features from the image using a group of convolutional modules to obtain a feature map of the image, wherein the group of convolutional modules includes at least a first group of convolutional modules and a second group of convolutional modules, the first group of convolutional modules being used for downsampling feature extraction of the image, and the second group of convolutional modules being used for non-downsampling feature extraction of the downsampled image; determine the region and category of the at least one human body part in the image based on the feature map; the first group of convolutional modules is used for downsampling feature extraction of the image, including: continuously downsampling the image using at least three first convolutional modules; and non-downsampling feature extraction of the downsampled image using at least one second convolutional module; wherein the first group of convolutional modules includes at least a first convolutional module and a second convolutional module, and the at least one second convolutional module is disposed between the at least three first convolutional modules; A memory for storing the program used by the processor to execute the above-described processing procedure; The processor is also used for: The features of the output channels of different layers of the corresponding convolutional modules are transmitted interactively through the convolutional module; wherein, the convolutional module group includes at least: a convolutional module and an interactive convolutional module, the interactive convolutional module being set in correspondence with the convolutional module; at least some convolutional modules are followed by an interactive convolutional module, which is only used for information exchange between different output channel layers of its corresponding convolutional module.

8. A readable storage medium for storing at least one set of instructions; The instruction set is used to be invoked and to execute at least the image-based human body part detection method as described in any one of claims 1-5.

Images