Human body posture detection method and device, electronic device, and storage medium

Abstract

The present application provides a method and device for detecting human posture, an electronic device, and a computer-readable storage medium, the method comprising: determining the position information of a person to be detected in a target image; performing key point detection on the person image corresponding to the position information of the person in the target image to obtain a plurality of human key points; determining a plurality of limb angles in the target image based on the plurality of human key points; and determining the human posture of the person to be detected according to the plurality of limb angles corresponding to a plurality of consecutive frames of target images. The present application scheme, after identifying the person image where the person to be detected is located, obtains a plurality of limb angles by means of key point detection and limb angle conversion, and can realize accurate human posture recognition according to the limb angles of the plurality of frames of target images, avoiding interference from complex backgrounds, and improving recognition accuracy compared to directly identifying human posture by human key points.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for detecting human body posture, an electronic device, and a computer-readable storage medium. Background Art

[0002] In workplaces such as substations, bad behaviors of staff (such as smoking and climbing over walls) pose safety hazards. In this case, it is necessary to detect the body posture of the staff and issue an alarm based on the detection results. In related technologies, after collecting scene images, the area where the personnel are located is identified from the scene images, and key points are extracted from the image of the area where the personnel are located to determine the body posture of the personnel. However, due to the complex background of the workplace, the identification of personnel is often interfered by the background and accurate identification cannot be achieved; and the body posture directly determined by key points is not accurate enough. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a method and device for detecting human body posture, an electronic device, and a computer-readable storage medium for accurately identifying human body posture.

[0004] On the one hand, the present application provides a method for detecting a human body posture, comprising:

[0005] Determine the position information of the person to be detected in the target image;

[0006] For the person image corresponding to the person position information in the target image, key point detection is performed to obtain a plurality of human key points;

[0007] Based on the multiple human body key points, determining multiple limb angles in the target image;

[0008] The human body posture of the person to be detected is determined according to multiple limb angles corresponding to multiple consecutive frames of target images.

[0009] In one embodiment, determining the position information of the person to be detected in the target image includes:

[0010] The target image is used as an image to be detected, and the image to be detected is input into a trained target detection model to obtain personnel location information output by the target detection model.

[0011] In one embodiment, the target image is a visible light image;

[0012] The step of determining the position information of the person to be detected in the target image includes:

[0013] Acquire an infrared image corresponding to the visible light image as an image to be detected;

[0014] The image to be detected is input into a trained target detection model to obtain the personnel location information output by the target detection model.

[0015] In one embodiment, the target detection model includes a first feature extraction network and a second feature extraction network;

[0016] The step of inputting the image to be detected into a trained target detection model to obtain the personnel location information output by the target detection model includes:

[0017] Extracting high-resolution image features from the image to be detected based on the first feature extraction network;

[0018] Extracting a first designated image feature, a second designated image feature, and a third designated image feature from the high-resolution image features through the second feature extraction network;

[0019] According to the first designated image feature, the second designated image feature and the third designated image feature, the person position information of the person to be detected is obtained by regression.

[0020] In one embodiment, the second feature extraction network includes a first extraction module, a second extraction module, a third extraction module and a fourth extraction module;

[0021] The extracting the first designated image feature, the second designated image feature and the third designated image feature from the high-resolution image feature by the second feature extraction network comprises:

[0022] Extracting medium-resolution image features from the high-resolution image features based on the first extraction module;

[0023] Extracting low-resolution image features from the medium-resolution image features based on the second extraction module as the first designated image features;

[0024] Extracting a first intermediate image feature from the first designated image feature based on the third extraction module, and fusing the first intermediate image feature with the medium-resolution image feature to obtain a second designated image feature;

[0025] A second intermediate image feature is extracted from the second designated image feature based on the fourth extraction module, and the second intermediate image feature is fused with the high-resolution image feature to obtain a third designated image feature.

[0026] In one embodiment, the plurality of limb angles include angle information corresponding to a plurality of angle categories;

[0027] The method of determining the human posture of the person to be detected according to the multiple limb angles corresponding to the continuous multiple frames of target images includes:

[0028] For each angle category, input the angle information of the continuous multiple frames of target images in the angle category into a bidirectional recurrent neural network corresponding to the angle category to obtain multiple output parameters corresponding to the angle category;

[0029] For multiple output parameters corresponding to each angle category, a specified output parameter corresponding to the angle category is obtained by fusing;

[0030] Based on the specified output parameters corresponding to all angle categories, the human body posture is classified.

[0031] In one embodiment, before classifying and obtaining the human body posture based on the specified output parameters corresponding to all angle categories, the method further includes:

[0032] For each angle category, determining a plurality of differential information corresponding to the angle category according to angle information of a plurality of consecutive frames of target images in the angle category;

[0033] For each angle category, multiple differential information corresponding to each angle category are merged to obtain designated differential information corresponding to the angle category;

[0034] The method of classifying the human body posture based on the specified output parameters corresponding to all angle categories includes:

[0035] Based on the specified output parameters and specified differential information corresponding to all angle categories, the human body posture is classified and obtained.

[0036] On the other hand, the present application also provides a human body posture detection device, comprising:

[0037] A first determination module is used to determine the position information of the person to be detected in the target image;

[0038] A first detection module is used to perform key point detection on a person image corresponding to the person position information in the target image to obtain a plurality of human key points;

[0039] A second determination module, configured to determine a plurality of limb angles in the target image based on the plurality of human body key points;

[0040] The second detection module is used to determine the body posture of the person to be detected based on multiple limb angles corresponding to multiple consecutive frames of target images.

[0041] Furthermore, the present application also provides an electronic device, the electronic device comprising:

[0042] processor;

[0043] a memory for storing processor-executable instructions;

[0044] Wherein, the processor is configured to execute the above-mentioned human body posture detection method.

[0045] In addition, the present application also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by a processor to complete the above-mentioned human body posture detection method.

[0046] The present application scheme, after determining the position information of the person in the target image, can perform key point detection on the person image corresponding to the person position information in the target image to obtain multiple human key points, and determine multiple limb angles based on the human key points, and then identify the human posture based on the multiple limb angles corresponding to multiple consecutive frames of target images; after identifying the person image where the person to be detected is located, the present application scheme obtains multiple limb angles through key point detection and limb angle conversion, and can realize accurate human posture recognition based on the limb angles of multiple frames of target images, avoiding interference from complex backgrounds, and improving the recognition accuracy compared to directly identifying human posture by human key points. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solution of the embodiments of the present application, the drawings required for use in the embodiments of the present application are briefly introduced below.

[0048] Figure 1 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application;

[0049] Figure 2 A schematic diagram of a flow chart of a method for detecting a human body posture provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of a flow chart of a method for detecting a person's location provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of a feature extraction method according to an embodiment of the present invention;

[0052] Figure 5 A schematic diagram of a flow chart of a method for determining a human body posture provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of the structure of a method for determining a human body posture provided in an embodiment of the present application;

[0054] Figure 7 A schematic diagram of a flow chart of a method for determining a human body posture provided by another embodiment of the present application;

[0055] Figure 8A schematic diagram of the structure of a method for determining a human body posture provided by another embodiment of the present application;

[0056] Fig. 9 A block diagram of a human body posture detection device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0058] Similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0059] like Figure 1 As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12, Figure 1 A processor 11 is taken as an example. The processor 11 and the memory 12 are connected via a bus 10. The memory 12 stores instructions that can be executed by the processor 11. The instructions are executed by the processor 11 so that the electronic device 1 can execute all or part of the process of the method in the following embodiment. In one embodiment, the electronic device 1 can be an inspection robot equipped with a camera device, which is used to execute the human body posture detection method of the present application.

[0060] The memory 12 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0061] The present application also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by the processor 11 to complete the human body posture detection method provided in the present application.

[0062] See also Figure 2, is a flow chart of a method for detecting a human posture provided by an embodiment of the present application, such as Figure 2 As shown, the method may include the following steps 210 to 240.

[0063] Step 210: Determine the position information of the person to be detected in the target image.

[0064] The target image is an image that needs to be subjected to human posture detection; the target image can be an image of the workplace collected by the robot during the inspection process. The person appearing in the target image is the person to be detected.

[0065] The robot can collect images of the workplace through its onboard camera equipment (such as: gun camera, ball camera, panoramic camera, etc.) as the target image. In one case, during the inspection process, the robot can shoot the workplace and use each video frame as the target image. In another case, during the inspection process, the robot can shoot the area where people stay, and each time continuously shoot multiple images in a short period of time (for example: shoot 10 images in 1 second) as the target image.

[0066] For each frame of the target image, the robot can determine the position information of the person to be detected in the target image. The position information of the person can be expressed as the coordinates of any one of the upper left corner, upper right corner, lower left corner, lower right corner, and center point of the anchor box that defines the position of the person in the image coordinate system, as well as the length and width of the anchor box.

[0067] Step 220: Perform key point detection on the person image corresponding to the person position information in the target image to obtain a plurality of human body key points.

[0068] After obtaining the position information of the person in the target image, the robot can determine that there is a person to be detected in the target image. In this case, the body posture of the person to be detected needs to be detected. The robot can cut out the partial image where the person to be detected is located from the target image according to the position information of the person, as the person image, and detect the key points of the person image according to the key point detection algorithm to obtain multiple human key points. Here, the key point detection algorithm can be AlphaPose, which can be trained by the MPII data set.

[0069] Exemplarily, the detected human key points may include head, neck, right shoulder, left shoulder, right elbow, left elbow, left wrist, right wrist, chest, right hip, left hip, right knee, left knee, right ankle, and left ankle. The human key points may represent their positions in the target image according to the coordinates in the image coordinate system.

[0070] Performing key point detection after cropping the person image can reduce the interference of the image background on key point detection and greatly reduce the amount of calculation.

[0071] Step 230: Determine multiple limb angles in the target image based on multiple human body key points.

[0072] For any person image, after obtaining multiple human body key points corresponding to the person image, the robot can convert the human body key points into limb angles. The robot can determine a limb angle based on three human body key points.

[0073] Exemplarily, the robot can determine the head-neck-right-shoulder angle based on the coordinates of the three key points of the human body, namely the head, neck and right shoulder; determine the head-neck-left-shoulder angle based on the coordinates of the three key points of the human body, namely the head, neck and left shoulder; determine the neck-right-shoulder-right-elbow angle based on the coordinates of the three key points of the human body, namely the neck, left shoulder and right elbow; determine the neck-left-shoulder-left-elbow angle based on the coordinates of the three key points of the human body, namely the neck, left shoulder and left elbow; determine the right shoulder, right elbow and right wrist angle based on the coordinates of the three key points of the human body, namely the right shoulder, right elbow and right wrist; According to the coordinates of the three key points of the human body, the left shoulder, left elbow and left wrist, determine the angle between the left shoulder, left elbow and left wrist; according to the coordinates of the three key points of the human body, the chest, right hip and right knee, determine the angle between the chest, right hip and right knee; according to the coordinates of the three key points of the human body, the chest, left hip and left knee, determine the angle between the chest, left hip and left knee; according to the coordinates of the three key points of the human body, the right hip, right knee and right ankle, determine the angle between the right hip, right knee and right ankle; according to the coordinates of the three key points of the human body, the left hip, left knee and left ankle, determine the angle between the left hip, left knee and left ankle.

[0074] The robot can determine a limb angle based on two key points of the human body and a horizontal line. For example, the robot can determine the angle between the right knee and the right ankle horizontal line based on the coordinates of the two key points of the human body, the right knee and the right ankle, and the horizontal line in the image; the robot can determine the angle between the left knee and the left ankle horizontal line based on the coordinates of the two key points of the human body, the left knee and the left ankle, and the horizontal line in the image.

[0075] Step 240: Determine the body posture of the person to be detected based on multiple limb angles corresponding to the continuous multiple frames of target images.

[0076] The human body postures to be detected can be preset according to the application scenario requirements. For example, human body postures such as climbing over a wall, falling, smoking, running, and walking can be detected in the workplace.

[0077] For each frame of the target image, the robot can determine multiple limb angles. After executing the above steps for multiple consecutive frames of target images, multiple limb angles corresponding to the multiple frames of target images can be obtained. The robot can determine the human posture of the person to be detected based on the multiple limb angles corresponding to the target image, and accurately determine the human posture based on the multiple limb angles corresponding to the multiple consecutive frames of target images, avoiding misjudgment when detecting based on a single target image.

[0078] After obtaining the human posture, if the human posture is a designated human posture that requires an early warning, the robot can report early warning information containing the target image to the back-end device. Exemplarily, the designated human postures may include climbing over a wall, falling, and smoking. When the robot detects that the human posture of the person to be detected is any of the above designated human postures, it can report early warning information to avoid safety hazards in the workplace caused by illegal behaviors.

[0079] In one embodiment, when determining the position information of a person to be detected in a target image, the robot may use the target image as the image to be detected. The image to be detected is an image that needs to be subjected to target detection. The robot may input the image to be detected into a trained target detection model, thereby obtaining the position information of the person output by the target detection model.

[0080] Among them, the target detection model can be YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), Fast RCNN (Fast Regions with Convolutional Neural Networks features), Faster RCNN (Fastsr Regions with Convolutional Neural Networks features) and other models.

[0081] In one embodiment, the robot can be equipped with an infrared camera, which can collect images of the same visual field simultaneously with a conventional camera (visible light camera). In this case, the robot uses the visible light image collected by the visible light camera as the target image.

[0082] When determining the position information of the person to be detected in the target image, the robot can obtain an infrared image corresponding to the visible light image as the target image as the image to be detected. The infrared image is collected by an infrared camera device; the infrared image collected at the same time as the visible light image is the infrared image corresponding to the visible light image.

[0083] The robot can input the image to be detected into the trained target detection model to obtain the personnel location information output by the target detection model. The target detection model can be YOLO, SSD, Fast RCNN, Faster RCNN and other models. Since the infrared image corresponding to the visible light image has the same field of view and the same acquisition time as the visible light image, the personnel location information of the person to be detected in the infrared image is the personnel location information of the person to be detected in the visible light image (target image).

[0084] Since infrared images have better target detection effects, determining the location information of people based on infrared images can avoid missed detection and improve the accuracy of human posture detection.

[0085] In one embodiment, the object detection network may include a first feature extraction network and a second feature extraction network. Figure 3 , is a flow chart of a method for detecting a person's location provided in an embodiment of the present application, such as Figure 3 As shown, when the robot determines the personnel position information based on the target image or the infrared image, it can execute the following steps 310 to 330.

[0086] Step 310: extracting high-resolution image features from the image to be detected based on the first feature extraction network.

[0087] The first feature extraction network may be an adjusted VGG16 (Visual Geometry Group Network 16) network. Based on the conventional VGG16 network, the first fully connected layer of the VGG16 network may be changed to a convolutional layer with a convolution kernel of 3*3, the second fully connected layer of the VGG16 network may be changed to a convolutional layer with a convolution kernel of 1*1, and the third fully connected layer may be removed. The adjusted VGG16 network may pay more attention to the global background and target details.

[0088] The robot can input the target image into the adjusted VGG16 network, and extract high-resolution image features from the target image through the adjusted VGG16 network. Here, the image features can be feature maps.

[0089] Step 320: extracting the first designated image feature, the second designated image feature and the third designated image feature from the high-resolution image feature through the second feature extraction network.

[0090] After obtaining the high-resolution image features, the robot can calculate the high-resolution image features based on the second feature extraction network to obtain the first specified image features, the second specified image features and the third specified image features.

[0091] Step 330: Based on the first specified image feature, the second specified image feature and the third specified image feature, regress and obtain the person position information of the person to be detected.

[0092] The robot can perform bounding box regression based on the first specified image feature, the second specified image feature and the third specified image feature, and obtain the anchor frame of the person to be detected in the target image as the person's position information through non-maximum suppression.

[0093] In one embodiment, the second feature extraction network may include a first extraction module, a second extraction module, a third extraction module, and a fourth extraction module. Figure 4 , is a flow chart of a feature extraction method provided in an embodiment of the present application, such as Figure 4 As shown, when the robot extracts image features based on the second feature extraction network, it can execute the following steps 321-324.

[0094] Step 321: extracting medium-resolution image features from high-resolution image features based on the first extraction module.

[0095] Among them, the first extraction module may include a convolution layer with a convolution kernel of 1*1 and a convolution layer with a convolution kernel of 3*3 and a step size of 2.

[0096] After the robot inputs the high-resolution image features into the first extraction module, it performs convolution calculations on the high-resolution image features through two convolution layers to obtain medium-resolution image features.

[0097] Step 322: extracting low-resolution image features from the medium-resolution image features based on the second extraction module as first designated image features.

[0098] Among them, the second extraction module may include a convolution layer with a convolution kernel of 1*1 and a convolution layer with a convolution kernel of 3*3 and a step size of 2.

[0099] After the robot inputs the medium-resolution image features into the second extraction module, it performs convolution calculation on the medium-resolution image features through two convolution layers to obtain low-resolution image features, and the low-resolution image features can be used as the first designated image features.

[0100] Step 323: extracting a first intermediate image feature from the first designated image feature based on the third extraction module, and fusing the first intermediate image feature with the medium-resolution image feature to obtain a second designated image feature.

[0101] Among them, the third extraction module may include a convolution layer with a convolution kernel of 1*1 and an upsampling layer.

[0102] The robot inputs the first designated image feature into the third extraction module, performs convolution calculation through the convolution layer, and then performs upsampling processing to obtain the first intermediate image feature. The first intermediate image feature is the image feature output by the third extraction module.

[0103] The robot may fuse the first intermediate image feature and the medium-resolution image feature to obtain the second specified image feature. Here, the robot may perform convolution calculation after superimposing the first intermediate image feature and the medium-resolution image feature in the channel direction to achieve fusion processing.

[0104] Step 324: extracting a second intermediate image feature from the second designated image feature based on the fourth extraction module, and fusing the second intermediate image feature with the high-resolution image feature to obtain a third designated image feature.

[0105] Among them, the fourth extraction module may include a convolution layer with a convolution kernel of 1*1 and an upsampling layer.

[0106] The robot can input the second specified image feature into the fourth extraction module, perform upsampling after convolution calculation by the convolution layer, and obtain the second intermediate image feature. The second intermediate image feature is the image feature output by the fourth extraction module.

[0107] The robot may fuse the second intermediate image feature and the high-resolution image feature to obtain the third specified image feature. Here, the robot may perform convolution calculation after superimposing the first intermediate image feature and the medium-resolution image feature in the channel direction to achieve fusion processing.

[0108] The persons to be detected in the target image have different sizes in the image due to their different distances from the robot. By using image features of three resolutions to determine the specified image features for detection, the global features of the image can be effectively combined with the local features of the area where the persons to be detected are located, making the subsequent target detection results more accurate.

[0109] In one embodiment, see Figure 5 , is a flow chart of a method for determining a human body posture provided in an embodiment of the present application, such as Figure 5 As shown, when the robot determines the human body posture according to multiple limb angles corresponding to multiple consecutive frames of target images, it can execute the following steps 510-550.

[0110] Step 510: For each angle category, the angle information of multiple consecutive frames of target images in the angle category is input into the bidirectional recurrent neural network corresponding to the angle category to obtain multiple output parameters corresponding to the angle category.

[0111] See also Figure 6, is a schematic diagram of the architecture of a method for determining a human body posture provided by an embodiment of the present application, such as Figure 6 As shown, each angle category has a corresponding bidirectional recurrent neural network (BRNN).

[0112] Exemplarily, the angle categories may include the head-neck-right shoulder angle, the head-neck-left shoulder angle, the neck-right shoulder-right elbow angle, the neck-left shoulder-left elbow angle, the right shoulder-right elbow-right wrist angle, the left shoulder-left elbow-left wrist angle, the chest-right hip-right knee angle, the chest-left hip-left knee angle, the right hip-right knee-right ankle angle, the left hip-left knee-left ankle angle, the right knee-right ankle horizontal line angle, the left knee-left ankle horizontal line angle, and the like.

[0113] For each angle category, the robot can input the angle information of a specified number of consecutive target images into a bidirectional recurrent neural network, thereby obtaining multiple output parameters corresponding to the angle category. Here, the output parameter can be a single value. Exemplarily, the robot can input the angle information of 10 consecutive frames of target images at the head-neck-right-shoulder angle into a bidirectional recurrent neural network corresponding to the head-neck-right-shoulder angle, and obtain 10 output parameters.

[0114] After inputting multiple angle information of each angle category into the corresponding bidirectional recurrent neural network, multiple output parameters corresponding to all angle categories can be obtained.

[0115] Step 520: For each angle category, multiple output parameters corresponding to the angle category are merged to obtain a designated output parameter corresponding to the angle category.

[0116] like Figure 6 As shown, for multiple output parameters corresponding to any angle category, the robot can calculate the multiple output parameters through a nonlinear activation function and a fully connected layer, thereby fusing them to obtain the specified output parameters corresponding to the angle category.

[0117] Exemplarily, the robot fuses 10 output parameters corresponding to the head-neck-right-shoulder angle through a nonlinear activation function and a fully connected layer to obtain a specified output parameter corresponding to the head-neck-right-shoulder angle.

[0118] After fusing the output parameters of each angle category, the robot can obtain the specified output parameters corresponding to all angle categories.

[0119] Step 550: Based on the specified output parameters corresponding to all angle categories, classify and obtain the human body posture.

[0120] After obtaining the specified output parameters corresponding to all angle categories, such as Figure 6As shown, the robot can fuse multiple specified output parameters through nonlinear activation functions and fully connected layers, and normalize the fusion results through normalization functions to obtain a multidimensional vector. The number of elements in the multidimensional vector is the same as the preset human posture types, and each element corresponds to a human posture category, indicating the confidence of the human posture.

[0121] Exemplarily, the present application scheme detects five human postures, namely, climbing over a wall, falling, smoking, running, and walking. Then, the multidimensional vector contains 5 elements, each of which is the confidence level of the corresponding human posture.

[0122] The robot can determine that the human body posture with the highest confidence is the human body posture of the person to be detected.

[0123] In one embodiment, see Figure 7 , is a flow chart of a method for determining a human body posture provided by another embodiment of the present application, such as Figure 7 As shown, Figure 7 As shown, when the robot determines the human body posture according to multiple limb angles corresponding to multiple consecutive frames of target images, it can execute the following steps 510-551.

[0124] Step 510: For each angle category, the angle information of multiple consecutive frames of target images in the angle category is input into the bidirectional recurrent neural network corresponding to the angle category to obtain multiple output parameters corresponding to the angle category.

[0125] Step 520: For each angle category, multiple output parameters corresponding to the angle category are merged to obtain a designated output parameter corresponding to the angle category.

[0126] See also Figure 8 , is a schematic diagram of the architecture of a method for determining a human body posture provided by another embodiment of the present application, such as Figure 8 As shown, each angle category has a corresponding bidirectional recurrent neural network.

[0127] After executing step 510 and step 520, the robot can obtain the specified output parameters corresponding to all angle categories.

[0128] Step 530: For each angle category, determine a plurality of differential information corresponding to the angle category according to the angle information of a plurality of consecutive frames of target images in the angle category.

[0129] For any angle category, the robot can subtract the angle information of the previous frame of the target image in the angle category from the angle information of the next frame of the target image in the angle category, thereby obtaining a plurality of differential information. For example, the robot performs human posture detection based on 10 consecutive frames of target images and can calculate 9 differential information corresponding to the angle between the head, neck and right shoulder.

[0130] After processing the angle information of each angle category, the robot can obtain multiple sets of differential information. For example, there are 12 angle categories and 10 frames of target images, and 12 sets of differential information can be determined, each set containing 9 differential information.

[0131] Step 540: For each angle category, multiple differential information corresponding to each angle category are merged to obtain designated differential information corresponding to the angle category.

[0132] like Figure 8 As shown, for multiple differential information corresponding to any angle category, the robot can calculate the multiple differential information through the fully connected layer, thereby fusing the specified differential information corresponding to the angle category.

[0133] Exemplarily, the robot can fuse 9 differential information corresponding to the head-neck-right-shoulder angle through a fully connected layer to obtain specified differential information corresponding to the head-neck-right-shoulder angle.

[0134] After fusing the differential information of each angle category, the robot can obtain the specified differential information corresponding to all angle categories.

[0135] Step 551: Based on the specified output parameters and specified differential information corresponding to all angle categories, classify and obtain the human body posture.

[0136] After obtaining the specified output parameters and specified differential information corresponding to all angle categories, such as Figure 8 As shown, the robot can fuse multiple specified output parameters and multiple specified differential information through nonlinear activation functions and fully connected layers, and further calculate the fusion results through nonlinear activation functions and fully connected layers, and obtain a multidimensional vector after normalizing the calculation results. The number of elements in the multidimensional vector is the same as the preset human posture types, and each element corresponds to a human posture category, indicating the confidence of the human posture.

[0137] The robot can determine that the human body posture with the highest confidence is the human body posture of the person to be detected.

[0138] Through the above measures, the angle information of the same angle category is calculated to obtain differential information and then used for human posture detection, so that a more accurate human posture can be obtained. For human postures with similar final postures but different in essence, such as falling and sitting down to rest, lying down to sleep and lying down after falling, etc., the differential information of limb angles between consecutive multi-frame images can be introduced to fully learn the action information that causes the human posture, so as to distinguish similar but different human postures.

[0139] Fig. 9 A human body posture detection device according to an embodiment of the present invention is provided. Fig. 9As shown, the device may include:

[0140] A first determination module 910 is used to determine the position information of the person to be detected in the target image;

[0141] A first detection module 920 is used to perform key point detection on the person image corresponding to the person position information in the target image to obtain a plurality of human key points;

[0142] A second determination module 930 is used to determine a plurality of limb angles in the target image based on the plurality of human body key points;

[0143] The second detection module 940 is used to determine the body posture of the person to be detected according to multiple limb angles corresponding to multiple consecutive frames of target images.

[0144] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned human body posture detection method, and will not be repeated here.

[0145] In several embodiments provided in the present application, the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and a part of a module, a program segment or a code contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0147] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

Claims

1. A method for detecting human body posture, characterized in that: include: Determine the position information of the person to be detected in the target image; For the person image corresponding to the person position information in the target image, key point detection is performed to obtain a plurality of human key points; Based on the multiple human body key points, determining multiple limb angles in the target image; Determining the human body posture of the person to be detected according to multiple limb angles corresponding to multiple consecutive frames of target images; wherein the plurality of limb angles include angle information corresponding to a plurality of angle categories; The method of determining the human posture of the person to be detected according to the multiple limb angles corresponding to the continuous multiple frames of target images includes: For each angle category, input the angle information of the continuous multiple frames of target images in the angle category into a bidirectional recurrent neural network corresponding to the angle category to obtain multiple output parameters corresponding to the angle category; For multiple output parameters corresponding to each angle category, a specified output parameter corresponding to the angle category is obtained by fusing; Based on the specified output parameters corresponding to all angle categories, the human body posture is classified.

2. The method according to claim 1, characterized in that The step of determining the position information of the person to be detected in the target image includes: The target image is used as an image to be detected, and the image to be detected is input into a trained target detection model to obtain personnel location information output by the target detection model.

3. The method according to claim 1, characterized in that The target image is a visible light image; The step of determining the position information of the person to be detected in the target image includes: Acquire an infrared image corresponding to the visible light image as an image to be detected; The image to be detected is input into a trained target detection model to obtain the personnel location information output by the target detection model.

4. The method according to claim 2 or 3, characterized in that: The target detection model includes a first feature extraction network and a second feature extraction network; The step of inputting the image to be detected into a trained target detection model to obtain the personnel location information output by the target detection model includes: Extracting high-resolution image features from the image to be detected based on the first feature extraction network; Extracting a first designated image feature, a second designated image feature, and a third designated image feature from the high-resolution image features through the second feature extraction network; According to the first designated image feature, the second designated image feature and the third designated image feature, the person position information of the person to be detected is obtained by regression.

5. The method according to claim 4, characterized in that The second feature extraction network includes a first extraction module, a second extraction module, a third extraction module and a fourth extraction module; The extracting the first designated image feature, the second designated image feature and the third designated image feature from the high-resolution image feature by the second feature extraction network comprises: Extracting medium-resolution image features from the high-resolution image features based on the first extraction module; Extracting low-resolution image features from the medium-resolution image features based on the second extraction module as the first designated image features; Extracting a first intermediate image feature from the first designated image feature based on the third extraction module, and fusing the first intermediate image feature with the medium-resolution image feature to obtain a second designated image feature; A second intermediate image feature is extracted from the second designated image feature based on the fourth extraction module, and the second intermediate image feature is fused with the high-resolution image feature to obtain a third designated image feature.

6. The method according to claim 1, characterized in that Before classifying and obtaining the human body posture based on the specified output parameters corresponding to all angle categories, the method further includes: For each angle category, determining a plurality of differential information corresponding to the angle category according to angle information of a plurality of consecutive frames of target images in the angle category; For each angle category, multiple differential information corresponding to each angle category are merged to obtain designated differential information corresponding to the angle category; The method of classifying the human body posture based on the specified output parameters corresponding to all angle categories includes: Based on the specified output parameters and specified differential information corresponding to all angle categories, the human body posture is classified and obtained.

7. A human body posture detection device, characterized in that: include: A first determination module is used to determine the position information of the person to be detected in the target image; A first detection module is used to perform key point detection on a person image corresponding to the person position information in the target image to obtain a plurality of human key points; A second determination module, configured to determine a plurality of limb angles in the target image based on the plurality of human body key points; A second detection module is used to determine the human body posture of the person to be detected based on multiple limb angles corresponding to multiple consecutive frames of target images; wherein the plurality of limb angles include angle information corresponding to a plurality of angle categories; The method of determining the human posture of the person to be detected according to the multiple limb angles corresponding to the continuous multiple frames of target images includes: For each angle category, input the angle information of the continuous multiple frames of target images in the angle category into a bidirectional recurrent neural network corresponding to the angle category to obtain multiple output parameters corresponding to the angle category; For multiple output parameters corresponding to each angle category, a specified output parameter corresponding to the angle category is obtained by fusing; Based on the specified output parameters corresponding to all angle categories, the human body posture is classified.

8. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to execute the human body posture detection method described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program can be executed by a processor to complete the human body posture detection method described in any one of claims 1-6.

Images