Fall posture detection method, device, electronic device and readable storage medium

The fall posture detection method integrates multiple models for real-time and accurate posture detection, addressing the limitations of existing technologies by combining VGG19 and SSD for key point and bounding box detection, and SVM for posture validation, thereby improving detection accuracy and robustness.

CN114596533BActive Publication Date: 2024-12-06NORTH CHINA UNIVERSITY OF TECHNOLOGY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210065950.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-20
Publication Date
2024-12-06
Estimated Expiration
2042-01-20

AI Technical Summary

Technical Problem

Existing multi-person posture detection technologies face challenges in achieving real-time detection with high accuracy, particularly in scenes with severe occlusion, and existing methods are either computationally intensive or fail to detect individuals accurately.

Method used

A fall posture detection method that combines a posture detection model and a target detection model to correct detection results, using a VGG19 convolutional network for key point detection and an SSD for bounding boxes, utilizing the Hungarian algorithm and NMS for stabilization, and SVM binary classifier for posture validation, and SVM binary classifier for posture detection.

Benefits of technology

The method effectively improves the recall rate of posture detection while ensuring real-time performance by correcting detection results using multiple models, enhancing accuracy and robustness in complex scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114596533B_ABST
    Figure CN114596533B_ABST
Patent Text Reader

Abstract

In the embodiment of the present application, a fall posture detection method, device, electronic device and readable storage medium are provided, which belongs to the field of image processing technology. The method includes: performing preset processing on the monitoring video to obtain an image to be identified; detecting the image to be identified according to a preset posture detection model to obtain a first detection result based on posture detection; detecting the image to be identified according to a preset target detection model to obtain a second detection result based on target detection; correcting the first detection result by the second detection result to obtain a corrected third detection result, the third detection result including a third human body boundary box and / or a second human body key point; judging whether the human body object in the image to be identified is in a fall posture according to the third detection result. Through the processing scheme disclosed in the present invention, the recall rate of detection can be effectively improved while ensuring the real-time detection of multi-person posture detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a fall posture detection method, device, electronic device and readable storage medium. Background Art

[0002] With the rapid development of people's living standards and urban infrastructure in my country, public transportation facilities such as public buses, high-speed trains and airplanes are facing the need to realize intelligent improvements to enhance passenger experience. The continuous fall phenomenon at public transportation facilities is a common dangerous situation for passengers in such scenarios. Timely detection of passenger falls and alarms are of great significance for the safety and handling of dangerous situations.

[0003] Existing multi-person posture detection technologies are divided into two implementation methods: top-down and bottom-up. The top-down method first uses the target detection algorithm to detect the human body bounding box in the image, and then runs single-person detection on each bounding box image. This method often has a huge amount of calculation and is difficult to achieve real-time detection. The bottom-up method first detects the key points of the human body and then connects them into a complete human body. Although this method can achieve real-time detection, it may not detect some people in scenes with severe occlusion.

[0004] Therefore, there is an urgent need for a posture detection method that can perform real-time detection and has high accuracy. Summary of the invention

[0005] In view of this, in order to solve the above technical problems, the embodiments of the present application provide a fall posture detection method, device, electronic device and readable storage medium, and the specific solutions are as follows:

[0006] In a first aspect, an embodiment of the present application provides a fall posture detection method, the fall posture detection method comprising:

[0007] Perform preset processing on the surveillance video to obtain the image to be identified;

[0008] Detecting the image to be recognized according to a preset posture detection model to obtain a first detection result based on posture detection, wherein the first detection result includes a first human body boundary box and a first human body key point;

[0009] Detect the image to be identified according to a preset target detection model to obtain a second detection result based on target detection, wherein the second detection result includes a second human body bounding box;

[0010] Correcting the first detection result by using the second detection result to obtain a corrected third detection result, where the third detection result includes a third human body bounding box and / or a second human body key point;

[0011] It is determined whether the human body object in the image to be identified is in a falling posture according to the third detection result.

[0012] According to a specific implementation of the embodiment of the present application, the step of detecting the image to be recognized according to a preset posture detection model to obtain a first detection result based on posture detection includes:

[0013] Extracting features from the image to be identified based on a VGG19 convolutional network to obtain a feature image group having a preset number of feature maps;

[0014] Extracting the confidence image group and the relevance image group from the feature image group based on a convolutional neural network respectively;

[0015] Generating predicted key point positions according to the confidence image group;

[0016] Connecting all predicted key point positions according to the association image group and the Hungarian algorithm to obtain a plurality of basic human frames corresponding to the image to be identified;

[0017] A minimum tangent bounding box is generated according to each basic human body frame to obtain a plurality of groups of first human body frames corresponding to the image to be identified and human body key points corresponding to each first human body bounding box.

[0018] According to a specific implementation of the embodiment of the present application, the step of detecting the image to be identified according to a preset target detection model to obtain a second detection result based on target detection includes:

[0019] Performing human body detection on the image to be identified according to a preset convolutional neural network to obtain an initial predicted human body bounding box;

[0020] The initial predicted human body bounding box is screened based on an NMS non-maximum suppression algorithm to obtain the second human body bounding box.

[0021] According to a specific implementation of the embodiment of the present application, before the step of correcting the first detection result by using the second detection result to obtain a corrected third detection result, the method further includes:

[0022] Calculating a matching score between each first human body bounding box in the first detection result and each second human body bounding box in the second detection result based on a preset weight allocation rule and a similarity calculation rule;

[0023] Combining the first human body bounding box and the second human body bounding box with a matching score greater than a preset score threshold into a to-be-selected matching pair;

[0024] The corresponding first human body bounding box and the human body key points corresponding to the first human body bounding box are corrected according to the second human body bounding box in each candidate matching pair.

[0025] According to a specific implementation of an embodiment of the present application, the step of correcting the corresponding first human body bounding box and the human body key points corresponding to the first human body bounding box according to the second human body bounding box in each to-be-selected matching pair includes:

[0026] In a candidate matching pair, selecting a human body bounding box with the highest confidence in the candidate matching pair from among the first human body bounding box and the second human body bounding box as a third human body bounding box, wherein the confidences of the first human body bounding box and the second human body bounding box are both greater than a preset confidence threshold;

[0027] All human key points within the third human body bounding box are retained, and all human key points outside the third human body bounding box are moved according to a preset rule, wherein the preset rule is to move all human key points outside the third human body bounding box to the third human body bounding box and an extension line of the bounding box.

[0028] According to a specific implementation of the embodiment of the present application, the step of judging whether the human object in the image to be identified is in a falling posture according to the third detection result includes:

[0029] When the third detection result includes only a third human body bounding box, comparing the aspect ratio of the third human body bounding box with a preset aspect ratio threshold, and marking the human body corresponding to the third human body bounding box having an aspect ratio greater than the preset aspect ratio threshold as being in a falling state;

[0030] When the third detection result includes a third human body bounding box and human body key points, the third human body bounding box and the human body key points are input into an SVM binary classifier for cross-validation by the SVM binary classifier, and whether the human body corresponding to the third human body bounding box is in a falling state is determined according to the cross-validation result.

[0031] According to a specific implementation of the embodiment of the present application, the step of performing preset processing on the surveillance video to obtain the image to be identified includes:

[0032] Performing frame cutting processing on the surveillance video to obtain initial images of continuous frames;

[0033] Each frame of the initial image is standardized in turn to obtain consecutive frames of images to be recognized with the same resolution and size.

[0034] In a second aspect, an embodiment of the present application provides a fall posture detection device, the fall posture detection device comprising:

[0035] An acquisition module is used to perform preset processing on the surveillance video to obtain an image to be identified;

[0036] A first detection module, configured to detect the image to be identified according to a preset posture detection model to obtain a first detection result based on posture detection, wherein the first detection result includes a first human body boundary box and a first human body key point;

[0037] A second detection module, configured to detect the image to be identified according to a preset target detection model to obtain a second detection result based on target detection, wherein the second detection result includes a second human body bounding box;

[0038] a correction module, configured to correct the first detection result by using the second detection result to obtain a corrected third detection result, wherein the third detection result includes a third human body bounding box and / or a second human body key point;

[0039] A judgment module is used to judge whether the human object in the image to be identified is in a falling posture according to the third detection result.

[0040] In a third aspect, an embodiment of the present application further provides an electronic device, the electronic device comprising:

[0041] at least one processor; and,

[0042] a memory communicatively connected to the at least one processor; wherein,

[0043] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the fall posture detection method in the aforementioned first aspect or any implementation manner of the first aspect.

[0044] In a fourth aspect, an embodiment of the present application further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the fall posture detection method in the aforementioned first aspect or any implementation of the first aspect.

[0045] The fall posture detection method, device, electronic device and readable storage medium in the embodiment of the present application, the fall posture detection method includes: performing preset processing on the monitoring video to obtain an image to be identified; detecting the image to be identified according to a preset posture detection model to obtain a first detection result based on posture detection, the first detection result includes a first human body boundary box and a first human body key point; detecting the image to be identified according to a preset target detection model to obtain a second detection result based on target detection, the second detection result includes a second human body boundary box; correcting the first detection result by the second detection result to obtain a corrected third detection result, the third detection result includes a third human body boundary box and / or a second human body key point; judging whether the human body object in the image to be identified is a fall posture according to the third detection result. The scheme of the present application detects the image to be identified respectively by a posture detection model and a target detection model, and corrects the detection result obtained by the posture detection model based on the detection result obtained by the target detection model, so that the recall rate of the detection can be effectively improved while ensuring the real-time detection of multi-person posture detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 A schematic diagram of a method flow of a fall posture detection method provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of the position of a first key point of a human body in a fall posture detection method provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of a construction process of a candidate matching pair for a fall posture detection method provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of a third human body bounding box area division step of a fall posture detection method provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of a device module of a fall posture detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0053] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0054] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0055] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0056] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0057] The embodiment of the present application provides a fall posture detection method. The fall posture detection method provided in this embodiment can be executed by a computing device, which can be implemented as software, or as a combination of software and hardware, and the computing device can be integrated in a server, terminal device, etc.

[0058] refer to Figure 1 , is a method flow diagram of a fall posture detection method provided in an embodiment of the present application. The fall posture detection method provided in an embodiment of the present application is as follows: Figure 1 As shown, the fall posture detection method includes:

[0059] Step S101, performing preset processing on the surveillance video to obtain an image to be identified;

[0060] In a specific embodiment, a monitoring device can be set in any scene, and the monitoring device is any terminal device that can obtain video data. The monitoring device is connected to the electronic device that executes the fall posture detection method, and is used to send monitoring video data to the electronic device.

[0061] After acquiring the monitoring video data, the electronic device performs corresponding frame cutting and standardization processing on the monitoring video to obtain an image to be recognized that can be used for posture detection.

[0062] According to a specific implementation of the embodiment of the present application, the step of performing preset processing on the surveillance video to obtain the image to be identified includes:

[0063] Performing frame cutting processing on the surveillance video to obtain initial images of continuous frames;

[0064] Each frame of the initial image is standardized in turn to obtain consecutive frames of images to be recognized with the same resolution and size.

[0065] In a specific embodiment, the number of the images to be identified may be one or more. The embodiment of the present application does not specifically limit the number of the images to be identified.

[0066] The monitoring video is subjected to a preset frame cutting process to obtain an initial image group having multiple continuous frames. The initial image group is subjected to a preset standardization process to obtain multiple continuous frames of to-be-recognized images having a fixed resolution and size.

[0067] By processing the surveillance video into images, the subsequent operations based on various convolutional neural network models can be guaranteed to be more stable. After obtaining multiple consecutive frames of images to be identified, the images to be identified are output frame by frame to detect the falling posture in each image to be identified.

[0068] Step S102, detecting the image to be recognized according to a preset posture detection model to obtain a first detection result based on posture detection, wherein the first detection result includes a first human body bounding box and a first human body key point;

[0069] In a specific embodiment, the preset posture detection model can adopt a bottom-up multi-person posture detection algorithm such as the OpenPose posture detection algorithm. It should be noted that the preset posture detection model can adopt any bottom-up multi-person posture detection algorithm, and the preset posture detection model in this embodiment is not specifically limited.

[0070] Specifically, the first human key points include points corresponding to the elbows, shoulders, and wrists of the human hands, points corresponding to the ankles, knees, and hips of the human legs, points corresponding to the hips and buttocks of the human body, and points corresponding to the eyes, chin, and top of the head.

[0071] like Figure 2 As shown, this embodiment uses a 14-point marking method for marking and detecting the first human key points, that is, 0 represents the center of the head, 1 represents the neck, 2 represents the right shoulder, 3 represents the right elbow, 4 represents the right hand, 5 represents the left shoulder, 6 represents the left elbow, 7 represents the left hand, 8 represents the right hip, 9 represents the right knee, 10 represents the right foot, 11 represents the left hip, 12 represents the left knee, and 13 represents the left foot.

[0072] The first human body bounding box is a minimum tangent box generated according to the detected human body skeleton.

[0073] The image to be identified may include one human body object or multiple human body objects, and the number of human bodies in the image to be identified is not limited here.

[0074] According to a specific implementation of the embodiment of the present application, the step of detecting the image to be recognized according to a preset posture detection model to obtain a first detection result based on posture detection includes:

[0075] Extracting features from the image to be identified based on a VGG19 convolutional network to obtain a feature image group having a preset number of feature maps;

[0076] Extracting the confidence image group and the relevance image group from the feature image group based on a convolutional neural network respectively;

[0077] Generating predicted key point positions according to the confidence image group;

[0078] Connecting all predicted key point positions according to the association image group and the Hungarian algorithm to obtain a plurality of basic human frames corresponding to the image to be identified;

[0079] A minimum tangent bounding box is generated according to each basic human body frame to obtain a plurality of groups of first human body frames corresponding to the image to be identified and human body key points corresponding to each first human body bounding box.

[0080] In a specific embodiment, the VGG19 convolutional network is a 19-layer deep convolutional neural network. By extracting features in the image to be identified through the VGG19 convolutional neural network, a set of feature graphs having multiple feature graphs can be obtained.

[0081] After obtaining the feature map group, based on the pre-trained OpenPose posture detection model, the confidence image group PCMs and the association image group PAFs in the feature map group are extracted through two parallel branches. Specifically, the confidence image group is a heat map responsible for predicting the position of key points, and the association image group is a vector map responsible for predicting the relationship between key point pairs.

[0082] After the predicted key point position is generated according to the confidence image group, the first human key point in the first detection result is obtained.

[0083] The association image group PAFs is also called affinity domain image group, through which the affinity domain image group can characterize the affinity between each pixel in the image and the human limbs, and the affinity PAF corresponding to each pixel can be identified as a unit vector in the direction of the limbs. Based on the association image group and the Hungarian algorithm, the first human key points can be connected to obtain multiple basic human frames corresponding to the number of human objects in the image to be identified.

[0084] According to each boundary of each basic human body frame, a corresponding minimum tangent boundary box is generated, that is, the first human body boundary box in the first detection result is obtained.

[0085] Specifically, the training process of the OpenPose posture detection model is:

[0086] Step 1: construct an image training set and annotate the key points of the human body in the image training set, wherein the image training set can be a continuous image captured from any video data.

[0087] Step 2: Generate the true label S of the confidence map group PCMs based on the annotated human key points * Each image generates n heat maps that conform to the normal distribution, where n is the number of key point classes in the image. When the pixel point p is close to the annotation point x j,k When , the normal distribution curve reaches its peak. The value of the pixel p in the PCM confidence map of the jth root node of the kth person is Where σ is a parameter that controls the spread of the peak, and exp() is an exponential function with the natural constant e as the base. The final confidence value of the jth key point at pixel p is the average value of all human bodies in the image.

[0088] Step 3: Generate the true label L of the association graph group PAFs based on the annotated human key points * Each image also generates m vector graphs, where m is the number of key point classes in the image. The value of pixel p in the PAF correlation graph of the cth limb of the kth person is The judgment condition of pixel point p on the limb is: 0≤v·(px j1,k )≤l c,k and|v ⊥ ·(px j1,k )|≤σ l , where σ l is the limb width, l c,k =||x j2,k -x j1,k ||2 is the limb length, v ⊥ is the perpendicular vector of the direction unit vector v. The actual value of the partial correlation degree for the cth limb at the final pixel point p is the average value of all human bodies in the image.

[0089] Step 4: Calculate the loss based on the true value and the predicted value. In order to allow the network to predict the confidence in the first branch and the correlation in the second branch, a loss function is added after each branch at each stage. The loss function is L2 loss. Considering that not all people are labeled in actual scenarios, weights are added to the loss function from a spatial perspective. The specific loss function is as follows:

[0090]

[0091]

[0092] in: is the loss function used to predict the confidence map, is the loss function used to predict the association degree, W is a binary mask, and when a key point is not marked at position p in the image, W(p) = 0. The final total loss function is

[0093] Step 5: Calculate the gradient of the loss value and back-propagate the network parameters according to the gradient value.

[0094] Step six, repeat the training according to the above steps and step five.

[0095] Step S103, detecting the image to be recognized according to a preset target detection model to obtain a second detection result based on target detection, wherein the second detection result includes a second human body bounding box;

[0096] In a specific embodiment, the preset target detection model may be a model constructed using an SSD target detection algorithm, or a model constructed using other target detection algorithms that can run in real time. The type of the target detection model is not specifically limited here.

[0097] Specifically, after the image to be identified is imported into the pre-trained SSD object detection model, a second human body bounding box corresponding to each human body in the image to be identified can be obtained.

[0098] According to a specific implementation of the embodiment of the present application, the step of detecting the image to be identified according to a preset target detection model to obtain a second detection result based on target detection includes:

[0099] Performing human body detection on the image to be identified according to a preset convolutional neural network to obtain an initial predicted human body bounding box;

[0100] The initial predicted human body bounding box is screened based on an NMS non-maximum suppression algorithm to obtain the second human body bounding box.

[0101] In a specific embodiment, the preset convolutional neural network is a neural network model that is pre-trained by the user through a training set. For a human object in the image to be identified, there may be multiple initial predicted human bounding boxes obtained based on the convolutional neural network model, and each initial predicted human bounding box is offset within the preset range of the human object. For example, for overlapping human objects in the image to be identified, multiple initial predicted human bounding boxes may appear at the same position.

[0102] The initial predicted human body bounding box is subjected to NMS non-maximum suppression, thereby ensuring that only one human body bounding box with a maximum value among the corresponding overlapping initial predicted human body bounding boxes is output.

[0103] After the initial predicted human body bounding box is screened by the NMS non-maximum suppression algorithm, a plurality of second human body bounding boxes corresponding to the identifiable human body in the image to be identified can be obtained.

[0104] Specifically, the training process of the SSD target detection model is:

[0105] Step 1: construct an image training set and annotate the human body bounding boxes in the image training set.

[0106] Step 2: Use the constructed convolutional neural network to obtain feature maps of different scales.

[0107] Step 3: Use the default box generation rules to generate a default box for each feature map.

[0108] Step 4: Calculate the similarity matrix between the default box and the annotated bounding box.

[0109] Step 5: Generate the matching results of the default box and the annotated bounding box. Specifically, first find the default box with the largest IOU with each annotated bounding box, and match the two to ensure that each annotated bounding box can match at least one default box.

[0110] Then, the remaining bounding boxes are tried to match with the default box one by one, and as long as the intersection-over-union ratio (IOU) between the two is greater than the matching threshold β, the bounding box and the default box are matched. The matched default box is marked as a positive sample, and the unmatched default box is marked as a negative sample.

[0111] Step 6: assign a target category to each default box according to the matching results. Since the SSD target detection model in this embodiment only detects human bodies, all default boxes are classified as human bodies. At the same time, the actual position adjustment amount t is calculated for the matched default box. x ,t y ,t w ,t h

[0112] Step 7: Use the feature map to predict the target classification probability and position adjustment amount t for each bounding box xp ,t yp ,t wp ,t hp .

[0113] Step 8: Randomly sample the default boxes from all default boxes for loss function calculation.

[0114] Step 9: Calculate the loss function. Specifically: the total loss function L(x,c,l,g) is defined as the position error L loc (x,l,g) and confidence error L conf The weighted sum of (x,c), Where N is the number of positive samples in the default box, c is the category confidence prediction value, l is the position prediction value of the bounding box corresponding to the default box, g is the position parameter of the default box, and α is the weight coefficient.

[0115] The position error uses smoothL1 loss:

[0116]

[0117]

[0118] where x ij ∈{0,1}, if x ij= 1, it means that the i-th default box matches the j-th annotated bounding box, (cx, cy) is the center coordinate of the default box d, w is the width of the default box, and h is the height of the default box. and are the horizontal and vertical coordinates of the center of the j-th annotated bounding box, and are the horizontal and vertical coordinates of the center of the i-th default box, and is the width and height of the jth annotated bounding box, and is the width and height of the i-th default box.

[0119] In addition, the confidence error uses softmax loss: Among them, p is the real target type; like Indicates that the i-th default box matches the j-th annotated bounding box.

[0120] Step 10: Calculate the gradient of the loss value and back-propagate and update the network parameters according to the gradient value.

[0121] Step 11: Perform cycle training according to the above steps 1 to 10.

[0122] Step S104, correcting the first detection result by using the second detection result to obtain a corrected third detection result, wherein the third detection result includes a third human body bounding box and / or a second human body key point;

[0123] In a specific embodiment, after obtaining the first human body bounding box, the first human body key points corresponding to the first human body bounding box, and the second human body bounding box based on a preset posture detection model and a preset target detection model, the first human body bounding box and the first key points need to be corrected according to the second human body bounding box.

[0124] Specifically, the correction step includes selecting the first human body bounding box or the second human body bounding box as a third human body bounding box for final fall posture judgment; and screening the position of the first key point.

[0125] There may be two situations for the third detection result. Situation 1 is that the third detection result only includes a third human body bounding box; situation 2 is that the third detection result includes a third human body bounding box and a second human body key point corresponding to the third human body bounding box.

[0126] According to a specific implementation of the embodiment of the present application, before the step of correcting the first detection result by using the second detection result to obtain a corrected third detection result, the method further includes:

[0127] Calculating a matching score between each first human body bounding box in the first detection result and each second human body bounding box in the second detection result based on a preset weight allocation rule and a similarity calculation rule;

[0128] Combining the first human body bounding box and the second human body bounding box with a matching score greater than a preset score threshold into a to-be-selected matching pair;

[0129] The corresponding first human body bounding box and the human body key points corresponding to the first human body bounding box are corrected according to the second human body bounding box in each candidate matching pair.

[0130] In a specific embodiment, for the same frame of image to be recognized, it is also necessary to match the first human body bounding box based on posture detection obtained by the above embodiment with the second human body bounding box based on target detection to ensure the correction effect and accuracy.

[0131] The matching method adopts the Hungarian algorithm, such as Figure 3 As shown, the specific matching process is:

[0132] Step 1: Calculate the similarity score S between each first human bounding box in the first human bounding box group and each second bounding box in the second human bounding box group. m Compared with the intersection score S IOU , calculate the total matching score Due to S m , S IOU ∈(0,1), and the similarity score is more important than the intersection-union ratio score, so the square root of the similarity score increases the weight, and the square of the intersection-union ratio score decreases the weight.

[0133] Step 2: Use the first human bounding box as the matching element and the second human bounding box as the matched element. Randomly sort all matching elements and perform matching among all matched elements. Set the matching score greater than the matching threshold β p The matching elements and the matched elements are constructed as a candidate matching pair.

[0134] Step 3: Check whether there are any unprocessed matching elements. If there are no unprocessed matching elements, it means that all elements have been matched. If there are unprocessed matching elements, proceed to the next step of judgment.

[0135] Step 4: Select the first unprocessed matching element with the highest ranking, and determine whether there is a matching pair to be selected for the first matching element. If there is no matching pair to be selected, mark the first matching element as a processed element. If there is a matching pair to be selected, proceed to the next step of determination.

[0136] Step 5: Select the candidate matching pair with the largest score of the first matching element, and determine whether the first matching element and the corresponding matched element have been matched. If no match is established, a matching relationship between the first matching element and the corresponding matched element is established to obtain a first candidate matching pair, and then return to step 3. If a match is established, proceed to the next step.

[0137] Step 6: Check whether the matching score of the first candidate matching pair corresponding to the first matching element is greater than the original candidate matching pair of the first matching element. If the score of the first candidate matching pair is greater than the score of the original candidate matching pair of the first matching element, replace the matched element in the original candidate matching pair with the matched object in the first candidate matching pair, record the match, mark the first matching element as processed, and mark the replaced matched element as unprocessed, and return to the above step 3. If the score of the first candidate matching pair is less than or equal to the score of the original candidate matching pair of the first matching element, delete the first candidate matching pair and return to the above step 5.

[0138] Step 7: When all matching elements are marked as processed, the matching process is completed. Finally, the best match between the two sets of frames is established, and the two matching frames indicate that the two frames are the same human target.

[0139] According to a specific implementation of an embodiment of the present application, the step of correcting the corresponding first human body bounding box and the human body key points corresponding to the first human body bounding box according to the second human body bounding box in each to-be-selected matching pair includes:

[0140] In a candidate matching pair, selecting a human body bounding box with the highest confidence in the candidate matching pair from among the first human body bounding box and the second human body bounding box as a third human body bounding box, wherein the confidences of the first human body bounding box and the second human body bounding box are both greater than a preset confidence threshold;

[0141] All human key points within the third human body bounding box are retained, and all human key points outside the third human body bounding box are moved according to a preset rule, wherein the preset rule is to move all human key points outside the third human body bounding box to the third human body bounding box and an extension line of the bounding box.

[0142] In a specific embodiment, after all candidate matching pairs between the first human body bounding box and the second human body bounding box are constructed, a preset correction action is performed on each candidate matching pair.

[0143] The step of correcting the human body bounding box includes: calculating the confidence of the first human body bounding box and the second human body bounding box respectively, and taking the human body bounding box with the highest confidence among the first human body bounding box and the second human body bounding box as the third human body bounding box.

[0144] The step of correcting the key points of the human body includes: retaining all the key points of the human body located in the third human body boundary frame. For the key points of the human body located outside the third human body boundary frame, processing them by region. Figure 4 As shown, the boundary line of the third human body bounding box is extended to obtain eight corresponding areas of area A, area B, area C, area D, area E, area F, area G and area H. For the human body key points in areas A, C, E and G, the human body key points are directly displaced to the four vertices of the third human body bounding box. For the human body key points in areas B, D, F and H, the human body key points are displaced to the corresponding extension lines.

[0145] In actual application scenarios, the human body object in the image to be identified may have three states.

[0146] State 1: The human object is detected by the SSD object detector but not by the OpenPose posture detector. If the confidence of the second human bounding box detected by the SSD object detector is greater than the threshold β SSD , then the second human body bounding box is retained, otherwise the second human body bounding box is considered invalid and discarded.

[0147] State 2: The human object is detected by the OpenPose pose detector but not by the SSD target detector. The average confidence value of all human key points detected by the OpenPose pose detector is taken as the confidence value of the human object in the OpenPose pose detector. If the value is greater than the threshold β OpenPose , then the first human body bounding box and the first human body key point are retained, otherwise the first human body bounding box is considered invalid and discarded.

[0148] In state three, the human object is detected by both the OpenPose pose detector and the SSD target detector. At this time, the second detection result of the SSD target detector is used to correct the first detection result of the OpenPose pose detector. First, the human bounding box is corrected, and the second human bounding box detected by the SSD target detector is believed to be accurate, that is, the second human bounding box is retained as the corrected third human bounding box.

[0149] Then correct the key points of the human body, keep all the key points within the third human body bounding box, and move all the key points outside the third human body bounding box. The moving method is as follows: Figure 4 As shown, the four vertices of the SSD human body frame are (x left ,y high ),(x right ,y high ),(x right ,y low ),(x left ,y low ), extend the four sides of the third person's bounding box, and divide the outside of the bounding box into 8 areas, namely area A, area B, area C, area D, area E, area F, area G and area H. For the key point (x i ,y i ), if it is in the A, C, E, G area, then its coordinates are directly corrected to (x left ,y high ),(x right ,y high ),(x right ,y low ),(x left ,y low ), if it is in the B, D, F, or H area, its coordinates are corrected to (x i ,y high ),(x right ,y i ),(x i ,y low ),(x left ,y i ).

[0150] Through the above correction steps, the third human body bounding box and the second human body key points obtained by the above detection model can be parameters that are closest to the actual human body situation, thereby effectively improving the accuracy of real-time detection of human body posture.

[0151] Step S105: determining whether the human body object in the image to be identified is in a falling posture according to the third detection result.

[0152] In a specific embodiment, after obtaining the third human body bounding box and the second human body key points, it can be determined whether the human body object is in a falling posture according to a pre-set judgment rule.

[0153] Specifically, the step of judging whether the human object in the image to be identified is in a falling posture according to the third detection result includes:

[0154] When the third detection result includes only a third human body bounding box, comparing the aspect ratio of the third human body bounding box with a preset aspect ratio threshold, and marking the human body corresponding to the third human body bounding box having an aspect ratio greater than the preset aspect ratio threshold as being in a falling state;

[0155] When the third detection result includes a third human body bounding box and human body key points, the third human body bounding box and the human body key points are input into an SVM binary classifier for cross-validation by the SVM binary classifier, and whether the human body corresponding to the third human body bounding box is in a falling state is determined according to the cross-validation result.

[0156] In a specific embodiment, as described in the above embodiment, the third detection result may be situation 1 in which the third detection result only includes a third human body bounding box, and situation 2 in which the third detection result includes a third human body bounding box and a second human body key point corresponding to the third human body bounding box.

[0157] If the third detection situation belongs to situation 1, the aspect ratio of the human body bounding box is calculated If the aspect ratio is greater than the threshold β wh , then mark the target as falling.

[0158] The third detection situation belongs to situation 2, so first normalize the key point coordinates Among them, x' i , y' i is the normalized coordinate of the ith key point, where i∈[0,13], (x t0 ,y t0 ) is the coordinate of the upper left corner of the target bounding box, (x t1 ,y t1 ) is the coordinate of the lower right corner.

[0159] Then construct 8 features: head and neck height h0 = |y'1-y'0|, body height Right leg height h2=|y' 10 -y'8|, left leg height h3=|y' 13 -y' 11 |, Neck angle Body angle Right leg angle Left leg angle

[0160] The 8 constructed features are input into the SVM binary classifier to determine whether the target is in a falling state.

[0161] The SVM binary classifier task is C-SVM classification, and the warp function is selected as the kernel function. The training method is:

[0162] Step 1: construct a human bounding box training dataset;

[0163] Step 2, using cross-validation to process the human body bounding box training data set, and selecting the optimal parameters C and g in the SVM binary classifier, where C is the penalty coefficient and g is the rate at which the kernel function reaches 0;

[0164] Step 3: cyclically train the SVM binary classifier model until the optimal

[0165] After the third detection result is input into the SVM binary classifier, a judgment result of whether the third detection result belongs to a falling state or a normal state is obtained, and based on the output of the SVM binary classifier, it is judged whether the human body corresponding to the third human body boundary box is in a falling state. Specifically, the output judgment parameter of the SVM binary classifier is not specifically limited and can be adaptively selected according to the actual application scenario.

[0166] The fall posture detection method provided in the embodiment of the present application can detect the same image to be identified based on different detection models, and obtain the human body boundary box and human body key points closest to the actual scene through preset correction actions, thereby effectively improving the accuracy of fall posture detection, and can effectively improve the recall rate of the bottom-up fall posture detection algorithm.

[0167] refer to Figure 5 , is a schematic diagram of a device module of a fall posture detection device 500 provided in an embodiment of the present application. The fall posture detection device 500 provided in an embodiment of the present application, such as Figure 5 As shown, the falling posture detection device 500 includes:

[0168] The acquisition module 501 is used to perform preset processing on the surveillance video to obtain an image to be identified;

[0169] A first detection module 502 is used to detect the image to be recognized according to a preset posture detection model to obtain a first detection result based on posture detection, wherein the first detection result includes a first human body bounding box and a first human body key point;

[0170] A second detection module 503 is used to detect the image to be identified according to a preset target detection model to obtain a second detection result based on target detection, wherein the second detection result includes a second human body bounding box;

[0171] A correction module 504 is used to correct the first detection result by using the second detection result to obtain a corrected third detection result, wherein the third detection result includes a third human body bounding box and / or a second human body key point;

[0172] The judgment module 505 is used to judge whether the human object in the image to be identified is in a falling posture according to the third detection result.

[0173] Figure 5 The device shown can correspondingly execute the contents of the above method embodiment. For the parts not described in detail in this embodiment, refer to the contents recorded in the above method embodiment and will not be repeated here.

[0174] The present application also provides an electronic device, the electronic device comprising:

[0175] at least one processor; and,

[0176] a memory communicatively connected to the at least one processor; wherein,

[0177] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the fall posture detection method in the aforementioned method embodiment.

[0178] The embodiment of the present application further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the fall posture detection method in the aforementioned method embodiment.

[0179] The electronic devices in the embodiments of the present application may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc.

[0180] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0181] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0182] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains at least two Internet Protocol addresses; sends a node evaluation request including the at least two Internet Protocol addresses to a node evaluation device, wherein the node evaluation device selects an Internet Protocol address from the at least two Internet Protocol addresses and returns it; receives the Internet Protocol address returned by the node evaluation device; wherein the obtained Internet Protocol address indicates an edge node in a content distribution network.

[0183] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device: receives a node evaluation request including at least two Internet Protocol addresses; selects an Internet Protocol address from the at least two Internet Protocol addresses; and returns the selected Internet Protocol address; wherein the received Internet Protocol address indicates an edge node in a content distribution network.

[0184] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0185] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0186] The units involved in the embodiments described in the present application may be implemented by software or hardware. The name of the unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0187] It should be understood that various parts of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof.

[0188] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for detecting falling posture, It is characterized in that The falling posture detection method comprises: Perform preset processing on the surveillance video to obtain the image to be identified; Detecting the image to be recognized according to a preset posture detection model to obtain a first detection result based on posture detection, wherein the first detection result includes a first human body boundary box and a first human body key point; Detect the image to be identified according to a preset target detection model to obtain a second detection result based on target detection, wherein the second detection result includes a second human body bounding box; Correcting the first detection result by using the second detection result to obtain a corrected third detection result, where the third detection result includes a third human body bounding box and / or a second human body key point; According to the third detection result, it is determined whether the human body object in the image to be identified is in a falling posture, wherein The step of detecting the image to be recognized according to a preset posture detection model to obtain a first detection result based on posture detection includes: Extracting features from the image to be identified based on a VGG19 convolutional network to obtain a feature image group having a preset number of feature maps; Extracting the confidence image group and the relevance image group from the feature image group based on a convolutional neural network respectively; Generating predicted key point positions according to the confidence image group; Connecting all predicted key point positions according to the association image group and the Hungarian algorithm to obtain a plurality of basic human frames corresponding to the image to be identified; Generate a minimum tangent bounding box according to each basic human body frame to obtain multiple groups of first human body frames corresponding to the image to be identified and human body key points corresponding to each first human body bounding box; wherein The step of detecting the image to be identified according to a preset target detection model to obtain a second detection result based on target detection includes: Performing human body detection on the image to be identified according to a preset convolutional neural network to obtain an initial predicted human body bounding box; The initial predicted human body bounding box is screened based on an NMS non-maximum suppression algorithm to obtain the second human body bounding box.

2. The falling posture detection method according to claim 1, It is characterized in that Before the step of correcting the first detection result by using the second detection result to obtain a corrected third detection result, the method further includes: Calculating a matching score between each first human body bounding box in the first detection result and each second human body bounding box in the second detection result based on a preset weight allocation rule and a similarity calculation rule; Combining the first human body bounding box and the second human body bounding box with a matching score greater than a preset score threshold into a to-be-selected matching pair; The corresponding first human body bounding box and the human body key points corresponding to the first human body bounding box are corrected according to the second human body bounding box in each candidate matching pair.

3. The falling posture detection method according to claim 2, It is characterized in that The step of correcting the corresponding first human body bounding box and the human body key points corresponding to the first human body bounding box according to the second human body bounding box in each to-be-selected matching pair comprises: In a candidate matching pair, selecting a human body bounding box with the highest confidence in the candidate matching pair from among the first human body bounding box and the second human body bounding box as a third human body bounding box, wherein the confidences of the first human body bounding box and the second human body bounding box are both greater than a preset confidence threshold; All human key points within the third human body bounding box are retained, and all human key points outside the third human body bounding box are moved according to a preset rule, wherein the preset rule is to move all human key points outside the third human body bounding box to the third human body bounding box and an extension line of the bounding box.

4. The falling posture detection method according to claim 1, It is characterized in that The step of judging whether the human body object in the image to be identified is in a falling posture according to the third detection result comprises: When the third detection result includes only a third human body bounding box, comparing the aspect ratio of the third human body bounding box with a preset aspect ratio threshold, and marking the human body corresponding to the third human body bounding box having an aspect ratio greater than the preset aspect ratio threshold as being in a falling state; When the third detection result includes a third human body bounding box and human body key points, the third human body bounding box and the human body key points are input into an SVM binary classifier for cross-validation by the SVM binary classifier, and whether the human body corresponding to the third human body bounding box is in a falling state is determined according to the cross-validation result.

5. The falling posture detection method according to claim 1, It is characterized in that The steps of performing preset processing on the surveillance video to obtain the image to be identified include: Performing frame cutting processing on the surveillance video to obtain initial images of continuous frames; Each frame of the initial image is standardized in turn to obtain consecutive frames of images to be recognized with the same resolution and size.

6. A falling posture detection device, It is characterized in that The falling posture detection device comprises: An acquisition module is used to perform preset processing on the surveillance video to obtain an image to be identified; A first detection module, configured to detect the image to be identified according to a preset posture detection model to obtain a first detection result based on posture detection, wherein the first detection result includes a first human body boundary box and a first human body key point; A second detection module, configured to detect the image to be identified according to a preset target detection model to obtain a second detection result based on target detection, wherein the second detection result includes a second human body bounding box; a correction module, configured to correct the first detection result by using the second detection result to obtain a corrected third detection result, wherein the third detection result includes a third human body bounding box and / or a second human body key point; A judgment module is used to judge whether the human object in the image to be identified is in a falling posture according to the third detection result; wherein The step of detecting the image to be recognized according to a preset posture detection model to obtain a first detection result based on posture detection includes: Extracting features from the image to be identified based on a VGG19 convolutional network to obtain a feature image group having a preset number of feature maps; Extracting the confidence image group and the relevance image group from the feature image group based on a convolutional neural network respectively; Generating predicted key point positions according to the confidence image group; Connecting all predicted key point positions according to the association image group and the Hungarian algorithm to obtain a plurality of basic human frames corresponding to the image to be identified; Generate a minimum tangent bounding box according to each basic human body frame to obtain multiple groups of first human body frames corresponding to the image to be identified and human body key points corresponding to each first human body bounding box; wherein The step of detecting the image to be identified according to a preset target detection model to obtain a second detection result based on target detection includes: Performing human body detection on the image to be identified according to a preset convolutional neural network to obtain an initial predicted human body bounding box; The initial predicted human body bounding box is screened based on an NMS non-maximum suppression algorithm to obtain the second human body bounding box.

7. An electronic device, It is characterized in that The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the fall posture detection method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the fall posture detection method according to any one of claims 1 to 5.