Human abnormal posture detection method

Abstract

The present invention discloses a method for detecting abnormal human postures, comprising the following steps: S11, acquiring a target image; S12, detecting the regional position of the human body in the target image and the human body posture in the target image; S13, determining whether the human body posture is an abnormal posture; S15, if there is an abnormal posture, determining whether the abnormal posture is the abnormal posture that first appears in the same human body; if the abnormal posture is the abnormal posture that first appears in the same human body, proceed to step S16, otherwise transfer to step S17; S16, storing the abnormal posture category, the number of abnormal times, the regional position of the human body in the target image, and the storage time corresponding to the abnormal posture of the human body as abnormal target data; S17, finding the region with the largest intersection-over-union ratio corresponding to the abnormal posture and comparing their similarities; S18, determining whether the similarity exceeds a first preset threshold; S19, if it exceeds the first preset threshold, generating an abnormal posture warning.

Description

Technical Field

[0001] The present invention relates to a method for detecting abnormal postures of the human body. Background Art

[0002] Currently, in the field of abnormal posture detection such as fall detection, there are mainly three research directions: 1. Detection based on sensors of wearable devices; 2. Detection based on environmental information of the Internet of Things; 3. Detection based on intelligent monitoring technology. The method based on wearable devices may not be able to detect abnormal states due to device wear or damage or forgetting to wear. The method based on environmental sensors (such as pressure, sound, vibration, etc.) cannot be well applied at present due to deployment costs or detection accuracy issues. Intelligent monitoring technology captures images through cameras and then recognizes the images, which not only has strong real-time performance, high accuracy, but also relatively low costs. Therefore, it is a relatively good choice to achieve abnormal posture detection through intelligent monitoring technology.

[0003] However, most of the current abnormal posture detections based on intelligent monitoring technology have the following two problems: 1) Limited applicable scenarios, mainly for single-person indoor scenarios, and existing monitoring devices cannot be reused; 2) The algorithm accuracy and speed have not reached a balance. Some inventions are based on traditional algorithms and manually construct features. Although the operation speed is fast, they are not robust enough and are greatly affected by scene complexity, lighting, and camera installation positions. Some inventions are based on human key point detection, with high accuracy, but the computational complexity is too large, and the data annotation cost is too high, making them not very practical. Summary of the Invention

[0004] Aiming at the deficiencies of the above-mentioned prior art, the technical problem to be solved by the present invention is: to provide a method for detecting abnormal postures of the human body, to solve the defects such as limited applicable scenarios, low recognition rate, susceptibility to the environment, and large computational complexity, and to provide a lightweight abnormal posture detection and tracking method and system based on deep learning.

[0005] To solve the above technical problems, a technical solution adopted by the present invention is: to provide a method for detecting abnormal postures of the human body, including the following steps:

[0006] S11. Obtain a target image;

[0007] S12. Detect the regional position of the human body in the target image and the human body posture in the target image;

[0008] S13. Determine whether the human body posture is an abnormal posture;

[0009] S15. If there is an abnormal posture, determine whether the abnormal posture is the first abnormal posture of the same human body; if the abnormal posture is the first abnormal posture of the same human body, enter step S16, otherwise transfer to step S17;

[0010] S16. Store the abnormal posture category, the number of abnormal occurrences, the regional position of the human body in the target image, and the storage time corresponding to the abnormal posture of the human body as abnormal target data;

[0011] S17. Find the region with the largest intersection over union with the regional position corresponding to the abnormal posture from the abnormal target data, and compare their similarities;

[0012] S18. Determine whether the similarity exceeds a first preset threshold;

[0013] S19. If it exceeds the first preset threshold, generate an abnormal posture warning.

[0014] Further, the step S10 includes the following sub-steps:

[0015] S111. Obtain the video streams from several camera devices in the target area;

[0016] S112. Perform frame extraction on the image video stream to obtain a single-frame target image.

[0017] Further, in the step S12, the regional position of the human body in the target image and the human body posture in the target image are detected through an abnormal posture detection network. The abnormal posture detection network includes a feature extraction network and a detection head network. The feature extraction network includes several layers of convolutional layers; the detection head network includes a locator and a classifier; the step S12 includes the following sub-steps:

[0018] S121. Input the target image as the detection image of the feature extraction network, and extract the features of the detection image;

[0019] S122. Select the last N feature response maps in the feature extraction network;

[0020] S123. Obtain the features of the anchor box regions corresponding to each feature response map;

[0021] S124. The locator generates a detection box offset according to the input features, and corrects the coordinates of the detection box offset according to the preset anchor box position and size, so as to obtain the center point coordinates, length and width. The classifier generates a human body posture label according to the input features; wherein, the human body posture label includes standing, squatting, lying down, and truncated;

[0022] S125. Eliminate the detection boxes with the label of background, so as to obtain the human body region detection box; wherein, non-maximum suppression is used to eliminate duplicate detection boxes;

[0023] S126. Output the human body region detection frame and the human body posture label;

[0024] In the S13 step, it is determined whether there is an abnormal posture of the human body in the target image through the output human body region detection frame and human body posture label.

[0025] Further, after the S13 step, it further includes:

[0026] S14. Send the human body region detection frame into the person re-identification module to extract Reid features. The person re-identification module includes a global feature extraction module and a local feature extraction module. This step includes the following sub-steps;

[0027] S141. Calculate the aspect ratio of the human body region detection frame;

[0028] S142. Determine whether the calculated aspect ratio is within a predetermined threshold range; if it is within the predetermined threshold range, go to the S143 step, if it is not within the predetermined threshold range, transfer to the S144 step;

[0029] S143. Then rotate the human body region detection frame clockwise by 90 degrees;

[0030] S144. The global feature extraction module performs global feature extraction on the human body region detection frame to obtain global features, and the local feature extraction module performs local feature extraction on the human body region detection frame to obtain local features;

[0031] S445. Concatenate the global feature and the local feature vector to obtain the Reid feature corresponding to the current abnormal posture;

[0032] In the S16 step, store the Reid feature corresponding to the abnormal posture of the human body, the abnormal posture category, the number of abnormal times, the regional position of the human body in the target image, and the storage time as abnormal target data;

[0033] In the S17 step, find the region with the largest intersection over union ratio with the regional position corresponding to the abnormal posture from the abnormal target data, and compare their Reid feature similarities.

[0034] Further, in the S144 step, it includes the following sub-steps:

[0035] S1441. Process the human body region detection frame to obtain a corresponding feature map;

[0036] S1442. Horizontally divide the feature map into multiple blocks to obtain multiple feature vectors as f = [f 1 , f 2 ,..., f n ​T ;

[0037] S1443. Perform a generalized average pooling operation on the multiple feature vectors to obtain the feature vector after the generalized pooling operation.

[0038] S1445. Vertically concatenate the multiple feature vectors into a single feature vector as the local feature f_local.

[0039] Further, in the step S1442, the generalized average pooling operation is obtained through the following formula:

[0040]

[0041] where p i is a learnable parameter included in the generalized average pooling, the initial value of pi is set to 1.0, and x represents the input feature, that is, the feature vector f.

[0042] Further, in the step S144, the global feature extraction includes the following sub-steps:

[0043] S1440'. Process the human body region detection frame to obtain the corresponding feature map;

[0044] S1441'. Perform a generalized average pooling operation on the feature map to obtain the global feature f_global after the generalized pooling operation.

[0045] Further, in the step S18, if the first preset threshold is not exceeded, then transfer to the step S16.

[0046] Further, after the step S18, it also includes:

[0047] S18a. If the first preset threshold is exceeded, determine whether the number of occurrences of the abnormal posture exceeds the second preset threshold; if it exceeds, enter the step S19, otherwise transfer to the step S11.

[0048] S19. If the second preset threshold is exceeded, generate an abnormal posture warning.

[0049] Further, before the step S11, it also includes:

[0050] S10. Initialize the abnormal target list;

[0051] In step S16, store the abnormal posture category, the number of abnormal occurrences corresponding to the abnormal posture of the human body, and the regional position of the human body in the target image into the abnormal target list as abnormal target data, record the position where the abnormal posture is stored in the abnormal target list, the posture category, the number of abnormal occurrences, and the last update time, and then transfer to step S11;

[0052] In step S17, find the region with the largest intersection-over-union ratio with the regional position corresponding to the abnormal posture from the abnormal target data of each abnormal target in the abnormal target list, and compare their abnormal posture similarities.

[0053] Further, after step S13, it further includes:

[0054] S13a. If the human body posture is not in an abnormal posture, traverse the abnormal target list, and determine whether the time when the abnormal target data corresponding to the last update storage in the abnormal target list exceeds the third preset threshold;

[0055] S13b. If it exceeds the third preset threshold, remove the corresponding abnormal target data and return to step S11.

[0056] Further, in step S15, determine whether the abnormal posture is the abnormal posture that first appears in the same human body by judging whether the abnormal target list is empty.

[0057] The human body abnormal posture detection method of the present invention has the following beneficial effects: 1) Aiming at the disadvantages of large calculation amount, high cost, high requirements for network bandwidth and quality in the prior art, the present invention designs a lightweight algorithm model and deploys the model to edge devices, reducing network latency and the bandwidth pressure on the server, which is especially suitable for scenarios with fewer computing nodes. 2) The present invention expands the applicable scenarios of the existing algorithms. Currently, most algorithms are limited to single-person indoor scenarios, and the present invention can utilize the cameras deployed in public areas to analyze multiple people simultaneously. 3) The present invention introduces the pedestrian re-identification technology to continuously track the target state and reduce the false alarm rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is the overall flowchart of the human body abnormal posture detection method of this application.

[0059] Figure 1a is the flowchart of the first embodiment of the human body abnormal posture detection method of this application.

[0060] Figure 2 is the flowchart of the second embodiment of the human body abnormal posture detection method of this application.

[0061] Figure 3 is the flowchart of the third embodiment of the human body abnormal posture detection method of this application.

[0062] Figure 4 It is a flowchart of the fourth embodiment of the human abnormal posture detection method of the present application.

[0063] Figure 5 It is a schematic diagram of the abnormal posture detection network.

[0064] Figure 6 It is a schematic diagram of the internal structure of the RFB receptive field module.

[0065] Figure 7 It is a schematic diagram of four postures.

[0066] Figure 8 It is a sub-flowchart for detecting the human body area and human body posture in the target image.

[0067] Figure 9 It is a schematic diagram of the detection frame of the human body area in the normal posture.

[0068] Figure 10 It is a schematic diagram of the detection frame of the human body area in the abnormal posture.

[0069] Figure 11 It is a schematic diagram of the person re-identification module.

[0070] Figure 12 It is the effect diagram of feature extraction and retrieval through the person re-identification module.

[0071] Figure 13 It is a sub-flowchart of Reid feature extraction. Specific implementation manners

[0072] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0073] Please refer to Figure 1 , Figure 1 It is a flowchart of the first embodiment of the human abnormal posture detection method of the present application. The human abnormal posture detection method of the present application includes the following steps:

[0074] S11. Obtain a target image;

[0075] In this step, the target image is obtained through the following steps:

[0076] S111. Obtain the video streams of several camera devices from the target area;

[0077] S112. Extract frames from the image video stream to obtain a single-frame target image.

[0078] The target area can be a certain shopping mall, a certain community, a certain specific place, or divided according to the requirements in the actual implementation.

[0079] S12. Detect the regional position and human posture of the human body in the target image.

[0080] The human posture includes, but is not limited to, standing, squatting, lying down, truncated, walking, falling, smoking, running, etc. In this step, the regional position and human posture in the target image can be detected through a detection network such as the SSD single-stage object detector (Single Shot MultiBox Detector).

[0081] S13. Determine whether the human posture is an abnormal posture.

[0082] In this embodiment, lying down and squatting are defined as abnormal postures.

[0083] S15. If there is an abnormal posture, determine whether this abnormal posture is the first abnormal posture of the same human body. If this abnormal posture is the first abnormal posture of the same human body, enter step S16, otherwise transfer to step S17.

[0084] S16. Store the abnormal posture category, the number of abnormal times corresponding to the abnormal posture of this human body, and the regional position of this human body in the target image as abnormal target data, and transfer to step S11. The posture category refers to the posture category label corresponding to this abnormal posture: lying down, squatting, and the posture category corresponding to this abnormal posture is shown in a label way. If this abnormal posture is the first occurrence of the same human body, then the number of abnormal times is recorded as 1.

[0085] S17. Find the area with the largest intersection over union with the regional position corresponding to this abnormal posture from the abnormal target data, and compare their abnormal posture similarities, so as to judge whether the people corresponding to the abnormal postures are the same person through this.

[0086] S18. Determine whether the similarity exceeds a first preset threshold. If it exceeds the first preset threshold, enter step S19. Otherwise, directly transfer to step S16 to update the abnormal target data.

[0087] S19. Generate an abnormal posture warning and transfer to step S16 to update the abnormal target data.

[0088] Specifically, taking an example, assume that a certain camera captures a target image of a person A lying on the ground. From the obtained target image, detect the regional position (x, y, width, height) of person A in the target image and the posture of person A lying down, that is, the x and y coordinates of person A in the target image, the width and height of the target image where person A is located. After determining that this lying-down posture belongs to an abnormal posture, if this is the first time person A shows this abnormal lying-down posture, store the abnormal posture of person A, its category, the number of abnormal occurrences (which is 1), and the corresponding regional position (x, y, width, height) of person A, so as to obtain the abnormal target data of person A. If person A is not showing this abnormal lying-down posture for the first time (among the several given images, the currently detected image is not the image when person A first shows this abnormal lying-down posture), then find the historical region with the largest intersection-over-union ratio with the region (x, y, width, height) where person A is located corresponding to the current abnormal posture from the stored abnormal target data, thereby judging their similarity. In this way, it can be determined whether person A has been continuously in such an abnormal posture within a continuous period, improving the accuracy of abnormal posture judgment, and further determining whether person A is a person who has truly fallen and needs help. If the determined similarity is within the preset range, it is considered that person A has been continuously in such an abnormal posture, which belongs to an abnormal posture, and further generate a warning for prompt.

[0089] In this embodiment, existing monitoring cameras can be used to obtain target images, and images can be obtained through multiple cameras in the target area and images obtained from multiple angles. The images obtained by multiple cameras in the target area are processed region by region. Taking the human body area, abnormal posture features, and the number of abnormal occurrences as the key judgment factors, only when the same human body shows abnormal postures multiple times and within a set range of regions is it regarded as an abnormal state, improving the judgment accuracy and greatly reducing the probability of misjudgment.

[0090] Please refer to Figure 2 , Figure 2 is the flowchart of the second embodiment of the human abnormal posture detection method of this application. The human abnormal posture detection method of this application includes the following steps:

[0091] S21. Obtain a target image;

[0092] S22. Detect the regional position of the human body in the target image and the human body posture in the target image;

[0093] S23. Determine whether the human body posture is in an abnormal posture;

[0094] S25. If there is an abnormal posture, determine whether this abnormal posture is the first abnormal posture of the same human body. If this abnormal posture is the first abnormal posture of the same human body, proceed to step S26; otherwise, transfer to step S27.

[0095] S26. Store the abnormal posture category, the number of abnormal occurrences, and the regional position of the human body in the target image corresponding to the abnormal posture of this human body as abnormal target data, and transfer to step S21.

[0096] S27. Find the region with the largest intersection-over-union ratio with the regional position corresponding to this abnormal posture from the abnormal target data, and compare their similarities.

[0097] S28. Determine whether the similarity exceeds a first preset threshold. If it does not exceed the first preset threshold, directly transfer to step S26, and regard the current abnormal target data as a new abnormal target data (that is, start a new abnormal target data packet).

[0098] S28a. If the similarity exceeds the first preset threshold, transfer to step S26 to update the abnormal target data (that is, update the abnormal target data in the original abnormal target data packet), and determine whether the number of abnormal occurrences of this abnormal posture exceeds a second preset threshold. If it exceeds, proceed to step S29; otherwise, transfer to step S21.

[0099] In this step, when the similarity exceeds the set first preset threshold, it indicates that there is abnormal target data in the stored historical abnormal target data that is associated with the abnormal posture of the current same human body. That is to say, this same human body maintains this abnormal posture. Thus, it is determined how many times the abnormal posture of the current same human body appears and whether it exceeds the second preset threshold.

[0100] S29. If it exceeds the second preset threshold, generate an abnormal posture warning.

[0101] In this embodiment, when the current abnormal posture is not the first occurrence, it is determined whether the historical abnormal target data has a similar abnormal posture of the same human body. If there is an abnormal posture of the same human body in the corresponding region, a second preset threshold determination is added, which further increases the accuracy of the abnormal posture determination, rather than simply generating a warning directly based on the fact that the current abnormal posture is not the first occurrence.

[0102] Please refer to Figure 3 , Figure 3 which is the flowchart of the third embodiment of the human body abnormal posture detection method of this application. The human body abnormal posture detection method of this application includes:

[0103] S30. Initialize the abnormal target list.

[0104] S31. Obtain a target image;

[0105] S32. Detect the regional position of the human body in the target image and the human body posture in the target image;

[0106] S33. Determine whether the human body posture is in an abnormal posture;

[0107] S33a. If the human body posture is not in an abnormal posture, traverse the abnormal target list;

[0108] S33a’. And determine whether the time when the abnormal target data corresponding to the last update and storage in the abnormal target list exceeds a third preset threshold; if it does not exceed the third preset threshold, then transfer to step S31;

[0109] S33b. If it exceeds the third preset threshold, remove the corresponding abnormal target data and return to step S31.

[0110] S35. If there is an abnormal posture, determine whether this abnormal posture is the first abnormal posture of the same human body by judging whether the abnormal target list is empty; if the abnormal target list is empty, enter step S26, otherwise transfer to step S27;

[0111] S36. Store the abnormal posture category, the number of abnormal times corresponding to the abnormal posture of this human body, and the regional position of the human body in the target image into the abnormal target list as abnormal target data, record the position, posture category, number of abnormal times and the last update time when the abnormal posture is stored in the abnormal target list, and transfer to step S31;

[0112] S37. Find the region with the largest intersection-over-union ratio with the regional position corresponding to this abnormal posture from the abnormal target data in the abnormal target list, and compare their similarities;

[0113] S38. Determine whether the similarity exceeds a first preset threshold; if it does not exceed the first preset threshold, then directly transfer to step S36, and regard the current abnormal target data as a new abnormal target data (that is, start a new abnormal target data packet);

[0114] S38a. If the similarity exceeds the first preset threshold, then transfer to step S36 to update the abnormal target data (that is, update the abnormal target data in the original abnormal target data packet), and determine whether the number of abnormal times when this abnormal posture appears exceeds a second preset threshold; if it exceeds, enter step S39, otherwise transfer to step S31;

[0115] In this step, when the similarity exceeds the set first preset threshold, it indicates that there is abnormal target data in the stored historical abnormal target data that is associated with the abnormal posture of the same human body currently, that is, the same human body maintains this abnormal posture. Thus, it is determined how many times the abnormal posture of the same human body currently appears and whether it exceeds the second preset threshold.

[0116] S39. If it exceeds the second preset threshold, generate an abnormal posture warning.

[0117] Specifically, in combination with an example, this human body abnormal posture detection method first initializes the abnormal target list and determines that the abnormal target list is empty. Then, the obtained video is frame-extracted to obtain a single image (i.e., the target image), and it is detected whether there is an abnormal posture of the human body in the target image through methods such as a target detection network.

[0118] Assume that a certain person A in the target image A is in a falling state, then it is regarded that person A has an abnormal posture. Determine whether the list of abnormal targets is empty. If it is empty, it means that this abnormal posture of person A appears for the first time in this target image. Store the abnormal posture of person A, the number of abnormal occurrences (the first occurrence, the number of abnormal occurrences is 1), and the regional position of person A in the target image (that is, the X and Y positions of person A's human body image in the target image, and position features such as the width and height occupied by person A's human body image) into the list of abnormal targets as abnormal target data. If it is not empty, it means that there is already historical abnormal target data stored in the list of abnormal targets. Then, search in the list of abnormal targets to check if there is abnormal target data corresponding to the abnormal posture of this person A. By matching the current abnormal target data of person A with the human body area and abnormal posture in the historical abnormal target data stored in the list of abnormal targets, compare their similarity by comparing the intersection-over-union ratio of the two. Through a first preset threshold regarding similarity, determine whether the similarity between the two exceeds the first preset threshold. If it exceeds the first preset threshold (indicating that the two have a relatively high similarity, regarded as person A still continuously in this abnormal posture and the current regional position still corresponding to the regional position of person A's historical abnormal target data, indicating that person A has not moved from the regional position of this abnormal posture, that is, person A has not quickly changed the posture and moved to other positions, further indicating that person A has been continuously in this abnormal posture and is more likely to be an object that needs to be focused on or helped), determine whether the current number of abnormal occurrences exceeds a second preset threshold. If it exceeds the set number of times, generate a warning message. The warning message can be broadcast by voice through surrounding surveillance cameras or other devices, or other means can be used for warning prompts to indicate that person A in the corresponding area is in an abnormal state to provide help for person A immediately. When the number of abnormal occurrences does not exceed the second preset threshold, it means that person A may have lasted for a period of time but has not been determined to be in a truly abnormal state, then continue to extract video frames to analyze the images.

[0119] Please refer to Figure 4 , Figure 4 is the flowchart of the fourth embodiment of the human abnormal posture detection method of the present application. The human abnormal posture detection method of the present application includes:

[0120] S40. Initialize the list of abnormal targets;

[0121] S41. Obtain the target image;

[0122] S42. Detect the regional position and human body posture of the human body in the target image in the target image;

[0123] In this step, the regional position and human body posture of the human body in the target image are detected through an abnormal posture detection network. Please refer toFigure 5 , the abnormal posture detection network includes a feature extraction network and a detection head network. The feature extraction network includes a backbone network composed of several layers of convolutional layers; the detection head network includes a locator and a classifier 100.

[0124] I. This feature extraction network is based on the SSD (Single Shot MultiBox Detector) single-stage object detector network, and the following improvements are made to the feature extraction network:

[0125] Based on the VGG16 deep convolutional neural network, the backbone network reduces the number of network layers and channels, and the maximum number of channels does not exceed 256, greatly reducing the number of parameters. At the same time, a batch normalization layer is added after each convolutional layer (each convolutional layer has a batch normalization layer) to accelerate training. A Receptive Field Block is added to the backbone network to better learn multi-scale features. The internal structure diagram of the RFB receptive field module (please refer to Figure 6 the conv4_3 in Figure 6 . The finally designed network has 12.23M parameters. Compared with the original SSD detector, the accuracy only drops by 2%, and the speed is increased by 4 times. With the acceleration framework provided by NVIDIA, a speed of 25fps (frame per second) can be achieved on the Jeston Nano device.

[0126] II. In this specific implementation, please refer to Figure 7 , four postures are designed in the output layer of the abnormal posture detection network: stand, bend, fall, and occluded (the upper body or lower body is blocked). Among them, when the posture is determined to be bend and fall, it is considered an abnormal posture and needs further analysis. The output layer of the feature extraction network uses a total of 6 feature maps to detect targets of different scales, that is, the feature response maps output by the last 6 convolutional layers in the feature extraction network. Each layer outputs the human body position coordinates (x, y, width, height) and the corresponding posture category, and finally the outputs of each layer are merged and non-maximum suppression is performed to obtain the final result. Among them, the number and size of the anchor boxes (default boxes) of each layer are obtained by k-means algorithm clustering.

[0127] The complete structure of the abnormal posture detection network is as follows in Table 1: conv: convolutional layer; stride: stride; Padding: padding; Batchnorm: batch normalization; Relu: rectified linear unit; Maxpool: max pooling layer; Dilation: dilation coefficient; RFB: receptive field module.

[0128]

[0129]

[0130]

[0131]

[0132] Table 1

[0133] Preferably, the main training process of the abnormal posture detection network is as follows:

[0134] D1. Collect monitoring pictures containing abnormal postures and normal postures to obtain a picture set, denoted as P, and the number of P is denoted as N; perform manual position annotation and category annotation on the pictures in the image set P, and the annotation result of each picture is denoted as, where denotes the upper left coordinate of the j-th target in the k-th image, denotes the width and height of the j-th target in the k-th image, denotes the category label of the j-th target in the k-th image, and T is the number of targets in the k-th image. Randomly divide the picture set P into a training set and a test set according to a predetermined ratio of r1:r2, denoted as P r1 and P r2 .

[0135] D2. Convert the training set P r1 and the test set P r2 into the standard coco dataset format, and change the picture size to 300x300 pixels.

[0136] D3. When the image size is determined to be 300x300 pixels, the abnormal posture detection network will generate 6 feature response maps (6 in this embodiment, but this quantity does not limit the scope of the present application), with sizes of 40x40, 20x20, 10x10, 5x5, 3x3, and 1x1 respectively. Traverse the 6 feature response maps respectively, and generate multiple sets of anchor boxes with different sizes and aspect ratios for each of the 6 feature response maps with each pixel on the feature response map as the center. Each anchor box is represented by the center point coordinates cx, cy and the length and width w, h. Among them:

[0137] The calculation formula for the center point coordinates is as follows:

[0138]

[0139] where |f k | is the size of the feature map, and k ∈ [1, 2, 3, 4, 5, 6].[[]]

[0140] The calculation formula for the length and width of the anchor box is as follows:

[0141] Based on the set size and aspect ratio, the length and width of the anchor box can be obtained

[0142]

[0143]

[0144]

[0145]

[0146] Among them, is the minimum size set for the k-th feature response map,

[0147] is the aspect ratio set for the k-th feature response map at scale Set aspect ratio,

[0148]

[0149]

[0150]

[0151]

[0152]

[0153]

[0154] is the maximum size set for the k-th feature response map,

[0155] is the aspect ratio set for the k-th feature response map at scale Set aspect ratio,

[0156]

[0157] A total of 9,698 anchor boxes are obtained for 6 feature response maps.

[0158] D4. According to the manually labeled box information in P r1 P r2 For each feature response map, the category is assigned to the anchor box and the coordinate offset is calculated, serving as the ground truth for the training of the network model. The network completes the training by continuously learning to make the output value approach the ground truth.

[0159] There are mainly two principles for matching anchor boxes and labeled boxes:

[0160] 1. For each labeled box in the image, find the anchor box with the largest intersection over union (IoU) with it, and match this anchor box with it. This can ensure that each labeled box is definitely matched with an anchor box.

[0161] 2. For the remaining unmatched anchor boxes, if the IoU of a certain labeled box is greater than a certain threshold (usually 0.5), then this anchor box is matched with this labeled box. The anchor box matched with a certain labeled box is called a positive sample. Conversely, if an anchor box is not matched with any labeled box, it is a negative sample.

[0162] Let the anchor box be d i , and the labeled box matched with it be g j ; when d i is a positive sample, the category of d i is the category of g j , and the corresponding coordinate offset is calculated as follows:

[0163]

[0164]

[0165] When d i is a negative sample, the category of d i is the background, and the coordinate offset is not calculated. Thus, the target values required for the training of the network model are calculated.

[0166] D5. Construct and initialize the abnormal pose network model, denoted as W ab , where W ab is composed of a backbone network, a locator, and a classifier. The backbone network is denoted as W feature , the locator is denoted as W loc , and the classifier is denoted as W cls .

[0167] D6. Network forward calculation. Each time, randomly select 64 samples from P r1 and send them into the network W ab for forward calculation to obtain the outputs of the W loc and W cls branches.

[0168] D7. Loss function calculation and network parameter update. The loss function L(x, c, l, g) is composed of the classification loss L conf and the localization loss L loc in two parts.

[0169]

[0170] The training ground truth g obtained in step D4 and the network prediction value l obtained in step D5 are fed into the loss function for loss calculation. Since the number of negative samples generated is much larger than that of positive samples, it will cause slow and unstable convergence of network training. Therefore, online hard example mining needs to be performed according to the classification loss of each negative sample prediction box in the network prediction value l, and finally the ratio of positive and negative samples is kept not exceeding 1:3. The selected positive and negative samples participate in the loss calculation of network backpropagation.

[0171] Among them, N is the number of positive samples, and α is 1. If N is 0, then the loss is 0.

[0172] L conf is the multi-class cross-entropy loss, and the calculation formula is as follows:

[0173]

[0174] L loc is the smooth L1 regression loss, and the calculation formula is as follows:

[0175]

[0176]

[0177] is the indicator function, which takes the value of 1 when the anchor box i matches the labeled box j with class k, and 0 otherwise.

[0178] Perform backpropagation on the loss function to obtain the updated values of network parameters, and use the updated values to update the network parameters. Repeat steps D6 and D7 until the number of iterations or the network loss drops to the preset value, then stop training.

[0179] Specifically, please refer to Figure 8 The S42 step includes the following sub-steps:

[0180] S421. Input the target image as the detection image of the feature extraction network and extract the features of the detection image; for each convolutional layer, pooling layer, and fully connected layer that the input target image passes through, corresponding features will be generated. Each time it passes through a convolutional layer, etc., it is a process of feature extraction. The features can be a multi-dimensional array or a one-dimensional array.

[0181] S422. Select the last 6 feature response maps in the feature extraction network; it can be understood that in other embodiments, 7, 5, 4, more or fewer feature response maps can be selected;

[0182] S423. Obtain the features of the anchor box regions corresponding to each feature response map;

[0183] S424. The locator generates a detection box offset according to the input features, and corrects the coordinates of the detection box offset based on the preset anchor box position and size, so as to obtain the center point coordinates, length and width, that is, the regional position of the human body in the target image mentioned above, namely the human body position coordinates (x, y, width, height). The classifier generates a human body pose label according to the input features. Among them, the human body pose label includes standing, squatting, lying down, and truncated. In this step, the features of the anchor box area are multi-dimensional features, and the dimensions are from loc_0 to loc_5 in the network structure table 1.

[0184] S425. Remove the detection boxes with the label of background, so as to obtain the human body area detection boxes. Among them, non-maximum suppression is used to eliminate duplicate detection boxes.

[0185] S426. Output the human body area detection boxes and the human body pose labels.

[0186] Please refer to Figure 9 and Figure 10 , Figure 9 The human body area detection box in Figure 10 represents the human body area detection box in the normal pose,

[0187] S43. Judge whether the human body pose is in an abnormal pose.

[0188] In this step, it is judged whether there is an abnormal pose of the human body in the target image through the output human body area detection box and the human body pose label.

[0189] S43a. If the human body pose is not in an abnormal pose, traverse the abnormal target list.

[0190] S43a'. And judge whether the time when the abnormal target data corresponding to the last update storage in the abnormal target list exceeds the third preset threshold.

[0191] S43b. If it exceeds the third preset threshold, remove the corresponding abnormal target data and return to step S41.

[0192] S44. Send the human body area detection box into the person re-identification module to extract Reid features. Please refer to Figure 11 , the person re-identification module includes a global feature extraction module and a local feature extraction module.

[0193] The pedestrian re-identification module is based on the OSNet pedestrian re-identification network, combines the human body position features to track people in abnormal postures, and improves the OSNet to obtain richer feature information. In the training stage, the global branch and the local branch are trained separately. In the testing stage, the feature vectors of f_global and f_local are taken respectively and concatenated to form a vector of (2048 + 512, 1, 1) dimensions as the reid feature of the input image for subsequent feature similarity calculation. The complete pedestrian re-identification network module is shown in Table 2 below: OS block: Omni-Scale module, full-scale network; GeM: Generalized Mean Pooling; Num_class: number of output classes; related to the number of pedestrians for training the pedestrian re-identification model, no specific value is set here; average pool: average pooling; fc: fully connected layer.

[0194]

[0195]

[0196] Table 2

[0197] The training process of pedestrian re-identification is as follows:

[0198] B1. Construct a training dataset. Collect pedestrian pictures containing M different IDs. Each pedestrian can correspond to multiple pictures to obtain a training picture set P, and the number of P is denoted as N, (N >= M). The annotation result corresponding to the training set is denoted as X = {(x i , y i ), i = 1, 2,..., N, y i ∈ [0, M - 1]}; Divide the M IDs into a training ID set P train and a test ID set P test in a ratio of 8:2 (of course, other preset ratios can also be used), and the pedestrian IDs in the training set and the test set do not repeat. Then all the pictures under the training ID set are the training set, and all the pictures under the test ID set are the test set. The size of the pictures in the set is transformed to 256x192 pixels (this pixel is only for illustration).

[0199] B2. Construct and initialize the pedestrian re-identification network model, denoted as W reid , W reid consists of a global branch and a local branch.

[0200] B3. Selection of training batch samples. Each time, randomly select pedestrians with 8 IDs, and randomly select 4 different pictures for each pedestrian. For each picture a among these 32 pictures, select one of the most difficult positive samples p and one of the most difficult negative samples n, and form a triplet with a. A total of 32 triplet sets are obtained as the training samples for the current batch.

[0201] B4. Forward calculation of the network. Feed the training samples selected in step 3) into the network W reid for forward calculation. Denote the local features and classification features obtained by the local branch as f_local and f_lcls respectively, and the global features and classification features obtained by the global branch as f_global and f_gcls respectively.

[0202] B5. Calculation of the loss function and update of network parameters. The total loss L of the network sum is the sum of the local branch loss L local and the global branch loss L global :

[0203] L sum = L local + L global

[0204] where two loss functions are defined for each of the local branch and the global branch, namely the classification loss and the triplet loss.

[0205] L local = L id_local + L tri_local

[0206] L global = L id_global + L tri_global

[0207] L id_local and L id_global The calculation formulas are as follows:

[0208] where N s is the number of samples in the current training batch.

[0209] L tri_local and L tri_global The calculation formulas are as follows:

[0210]

[0211] Perform backpropagation on the loss function L sum to obtain the updated values of the network parameters, and use the updated values to update the network parameters. Repeat the above steps until the number of iterations or the network loss drops to the preset value, then stop training.

[0212] Specifically, please refer to Figure 13 , this step includes the following sub-steps;

[0213] S441. Calculate the aspect ratio of the detected bounding box of the human body region;

[0214] S442. Determine whether the calculated aspect ratio is within a predetermined threshold range; if it is within the predetermined threshold range, proceed to step S443, and if it is not within the predetermined threshold range, transfer to step S444; in this step, the set threshold range is greater than 0.1 and less than 0.6;

[0215] S443. Then rotate the detected bounding box of the human body region clockwise by 90 degrees;

[0216] S444. The global feature extraction module performs global feature extraction on the detected bounding box of the human body region to obtain global features, and the local feature extraction module performs local feature extraction on the detected bounding box of the human body region to obtain local features; in this step, the local features are extracted through the following sub-steps:

[0217] S4441. Process the detected bounding box of the human body region to obtain a corresponding feature map;

[0218] S4442. Horizontally divide the feature map into multiple blocks to obtain multiple feature vectors f = [f 1 , f 2 ,..., f n T

[0219] In this specific embodiment, a local feature extraction branch is added after the conv4 module, and the feature map output by the conv4 module is horizontally divided into 4 blocks to obtain 4 feature vectors f = [f 1 , f 2 , f 3 , f 4 T .

[0220] S4443. Perform a generalized average pooling operation on the multiple feature vectors to obtain the feature vector after the generalized pooling operation The feature vector after the generalized average pooling operation is obtained through the following generalized average pooling formula:

[0221]

[0222] where p iSince the generalized average pooling contains learnable parameters, the initial value of pi is set to 1.0, x represents the input feature, that is, the feature vector f. Compared with the maximum pooling that takes the maximum value of the input feature, and the average pooling that takes the mean value of the input feature, the generalized average pooling contains a learnable parameter p i , first raise the input feature to the power of p i times, then take the mean value, and then perform the p i th root, which can obtain more compact features and improve the matching accuracy. The initial value of p i is set to 1.0. In this specific embodiment, n = 4.

[0223] S4445. Vertically splice the multiple feature vectors into a single feature vector as the local feature f_local. The features of 4 different scales are respectively fed into a unified aggregation gate for dynamic feature weighting, and finally the 4 weighted features are fused and the number of feature channels is increased to obtain a local feature vector consistent with the input size, that is, the local feature f_local.

[0224] The global feature is obtained through the following sub-steps:

[0225] S4441’. Process the human body region detection frame to obtain a corresponding feature map;

[0226] S4442’. Perform a generalized average pooling operation on the feature map to obtain the globally pooled global feature f_global;

[0227] In this specific embodiment, the average pooling in the gap module in the global branch is replaced with generalized average pooling, where the p in the generalized average pooling i The initial value is set to 6.5.

[0228] S445. Splice the global feature and the local feature vector to obtain the Reid feature corresponding to the current abnormal pose;

[0229] S45. Determine whether the abnormal pose is the abnormal pose of the same human body that appears for the first time by judging whether the abnormal target list is empty; if the abnormal target list is empty, enter step S46, otherwise transfer to step S47;

[0230] S46. Store the Reid feature, abnormal pose category, abnormal times corresponding to the abnormal pose of the human body, and the regional position of the human body in the target image as abnormal target data, record the position, pose category, abnormal times and the last update time of the abnormal pose stored in the abnormal target list, and transfer to step S41;

[0231] S47. Find the region with the largest intersection-over-union with the abnormal pose from the abnormal target data, and compare their Reid feature similarities;

[0232] In this step, the cosine distance of their reid features is compared as the similarity between features. Please refer to Figure 12 , Figure 12 is the effect diagram after feature extraction and feature retrieval by this person re-identification module. After the person re-identification module obtains the one-dimensional features of each picture, through the cosine similarity calculation formula, the similarity between pictures (between 0 and 1) can be obtained. Sorting according to the size of the similarity completes the retrieval process.

[0233] S48. Determine whether the similarity exceeds the first preset threshold; if it does not exceed the first preset threshold, directly transfer to step S46, and use the current abnormal target data as a new abnormal target data (that is, start a new abnormal target data packet);

[0234] S48a. If the similarity exceeds the first preset threshold, transfer to step S46 to update the abnormal target data (that is, update the abnormal target data in the original abnormal target data packet), and determine whether the number of occurrences of the abnormal pose exceeds the second preset threshold; if it exceeds, enter step S49, otherwise transfer to step S41;

[0235] S49. If it exceeds the second preset threshold, generate an abnormal pose warning.

[0236] The above are only the implementation manners of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied to other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A method for detecting abnormal postures of the human body, comprising the following steps: S11. Obtain a target image; S12. Detect the regional position and human posture of the human body in the target image in the target image; wherein, the abnormal posture detection network is used to detect the regional position and human posture of the human body in the target image in the target image. The abnormal posture detection network includes a feature extraction network and a detection head network. The feature extraction network includes several layers of convolutional layers; the detection head network includes a locator and a classifier; this step includes the following sub-steps: S121. Input the target image as the detection image of the feature extraction network and extract the features of the detection image; S122. Select the last N feature response maps in the feature extraction network; S123. Obtain the features of the anchor box regions corresponding to each feature response map; S124. The locator generates a detection box offset according to the input features, and corrects the coordinates of the detection box offset according to the preset anchor box position and size, so as to obtain the center point coordinates, length and width. The classifier generates a human posture label according to the input features; wherein, the human posture label includes standing, squatting, lying down, and truncated; S125. Eliminate the detection boxes with the label of background, so as to obtain the human body region detection boxes; wherein, non-maximum suppression is used to eliminate duplicate detection boxes; S126. Output the human body region detection boxes and human posture labels; S13. Judge whether the human posture is an abnormal posture; S15. If there is an abnormal posture, judge whether the abnormal posture is the first abnormal posture of the same human body; if the abnormal posture is the first abnormal posture of the same human body, enter step S16, otherwise transfer to step S17; S16. Store the abnormal posture category, the number of abnormal times, the regional position of the human body in the target image, and the storage time corresponding to the abnormal posture of the human body as abnormal target data; S17. Find the region with the largest intersection over union with the regional position corresponding to the abnormal posture from the abnormal target data and compare their similarities; S18. Judge whether the similarity exceeds a first preset threshold; S19. If it exceeds the first preset threshold, generate an abnormal posture warning.

2. The method for detecting abnormal postures of the human body according to claim 1, wherein, the S11 step includes the following sub-steps: S111. Obtain the video streams of several camera devices from the target area; S112. Perform frame extraction on the video stream to obtain a single-frame target image.

3. The method for detecting abnormal postures of the human body according to claim 1, wherein, in the S13 step, it is judged whether there is an abnormal posture of the human body in the target image through the output human body region detection boxes and human posture labels.

4. The method for detecting abnormal postures of the human body according to claim 3, wherein, after the S13 step, it further includes: S14. Send the human body region detection box to the person re-identification module to extract Reid features. The person re-identification module includes a global feature extraction module and a local feature extraction module. This step includes the following sub-steps; S141. Calculate the aspect ratio of the human body region detection box; S142. Determine whether the calculated aspect ratio is within a predetermined threshold range; if it is within the predetermined threshold range, go to step S143, and if it is not within the predetermined threshold range, transfer to step S144; S143. Then rotate the human body region detection box clockwise by 90 degrees; S144. The global feature extraction module performs global feature extraction on the human body region detection box to obtain global features, and the local feature extraction module performs local feature extraction on the human body region detection box to obtain local features; S145. Concatenate the global feature and the local feature vector to obtain the Reid feature corresponding to the current abnormal posture; In step S16, store the Reid feature corresponding to the abnormal posture of the human body, the abnormal posture category, the number of abnormal times, the region position of the human body in the target image, and the storage time as abnormal target data; In step S17, find the region with the largest intersection over union ratio with the region position corresponding to the abnormal posture from the abnormal target data, and compare their Reid feature similarities.

5. The human body abnormal posture detection method according to claim 4, characterized in that, In step S144, it includes the following sub-steps: S1441. Process the human body region detection box to obtain a corresponding feature map; S1442. Horizontally divide the feature map into multiple blocks to obtain multiple feature vectors \(f = [f 1 , f 2 , \cdots, f n T ;​ S1443. Perform a generalized average pooling operation on the multiple feature vectors to obtain multiple feature vectors after the generalized average pooling operation. S1444. Vertically concatenate the multiple feature vectors into a single feature vector as the local feature f_local.

6. The human body abnormal posture detection method according to claim 5, characterized in that, In step S144, the global feature extraction includes the following sub-steps: S1440’. Process the human body region detection box to obtain a corresponding feature map; S1441’. Perform a generalized average pooling operation on the feature map to obtain the global feature f_global after the generalized average pooling operation.

7. The human body abnormal posture detection method according to any one of claims 1 to 6, characterized in that, In step S18, if the first preset threshold is not exceeded, transfer to step S16.

8. The human body abnormal posture detection method according to claim 7, characterized in that, Before step S11, it further includes: S10. Initialize the abnormal target list; In step S16, store the abnormal posture category, the number of abnormal times corresponding to the abnormal posture of the human body, and the region position of the human body in the target image to the abnormal target list as abnormal target data, record the position, posture category, number of abnormal times, and the last update time when the abnormal posture is stored in the abnormal target list, and transfer to step S11; In step S17, find the region with the largest intersection over union ratio with the region position corresponding to the abnormal posture from each abnormal target data in the abnormal target list, and compare their abnormal posture similarities.

9. The human abnormal posture detection method according to claim 8, characterized in that, after step S13, it further includes: S13a. If the human posture is not in an abnormal posture, traverse the abnormal target list and determine whether the time when the abnormal target data corresponding to the last update storage in the abnormal target list exceeds a third preset threshold; S13b. If it exceeds the third preset threshold, remove the corresponding abnormal target data and return to step S11.

10. The human abnormal posture detection method according to claim 8, characterized in that, in step S15, it is determined whether the abnormal posture is the abnormal posture that first appears in the same human body by determining whether the abnormal target list is empty or similar abnormal target feature data cannot be found in the abnormal target list.

Images