Behavior state detection method and device for human body target
By obtaining the coordinates of the key points of the human target neck and crotch in the image frame in public places, and combining deep learning algorithms to judge the behavior status of pedestrians, the problems of high false alarm rate and strong subjectivity in the current technology are solved. Achieve more accurate behavioral state recognition.
Patent Information
- Application Number
- CN202310592268.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-23
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the existing technology, in the detection of pedestrian falls in public places, the detection false alarm rate based on hardware devices is high, and traditional machine learning relies on manual extraction of features with strong subjectivity, making it difficult to accurately identify behavioral states in complex scenarios.
By obtaining the coordinates of the key points of the neck and crotch of the human target in the image frame, calculating the detection angle and detection distance, combined with deep learning algorithms such as CSPDarknet53 and OpenPose, we can judge the behavior status of pedestrians, including falling, lying down, walking or squatting. .
It improves the accuracy and robustness of pedestrian behavior status detection, reduces the false alarm rate, and is suitable for multi-person detection in complex public scenarios.
Smart Images

Figure CN119028009A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of target detection, and in particular to a method and device for detecting the behavior state of a human target. Background Art
[0002] There are many people in public places, and crowding, trampling and other situations occasionally occur. In these situations, falling is the earliest behavior. Therefore, in scenes with complex personnel, it is necessary to accurately and timely detect whether pedestrians have fallen, and when a fall event is detected, quickly report it to facilitate the management personnel of public places to handle it, which can avoid danger to a certain extent.
[0003] In the related technology, detection of human falls includes detection based on hardware devices and detection based on traditional machine learning. Among them, detection based on hardware devices includes detection based on wearable devices and detection based on environmental devices. Wearable devices are suitable for single-person detection and are not suitable for multi-person detection in public places. Detection based on environmental devices determines whether a fall has occurred by analyzing the current or audio information collected from radar or sensors. However, the movement of any object can generate vibration, the false alarm rate is extremely high, and the detection accuracy is low. In addition, traditional detection based on machine learning relies too much on the quality of manually extracted features, is highly subjective, and affects the accuracy of behavior recognition and classification. Summary of the invention
[0004] In an exemplary embodiment of the present application, a method and device for detecting the behavior state of a human target are provided to improve the accuracy of detecting the behavior state of pedestrian targets in public places.
[0005] According to a first aspect of an exemplary embodiment, a method for detecting a behavior state of a human target is provided, the method comprising at least the following steps:
[0006] Acquire an image frame from an image detection device;
[0007] If the image frame includes a human target, obtaining coordinates of a plurality of key points of the human target included in the image frame;
[0008] Among the coordinates of the multiple key points, obtain a first coordinate of a neck key point of a human target in an image frame previous to the image frame, as well as a second coordinate of the neck key point and a third coordinate of a hip key point of the human target in the image frame;
[0009] Determine the detection distance and detection angle of the image frame according to the first coordinate, the second coordinate and the third coordinate; wherein the detection angle is used to represent the movement direction of the human target in the image frame, and the detection distance is used to represent the movement distance of the human target in the image frame;
[0010] According to the detection angle and detection distance, the behavior state of the human target in the image frame is determined.
[0011] According to a second aspect of the exemplary implementation, there is provided a behavior state detection device, the device comprising a data transmission unit and a processor;
[0012] The data transfer unit is configured to perform:
[0013] Acquire an image frame from an image detection device;
[0014] The processor is configured to execute:
[0015] If the image frame includes a human target, then obtaining coordinates of multiple key points of the human target included in the image frame; wherein the multiple key points include a neck key point and a hip key point;
[0016] Among the coordinates of the multiple key points, obtain a first coordinate of a neck key point of a human target in an image frame previous to the image frame, as well as a second coordinate of the neck key point and a third coordinate of a hip key point of the human target in the image frame;
[0017] Determine the detection distance and detection angle of the image frame according to the first coordinate, the second coordinate and the third coordinate; wherein the detection angle is used to represent the movement direction of the human target in the image frame, and the detection distance is used to represent the movement distance of the human target in the image frame;
[0018] According to the detection angle and detection distance, the behavior state of the human target in the image frame is determined.
[0019] According to a third aspect of an exemplary embodiment, a device for detecting a behavior state of a human target is provided, the device comprising:
[0020] A data acquisition unit, used to: acquire an image frame from an image detection device;
[0021] The behavior state determination unit is used to: if the image frame includes a human target, obtain the coordinates of multiple key points of the human target included in the image frame;
[0022] Among the coordinates of the multiple key points, obtain a first coordinate of a neck key point of a human target in an image frame previous to the image frame, as well as a second coordinate of the neck key point and a third coordinate of a hip key point of the human target in the image frame;
[0023] Determine the detection distance and detection angle of the image frame according to the first coordinate, the second coordinate and the third coordinate; wherein the detection angle is used to represent the movement direction of the human target in the image frame, and the detection distance is used to represent the movement distance of the human target in the image frame;
[0024] According to the detection angle and detection distance, the behavior state of the human target in the image frame is determined.
[0025] According to a fourth aspect of the exemplary implementation, a computer storage medium is provided, in which computer program instructions are stored. When the instructions are executed on a computer, the computer executes the method for detecting the behavioral state of a human target as in the first aspect.
[0026] In an embodiment of the present application, for an image frame from an image detection device, when the image frame includes a human target, the coordinates of multiple key points of the human target included in the image frame are obtained. Moreover, among the coordinates of multiple key points, the first coordinate of the neck key point of the human target in the previous image frame of the image frame, as well as the second coordinate of the neck key point of the human target in the image frame and the third coordinate of the hip key point are obtained; according to the first coordinate, the second coordinate and the third coordinate, the detection distance and the detection angle of the image frame are determined. Since the positional relationship between the hip key point and the neck key point can be determined, the detection angle can represent the movement direction of the human target in the image frame, and the detection distance can represent the movement distance of the human target in the image frame. Then, the behavior state of the human target, including but not limited to falling, lying down, walking or squatting, can be judged by the movement direction and movement distance. Such a design improves the accuracy of detecting the behavior state of pedestrian targets in public places. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0028] Figure 1 A flowchart of a method for detecting a behavior state of a human target provided by an embodiment of the present application is exemplarily shown;
[0029] Figure 2 The network structure diagram of CSPDarknet53 provided in an embodiment of the present application is exemplarily shown;
[0030] Figure 3 A schematic diagram of a key point identification stage provided by an embodiment of the present application is exemplarily shown;
[0031] Figure 4 A schematic diagram of a joint extraction stage provided in an embodiment of the present application is exemplarily shown;
[0032] Figure 5A schematic diagram exemplarily shows the comparison effect of determining key points using different algorithms provided in an embodiment of the present application;
[0033] Figure 6 A schematic diagram of 25 key points output by a second neural network based on the OpenPose algorithm provided in an embodiment of the present application is exemplarily shown;
[0034] Figure 7 A schematic diagram exemplarily shows a change in the cosine value of a detection angle under different behavior states provided by an embodiment of the present application;
[0035] Figure 8 A schematic diagram exemplarily shows a change in detection angle under different behavior states provided by an embodiment of the present application;
[0036] Fig. 9 A schematic diagram exemplarily shows a change in detection distance under different behavior states provided in an embodiment of the present application;
[0037] Fig.10 A flowchart of a complete method for detecting the behavior state of a human target provided by an embodiment of the present application is exemplified;
[0038] Fig.11 The following is a schematic diagram showing a structure of a device for detecting a behavior state of a human target provided by an embodiment of the present application;
[0039] Fig.12 The structural diagram of a human target behavior state detection device provided in an embodiment of the present application is exemplarily shown. DETAILED DESCRIPTION
[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0041] There are many people in public places, and crowding, trampling and other situations occasionally occur. In these situations, falling is the earliest behavior. Therefore, in scenes with complex personnel, it is necessary to accurately and timely detect whether pedestrians have fallen, and when a fall event is detected, quickly report it to facilitate the management personnel of public places to handle it, which can avoid danger to a certain extent.
[0042] In the related art, human fall detection includes detection based on hardware devices and detection based on traditional machine learning. Detection based on hardware devices includes detection based on wearable devices and detection based on environmental devices.
[0043] In the detection based on wearable devices, each pedestrian wears a wearable device to obtain data. The sensors in the wearable device mainly include accelerometers, gyroscopes or barometers, etc. The speed, inclination and air pressure changes of the monitored object are obtained through the sensors, thereby realizing the detection of fall events. This method requires wearing wearable devices at all times, which brings inconvenience to the daily life of the monitored object, and there may be situations where the detected object forgets to wear it and misses the detection. Therefore, this detection method is only suitable for single-person detection and is not suitable for monitoring a large number of people in public places.
[0044] In the detection of environmental equipment, the current or audio information collected by radar or sensors is used to detect changes in ground vibration, ground pressure, abnormal sounds or abnormal movements under infrared sensors in the monitoring area to detect fall events. This method is complex to implement, has high installation costs, and is very sensitive to external noise interference. The movement of any object can generate vibration, resulting in a very high false alarm rate and low detection accuracy.
[0045] In addition, traditional machine learning-based detection relies too much on the quality of manually extracted features, is highly subjective, and has difficulty learning more abstract high-dimensional features, which affects the accuracy of behavior recognition and classification, and has low dilution and robustness.
[0046] To this end, an embodiment of the present application provides a method for detecting the behavioral state of a human target, which calculates a detection angle and a detection distance for each image frame including at least one human target obtained by processing a video to be processed, based on the coordinates of the neck key points of the human target in two adjacent image frames, the image frame and the previous image frame, and the coordinates of the crotch key points of the human target in the image frame. The detection angle can represent the movement direction of the human target in the image frame, and the detection distance can represent the movement distance of the human target in the image frame. With such a design, the behavioral state of the human target in the image frame can be determined based on the detection angle and the detection distance. Exemplarily, the behavioral state may include falling, lying down, walking, or squatting, etc.
[0047] To further illustrate the technical solution provided by the embodiment of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiment of the present application provides the method operation steps shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided in the embodiment of the present application.
[0048] refer to Figure 1A flow chart of a method for detecting the behavior state of a human target is shown, and the method is applied to a behavior state detection device, and the behavior detection device can be a local electronic device or a cloud server. The technical solution provided by the embodiment of the present application is described.
[0049] S101: The behavior state detection device acquires an image frame from an image detection device.
[0050] S102: If the image frame includes a human target, the behavior state detection device obtains coordinates of multiple key points of the human target included in the image frame.
[0051] S103: The behavior state detection device obtains, from the coordinates of multiple key points, the first coordinate of the neck key point of the human target in the previous image frame, as well as the second coordinate of the neck key point and the third coordinate of the hip key point of the human target in the image frame.
[0052] S104: The behavior state detection device determines the detection distance and detection angle of the image frame according to the first coordinate, the second coordinate, and the third coordinate.
[0053] The detection angle is used to represent the movement direction of the human target in the image frame, and the detection distance is used to represent the movement distance of the human target in the image frame.
[0054] S105: The behavior state detection device determines the behavior state of the human target in the image frame according to the detection angle and the detection distance.
[0055] In an embodiment of the present application, for an image frame from an image detection device, when the image frame includes a human target, the coordinates of multiple key points of the human target included in the image frame are obtained. Moreover, among the coordinates of multiple key points, the first coordinate of the neck key point of the human target in the previous image frame of the image frame, as well as the second coordinate of the neck key point of the human target in the image frame and the third coordinate of the hip key point are obtained; according to the first coordinate, the second coordinate and the third coordinate, the detection distance and the detection angle of the image frame are determined. Since the positional relationship between the hip key point and the neck key point can be determined, the detection angle can represent the movement direction of the human target in the image frame, and the detection distance can represent the movement distance of the human target in the image frame. Then, the behavior state of the human target, including but not limited to falling, lying down, walking or squatting, can be judged by the movement direction and movement distance. Such a design improves the accuracy of detecting the behavior state of pedestrian targets in public places.
[0056] In S101, in a public place, each image detection device (such as a monitoring device) can process the captured video to obtain multiple image frames, and transmit the multiple image frames to the behavior state detection device. Alternatively, the image detection device can transmit the captured video to the behavior state detection device, which processes the video to obtain multiple video frames.
[0057] The behavior state detection device performs the same operation on each image frame to obtain the behavior state of the human target in the image frame. In actual application, one image frame may include multiple human targets. In the embodiment of the present application, a human target is taken as an example to illustrate the process of determining the behavior state of the human target. The process of determining the behavior state of other pedestrian targets is the same as the process of determining the behavior state of the human target.
[0058] In S102, if the image frame includes a human target, the coordinates of multiple key points of the human target included in the image frame are obtained; if the image frame does not include a human target, the image frame is discarded.
[0059] The multiple key points include the neck key point and the hip key point. This process can be implemented through steps A1-A3:
[0060] A1: The behavior state detection device inputs the image frame into the first neural network to determine the position information of the human target in the image frame.
[0061] Since the image frame may also include other objects, these objects may also be identified as key points, such as tables or chairs and other objects with obvious joints. Therefore, in order to improve accuracy, the behavior state detection device first filters out other non-human targets and extracts human targets. In this example, it can be implemented through the first neural network, using the image frame as the input of the neural network, and the output of the neural network is the location information of at least one human target. Taking the human target as an example, the execution process of the first neural network is explained.
[0062] The first neural network is a neural network based on the yolov5 algorithm; the backbone network of the first neural network is CSPDarknet53, and a Cross StagePartial structure is added after each Residual block structure in the first neural network. Optional, Figure 2 A network structure diagram of CSPDarknet53 provided in an embodiment of the present application. In complex application scenarios, the multi-scale changes and occlusion problems caused by the different distances of the same human target can be solved by CSPDarknet53.
[0063] Optionally, the behavior state detection device may determine the position information of the human target in the image frame through steps A1-1 to A1-3:
[0064] A1-1: The behavior state detection device inputs the image frame into the first neural network, and obtains at least two feature maps obtained after CSPDarknet53 performs feature extraction.
[0065] In the scene of pedestrian target detection in the embodiment of the present application, if only one feature map is obtained, some information may be lost, such as position information and multi-scale information. Position information and multi-scale information are particularly important for detecting key points. Therefore, in this example, at least two feature maps obtained after feature extraction by CSPDarknet53 can be obtained. These at least two feature maps include at least one high-level feature map and one low-level feature map, wherein the high-level feature map can provide high-level semantic information of the target in a complex scene, and the low-level feature map can provide the position information and multi-scale information of the target in the complex target scene.
[0066] When the number of feature maps obtained includes three, the three feature maps may include a first feature map, a second feature map, and a third feature map. The size of the image frame input to the first neural network is 608×608 as an example. The first feature map is used to describe the position information and multi-scale information of the human target in the image frame, and its size may be, for example, 76×76; the second feature map is used to describe the multi-scale information of the human target in the image frame, and its size may be, for example, 38×38; the third feature map is used to describe the high-level semantic information of the human target in the image frame, and its size may be, for example, 19×19. The high-level semantic information here may be information on specific details such as the nose and eyes.
[0067] A1-2: The behavior state detection device fuses at least two feature maps to obtain a fused feature map.
[0068] In order to fully integrate location information, multi-scale information and high-level voice information, and thus obtain more accurate pedestrian detection results, the behavior state detection device fuses at least two feature maps to obtain a fused feature map.
[0069] A1-3: The behavior state detection device determines the position information of the human target in the image frame based on the fused feature map.
[0070] The behavior state detection device runs a first neural network to determine the position information of the human target in the image frame according to the fusion feature map. The operation process of the first neural network can refer to the operation process of the neural network in the related art, which will not be repeated here.
[0071] In the above embodiment, the shallow network is convolved with the deep network to obtain feature fusion information of different scales, and the fusion feature map carries the feature fusion information. Such a design can improve the problem of small target details being lost as the number of network layers increases during the feature extraction process of the first neural network, and improve the accuracy of occlusion and small target pedestrian detection and positioning in complex scenes.
[0072] Optionally, adding a Cross StagePartial structure after each Residual block structure in the first neural network can reduce training parameters and make it easier to train.
[0073] A2: The behavior state detection device uses the position information to mark the image frame to obtain a first reference image frame.
[0074] The marking may refer to marking the position information of each pedestrian target in the image frame in the form of a detection frame, and obtaining an image frame including the detection frame, which is called the first reference image frame. That is, the first reference image frame includes the position information of the human target.
[0075] A3: The behavior state detection device inputs the reference image frame into the second neural network to determine the coordinates of multiple key points of the human target in the image frame.
[0076] Optionally, the second neural network is a neural network based on the OpenPose algorithm.
[0077] For the reference image frame, a second neural network based on the OpenPose algorithm is used to extract human skeleton points and output the coordinates of multiple key points of the human target. The key points here can be bone joints.
[0078] Among them, the OpenPose algorithm is a human posture estimation algorithm developed based on CNN. First, the input image is a reference image frame, which is processed by the front CNN network to generate a confidence map R and a set of two-dimensional vector fields L used to encode the local correlation vector field of the human body. The confidence map set is recorded as R = (R1, R2, R J ), the vector field set is recorded as L = (L1, L2, L C ). Each key point has J confidence maps, where R j ∈S w×h , j∈{1J}. Each limb has C vector fields, where L C ∈S w ×h×2 ,c∈{1C}, the position L in the reference image frame CCorresponding to a 2D vector. Then generate a normal vector graph, and construct a line connecting the next key point in the normal vector graph through a greedy algorithm. The confidence of each point is used to infer the two points with the highest probability on the same line, and the minimum spanning tree is used for iteration to obtain all the key point lines, that is, the human skeleton information.
[0079] Among them, the process of determining the coordinates of the key points can be divided into a key point recognition stage and a corresponding joint extraction stage. Figure 3 A schematic diagram of a key point identification stage provided in an embodiment of the present application, Figure 3 (a) and (b) are schematic diagrams of the key point recognition stage in two different situations. Figure 4 A schematic diagram of a joint extraction stage provided in an embodiment of the present application, wherein: Figure 4 (a) is the same as Figure 3 (a) in FIG. 1 is a schematic diagram of the joint extraction stage corresponding to FIG. 1 , which includes pedestrian targets in three walking states. Figure 4 (b) is the same as Figure 3 (b) in FIG. 1 is a schematic diagram of the joint extraction stage corresponding to the joint extraction stage, which includes two pedestrian targets in a walking state and one pedestrian target in a lying state.
[0080] Since the OpenPose algorithm is a bottom-up algorithm, if the second neural network based on the OpenPose algorithm is directly applied to determine the coordinates of the key points in the image frame, key points that do not belong to human targets may also be detected, such as key points of targets such as tables or chairs. Therefore, in the embodiment of the present application, the image frame is first processed by the first neural network based on the yolov5 algorithm to obtain the position information of the human targets in the image frame. This step filters the non-human targets; the position information is then used to mark the image frame to obtain a reference image frame, and finally the reference image frame is used as the second neural network based on the OpenPose algorithm. In this way, the key points obtained are all key points of human targets, excluding key points of non-human targets, thereby avoiding the situation of false detection of targets.
[0081] Figure 5 A schematic diagram of the comparison effect of determining key points using different algorithms provided in an embodiment of the present application, wherein: Figure 5 (a) is a schematic diagram of key points determined using only the OpenPose algorithm without using yolov5, including the key point 51 of the detected chair. Figure 5 (b) is a schematic diagram of the key points determined using the yolov5 and OpenPose algorithms. It can be seen that only the key points of the pedestrian target are detected.
[0082] This process can determine the coordinates of multiple key points of each human target. For the convenience of description, the human target is taken as an example to illustrate the process of determining the coordinates of multiple key points of the human target in the image frame. Generally, if a standard OpenPose-based second neural network is used, the coordinates of multiple key points of the human target output are usually the coordinates of the set 25 key points. Figure 6 A schematic diagram of 25 key points output by a second neural network based on the OpenPose algorithm provided in an embodiment of the present application.
[0083] Regarding S103, it can be determined through practical application that, among multiple key points, the positional relationship between the neck key point and the hip key point is more important for fall detection and can be used as the main parameter for fall detection.
[0084] Therefore, among the coordinates of the multiple key points, the first coordinate of the neck key point of the human target in the previous image frame of the image frame, as well as the second coordinate of the neck key point and the third coordinate of the hip key point of the human target in the image frame are obtained. Figure 6 , the neck key point is key point No. 0, and the hip key point is key point No. 8.
[0085] In S103, the behavior state detection device obtains the first coordinate of the neck key point of the human target in the previous image frame of the image frame, as well as the second coordinate of the neck key point and the third coordinate of the hip key point of the human target in the image frame.
[0086] The neck key point of the human target in the image frame is recorded as p1, the hip key point is recorded as p2, and the neck key point of the human target in the previous image frame is recorded as p0. Optionally, the first coordinate is p0 = (x0, y0), the second coordinate is p1 = (x1, y1), and the third coordinate is p2 = (x2, y2).
[0087] In this example, the image frame is any image frame after the first frame of the multiple image frames. For the first image frame of the multiple image frames, when determining the detection angle and detection distance of the first image frame, the first coordinate applied can take the default value. In addition, when obtaining the first coordinate, the second coordinate and the third coordinate, they are for the same human target. When at least two human targets are included in an image frame, for each pedestrian target, the respective first coordinate, second coordinate and third coordinate are obtained respectively.
[0088] In S104, the behavior state detection device determines the detection distance and detection angle of the image frame according to the first coordinate, the second coordinate and the third coordinate.
[0089] Optionally, the step of determining the detection angle and the detection distance of the image frame by the behavior state detection device can be implemented through steps B1-B2:
[0090] B1: The behavior state detection device determines the difference between the ordinate of the second coordinate and the ordinate of the first coordinate as the detection distance.
[0091] Among them, the detection distance is recorded as K, and the difference between the ordinate of the second coordinate and the ordinate of the first coordinate is K=y1-y0. In this example, K is the ordinate of the key point of the neck of the human target in the image frame and the ordinate of the key point of the neck of the human target in the previous frame of the image frame. Therefore, the difference between the two is the difference in the longitudinal drop distance of the neck between two adjacent frames, which can represent the movement of the key point of the neck of the human target in the image frame relative to the key point of the neck of the human target in the previous frame. Therefore, the detection distance can be used as an evaluation parameter for falling, lying down, walking or squatting.
[0092] B2: The behavior state detection device determines the movement direction of the human target according to the second coordinate and the third coordinate, and determines the detection angle according to the movement direction and the gravity direction unit vector.
[0093] The direction vector of the line between the key points of the neck of the human target in the image frame and the key points of the hips of the human target in the image frame is l = (x1-x2, y1-y2), and the direction represented by the direction vector L is the direction of movement of the human target. The unit vector of the gravity direction is e = (0, -1). The detection angle is represented by α, then therefore,
[0094] In this example, α is the angle between the coordinate line of the neck and hip key points and the vertical direction of gravity, that is, it is determined by the second coordinate of the neck key point and the third coordinate of the hip key point of the human target in the image frame, and combines the direction vector between the neck key point and the hip key point and the unit vector of the gravity direction. Therefore, α can represent the relative movement of the neck key point and the hip key point of the human target in the image frame. Therefore, the detection distance can be used as an evaluation parameter for falling, lying down, walking or squatting.
[0095] In S104: the behavior state detection device determines the behavior state of the human target in the image frame according to the detection angle and the detection distance, which may include the following situations:
[0096] The first situation Q1: falling state.
[0097] If the detection angle is greater than or equal to the angle threshold, and the detection distance is greater than or equal to the distance threshold, it is determined that the behavior state of the human target in the image frame is a falling state.
[0098] The second situation Q2: lying down state.
[0099] If the detection angle is greater than or equal to the angle threshold, and the detection distance is less than the distance threshold, it is determined that the behavior state of the human target in the image frame is a lying state.
[0100] The third situation Q3: walking state or squatting state.
[0101] If the detection angle is less than the angle threshold, it is determined that the behavior state of the human target in the image frame is a walking state or a squatting state.
[0102] That is to say, when the detection angle is greater than or equal to the angle threshold, the behavior state detection device can determine that the behavior state of the human target in the image frame is a falling state or a lying down state. When the detection angle is less than the angle threshold, the behavior state detection device can determine that the behavior state of the human target in the image frame is a walking state or a squatting state. However, it is impossible to continue to distinguish between the falling state and the lying down state. Therefore, in order to distinguish between the falling state and the lying down state, the detection distance parameter is further introduced for judgment. The purpose of the embodiment of the present application is to identify the falling behavior. Therefore, there is no need to further distinguish between the walking state or the squatting state. In the actual application process, if there is a need to distinguish, it can also be further distinguished according to other evaluation parameters.
[0103] Figure 7 The embodiment of the present application provides a change in the cosine value of the detection angle under different behavior states. Figure 7 , the α under falling behavior and walking behavior will be significantly different. At the moment when walking behavior turns into falling behavior, the angle α between the line vector of the key points of the neck and hips and the vertical direction of gravity will change significantly. When walking normally, the cosine value of the angle α is infinitely close to 1. When falling behavior occurs, the cosine value of the angle α decreases rapidly at about 10 frames and then tends to be flat. Figure 7 The figure shows the changes in the cosine value of the angle α under multiple shooting angles and different types of falling behaviors, such as falling from the side, falling from the front, and falling from the back. Among them, curve 71 is the cosine value of the detection angle of the side fall state, curve 72 is the cosine value of the detection angle of the front fall state, curve 73 is the cosine value of the detection angle of the back fall state, and curve 4 is the cosine value of the detection angle of the walking state. Figure 7 It can be seen that the cosine value of the detection angle of normal walking is close to 1, that is, the detection angle of the walking state is close to 0, that is, the direction of the line connecting the neck key point and the hip key point in the walking state is basically consistent with the direction of gravity.
[0104] Figure 8The embodiment of the present application provides a change in the detection angle under different behavior states. Among them, curve 81 is the detection angle of the falling state, curve 82 is the detection angle of the walking state, curve 83 is the detection angle of the squatting state, and curve 84 is the detection angle of the lying state. Figure 8 It can be seen that the detection angles of the falling state and the lying down state are both greater than or equal to 15 degrees, and the detection angles of the walking state and the squatting state are both less than 15 degrees.
[0105] In this example, the angle threshold may be 15 degrees, and when α≥15 degrees, the behavior state detection device may determine that the human target in the image frame is in a falling state or a lying state. However, this cannot further distinguish the falling state from the lying state.
[0106] In order to further distinguish between the falling state and the lying down state, it is determined whether the detection distance K is greater than or equal to the distance threshold. When the falling behavior occurs, K will change dramatically in adjacent frames. Exemplarily, the distance threshold can be 10. If K ≥ 10, the behavior state detection device can determine that the human target is in the falling state; if K < 10, the behavior state detection device can determine that the human target is in the lying down state.
[0107] Fig. 9 The embodiment of the present application provides a change in the detection distance under different behavior states. Curve 91 is the detection distance in the falling state, curve 92 is the detection distance in the walking state, curve 93 is the detection distance in the squatting state, and curve 94 is the detection distance in the lying state. Fig. 9 It can be seen that the detection distance of the falling state is greater than or equal to 10, and the detection distance of the lying down state is less than 10.
[0108] The angle threshold may be 15 degrees. When α<15 degrees, the behavior state detection device may determine that the human target in the image frame is in a walking state or a squatting state.
[0109] Optionally, when the detection angle α≥15 and K≥10, the behavior state detection device can determine that the human target is in a falling state; when the detection angle α≥15 and K<10, the behavior state detection device can determine that the human target is in a lying down state; when the detection angle α<15, the behavior state detection device can determine that the human target is in a walking state or a squatting state.
[0110] In the above embodiment, after determining the position information of the human target, the key points of the neck and hips of the human body can be extracted, and whether a fall behavior has occurred can be determined based on the characteristic changes of the key points of the neck and hips, effectively solving the problem of false detection caused by interfering actions such as squatting, and greatly improving the detection accuracy. The fall detection features of interfering behaviors such as squatting and lying down can be effectively distinguished.
[0111] Through the above embodiments, the behavior state detection device can determine the behavior state of each human target in each image frame. In order to determine the behavior state of the human target in the entire video (the video to be processed), still taking the human target as an example, the behavior state detection device can determine the behavior state of the human target in the video to be processed based on multiple behavior states of the human target in multiple image frames.
[0112] Exemplarily, if the number of image frames from the image detection device is M, the behavior state of the human target in the M image frames can be determined. For example, when the behavior state of the human target in a set number N (e.g., N=2 / 3*M) of image frames is all a fall state, it can be determined that the behavior state of the human target in the video including M image frames is a fall state. The method for determining the behavior state of other human targets in the video is the same as that of the human target, and will not be repeated here.
[0113] In order to make the technical solution of this application more perfect, Fig.10 A flowchart of a complete method for detecting the behavioral state of a human target provided in an embodiment of the present application.
[0114] S1001: The behavior state detection device obtains an image frame from the image detection device.
[0115] The implementation method of this step may refer to step S101 and will not be described in detail here.
[0116] S1002: If the image frame includes a human target, the behavior state detection device inputs the image frame into the first neural network to obtain at least two feature maps obtained after CSPDarknet53 performs feature extraction.
[0117] The implementation method of this step can refer to step A1-1 and will not be repeated here.
[0118] S1003: The behavior state detection device fuses at least two feature maps to obtain a fused feature map.
[0119] The implementation method of this step can refer to step A1-2, which will not be repeated here.
[0120] S1004: The behavior state detection device determines the position information of the human target in the image frame according to the fused feature map.
[0121] The implementation method of this step can refer to steps A1-3, which will not be repeated here.
[0122] S1005: The behavior state detection device uses the position information to mark the image frame to obtain a reference image frame.
[0123] The implementation method of this step can refer to step A2 and will not be repeated here.
[0124] S1006: The behavior state detection device inputs the reference image frame into the second neural network to determine the coordinates of multiple key points of the human target in the image frame.
[0125] The implementation method of this step can refer to step A3 and will not be repeated here.
[0126] S1007: The behavior state detection device obtains, from the coordinates of multiple key points, the first coordinate of the neck key point of the human target in the previous image frame, as well as the second coordinate of the neck key point and the third coordinate of the hip key point of the human target in the image frame.
[0127] The implementation method of this step may refer to step S103 and will not be described in detail here.
[0128] S1008: The behavior state detection device determines the difference between the ordinate of the second coordinate and the ordinate of the first coordinate as the detection distance.
[0129] The implementation method of this step can refer to step B1 and will not be repeated here.
[0130] S1009: The behavior state detection device determines the movement direction of the human target according to the second coordinate and the third coordinate, and determines the detection angle according to the movement direction and the gravity direction unit vector.
[0131] Among them, the first coordinate is the coordinate of the neck key point of the human target in the previous image frame of the image frame, the second coordinate is the coordinate of the neck key point of the human target in the image frame, and the third coordinate is the coordinate of the hip key point of the human target in the image frame.
[0132] The implementation method of this step can refer to step B2 and will not be repeated here.
[0133] It should be noted that there is no obvious order relationship between step S1008 and step S1009. This is just an example and does not constitute a specific limitation.
[0134] S1010: If the detection angle is greater than or equal to the angle threshold, and the detection distance is greater than or equal to the distance threshold, the behavior state detection device determines that the behavior state of the human target in the image frame is a falling state.
[0135] The implementation method of this step can refer to situation Q1, which will not be repeated here.
[0136] S1011: If the detection angle is greater than or equal to the angle threshold, and the detection distance is less than the distance threshold, the behavior state detection device determines that the behavior state of the human target in the image frame is a lying state.
[0137] The implementation method of this step can refer to situation Q2 and will not be repeated here.
[0138] S1012: If the detection angle is less than the angle threshold, the behavior state detection device determines that the behavior state of the human target in the image frame is a walking state or a squatting state.
[0139] The implementation method of this step can refer to situation Q3, which will not be repeated here.
[0140] S1013: If the human target in N consecutive image frames is in a falling state, the behavior state detection device determines that the behavior state of the human target in the video to be processed is a falling state.
[0141] The implementation method of this step can refer to the above-mentioned embodiment of determining the behavior state in the video to be processed, which will not be repeated here.
[0142] Compared with other behavioral events, the most obvious difficulty of falling behavior is that the posture of people changes greatly and is easily affected by environmental occlusion. In a complex scene with multiple people in a public area, it is easy to appear that people are occluded and the detection of small-target pedestrians at a distance is incomplete, resulting in missed detection or misdetection of small targets, and interfering actions such as squatting greatly affect the accuracy of fall detection. Therefore, in the embodiment of the present application, a multi-scale dual-stream heterogeneous convolutional neural network (first neural network and second neural network) based on deep learning is used to perform deep learning of falling behavior, which can meet the small target detection in complex scenes with multiple people, greatly reduce the probability of missed detection, and directly lock the person who falls. In addition, by extracting the key points of the human body, the downward trend of the key points of the neck and the change of the angle between the neck-hip connection vector and the gravity direction vector are used to jointly determine the fall event, and interfering actions such as squatting can be effectively avoided, ensuring the real-time, effectiveness and advancement of the detection of the fall event. Without considering the influence of the body shape of the pedestrian target, the detection object can be quickly read and locked, which is suitable for public places, without the need for adaptive training in advance, and deep learning is performed during the detection process, which is more accurate and more robust. Therefore, accurate detection and early warning of people falling in complex scenarios in public areas can be achieved.
[0143] In order to demonstrate the detection effect of the embodiment of the present application, Table 1 shows the verification result of a method in the embodiment of the present application on a self-owned data set, wherein the total number of video samples is the number of videos to be detected.
[0144] Table 1 Verification results of the method in the embodiment of the present application on the own data set
[0145]
[0146] Table 2 shows a comparison table of the effects of a method in the prior art and the method in the embodiment of the present application.
[0147] method Common Datasets Own dataset IDTs 85.4% 83.2% C3D 92.3% 87.6% LRCN 89.1% 90.8% Two-stream 93.7% 91.5% Method of the present application 96% 94%
[0148] Among them, the proprietary data set can be the videos to be processed collected by the embodiments of the present application, and the general data set refers to the internationally used fall data set, including 30 videos of fall situations and 40 videos of daily life activities. IDTs include models IDT[1] and IDT[2], which are two general improved dense trajectory (IDT) models. C3D refers to an action recognition model based on 3D convolution (convolutional dimension, C3D), and LRCN refers to a long-term recurrent convolutional neural network (long-term recurrent convolutional networks, LRCN) model. Two-stream refers to a homogeneous network model of two streams (time stream and space stream).
[0149] like Fig.11 As shown, based on the same inventive concept, an embodiment of the present application provides a behavior state detection device for a human target, including: a data acquisition unit 111 and a behavior state determination unit 112.
[0150] A data acquisition unit, used to: acquire an image frame from an image detection device;
[0151] The behavior state determination unit is used to: if the image frame includes a human target, obtain the coordinates of multiple key points of the human target included in the image frame;
[0152] Among the coordinates of the multiple key points, obtain a first coordinate of a neck key point of a human target in an image frame previous to the image frame, as well as a second coordinate of the neck key point and a third coordinate of a hip key point of the human target in the image frame;
[0153] Determine the detection distance and detection angle of the image frame according to the first coordinate, the second coordinate and the third coordinate; wherein the detection angle is used to represent the movement direction of the human target in the image frame, and the detection distance is used to represent the movement distance of the human target in the image frame;
[0154] According to the detection angle and detection distance, the behavior state of the human target in the image frame is determined.
[0155] In an optional implementation, the behavior state determination unit 112 is specifically configured to:
[0156] Determine the difference between the ordinate of the second coordinate and the ordinate of the first coordinate as the detection distance;
[0157] The moving direction of the human target is determined according to the second coordinate and the third coordinate, and the detection angle is determined according to the moving direction and the gravity direction unit vector.
[0158] In an optional implementation, the behavior state determination unit 112 is specifically configured to:
[0159] If the detection angle is greater than or equal to the angle threshold, and the detection distance is greater than or equal to the distance threshold, it is determined that the behavior state of the human target in the image frame is a falling state.
[0160] In an optional implementation, the behavior state determination unit 112 is specifically configured to:
[0161] If the detection angle is greater than or equal to the angle threshold, and the detection distance is less than the distance threshold, it is determined that the behavior state of the human target in the image frame is a lying state.
[0162] In an optional implementation, the behavior state determination unit 112 is specifically configured to:
[0163] If the detection angle is less than the angle threshold, it is determined that the behavior state of the human target in the image frame is a walking state or a squatting state.
[0164] In an optional implementation, the behavior state determination unit 112 is specifically configured to:
[0165] Inputting the image frame into the first neural network to determine the position information of the human target in the image frame;
[0166] Using the position information to mark the image frame to obtain a reference image frame; wherein the reference image frame includes the position information of the human target;
[0167] The reference image frame is input into the second neural network to determine the coordinates of multiple key points of the human target in the image frame.
[0168] In an optional implementation, the backbone network of the first neural network is CSPDarknet53, and a Cross Stage Partial structure is added after each Residual block structure in the first neural network; the behavior state determination unit 112 is specifically used for:
[0169] Input the image frame into the first neural network to obtain at least two feature maps obtained after feature extraction by CSPDarknet53;
[0170] Fusing at least two feature maps to obtain a fused feature map;
[0171] According to the fused feature map, the position information of the human target in the image frame is determined.
[0172] In an optional implementation, the first neural network is a neural network based on the yolov5 algorithm, and the second neural network is a neural network based on the OpenPose algorithm.
[0173] Since the device is the device in the method in the embodiment of the present application, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0174] like Fig.12 As shown, based on the same inventive concept, an embodiment of the present application provides a human target behavior state detection device, which includes: a processor 1201 and a data transmission unit 1202.
[0175] The data transmission unit 1202 is configured to perform:
[0176] Acquire an image frame from an image detection device;
[0177] The processor 1201 is configured to execute:
[0178] If the image frame includes a human target, then obtaining coordinates of multiple key points of the human target included in the image frame; wherein the multiple key points include a neck key point and a hip key point;
[0179] Among the coordinates of the multiple key points, obtain a first coordinate of a neck key point of a human target in an image frame previous to the image frame, as well as a second coordinate of the neck key point and a third coordinate of a hip key point of the human target in the image frame;
[0180] Determine the detection distance and detection angle of the image frame according to the first coordinate, the second coordinate and the third coordinate; wherein the detection angle is used to represent the movement direction of the human target in the image frame, and the detection distance is used to represent the movement distance of the human target in the image frame;
[0181] According to the detection angle and detection distance, the behavior state of the human target in the image frame is determined.
[0182] In an optional implementation, the processor 1201 is specifically configured to execute:
[0183] Determine the difference between the ordinate of the second coordinate and the ordinate of the first coordinate as the detection distance;
[0184] The moving direction of the human target is determined according to the second coordinate and the third coordinate, and the detection angle is determined according to the moving direction and the gravity direction unit vector.
[0185] In an optional implementation, the processor 1201 is specifically configured to execute:
[0186] If the detection angle is greater than or equal to the angle threshold, it is determined that the behavior state of the human target in the image frame is a falling state or a lying state.
[0187] In an optional implementation, the processor 1201 is specifically configured to execute:
[0188] If the detection angle is less than the angle threshold, it is determined that the behavior state of the human target in the image frame is a walking state or a squatting state.
[0189] In an optional implementation, the processor 1201 is specifically configured to execute:
[0190] If the detection angle is less than the angle threshold, it is determined that the behavior state of the human target in the image frame is a walking state or a squatting state.
[0191] In an optional implementation, the processor 1201 is specifically configured to execute:
[0192] Inputting the image frame into the first neural network to determine the position information of the human target in the image frame;
[0193] Using the position information to mark the image frame to obtain a reference image frame; wherein the reference image frame includes the position information of the human target;
[0194] The reference image frame is input into the second neural network to determine the coordinates of multiple key points of the human target in the image frame.
[0195] In an optional implementation, the backbone network of the first neural network is CSPDarknet53, and a Cross Stage Partial structure is added after each Residual block structure in the first neural network;
[0196] In an optional implementation, the processor 1201 is specifically configured to execute:
[0197] Input the image frame into the first neural network to obtain at least two feature maps obtained after feature extraction by CSPDarknet53;
[0198] Fusing at least two feature maps to obtain a fused feature map;
[0199] According to the fused feature map, the position information of the human target in the image frame is determined.
[0200] In an optional implementation, the first neural network is a neural network based on the yolov5 algorithm, and the second neural network is a neural network based on the OpenPose algorithm.
[0201] The embodiment of the present application also provides a computer storage medium, in which computer program instructions are stored. When the instructions are executed on a computer, the computer executes the steps of the above-mentioned method for detecting the behavioral state of a human target.
[0202] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0203] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0204] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0205] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0206] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A method for detecting the behavior state of a human target, characterized in that: include: Acquire an image frame from an image detection device; If the image frame includes a human target, obtaining coordinates of a plurality of key points of the human target included in the image frame; Among the coordinates of the multiple key points, obtain a first coordinate of a neck key point of the human target in an image frame previous to the image frame, as well as a second coordinate of the neck key point and a third coordinate of a hip key point of the human target in the image frame; Determine the detection distance and detection angle of the image frame according to the first coordinate, the second coordinate and the third coordinate; wherein the detection angle is used to represent the movement direction of the human target in the image frame, and the detection distance is used to represent the movement distance of the human target in the image frame; The behavior state of the human target in the image frame is determined according to the detection angle and the detection distance.
2. The method according to claim 1, characterized in that The step of determining the detection angle and the detection coordinates of the image frame according to the first coordinate, the second coordinate, and the third coordinate includes: Determine the difference between the ordinate of the second coordinate and the ordinate of the first coordinate as the detection distance; The moving direction of the human target is determined according to the second coordinate and the third coordinate, and the detection angle is determined according to the moving direction and a unit vector of the gravity direction.
3. The method according to claim 1, characterized in that The determining the behavior state of the human target in the image frame according to the detection angle and the detection distance includes: If the detection angle is greater than or equal to an angle threshold, and the detection distance is greater than or equal to a distance threshold, it is determined that the behavior state of the human target in the image frame is a falling state.
4. The method according to claim 1, characterized in that: The determining the behavior state of the human target in the image frame according to the detection angle and the detection distance includes: If the detection angle is greater than or equal to an angle threshold, and the detection distance is less than a distance threshold, it is determined that the behavior state of the human target in the image frame is a lying state.
5. The method according to claim 1, characterized in that The determining the behavior state of the human target in the image frame according to the detection angle and the detection distance includes: If the detection angle is less than the angle threshold, it is determined that the behavior state of the human target in the image frame is a walking state or a squatting state.
6. The method according to claim 1, characterized in that The acquiring of the coordinates of the multiple key points of the human target included in the image frame comprises: Inputting the image frame into a first neural network to determine the position information of the human target in the image frame; Using the position information to mark the image frame to obtain a reference image frame; wherein the reference image frame includes the position information of the human target; The reference image frame is input into a second neural network to determine the coordinates of multiple key points of the human target in the image frame.
7. The method according to claim 6, characterized in that The backbone network of the first neural network is CSPDarknet53, and a Cross StagePartial structure is added after each Residual block structure in the first neural network; The step of inputting the image frame into a first neural network to determine the position information of the human target in the image frame includes: Inputting the image frame into the first neural network, obtaining at least two feature maps obtained after feature extraction by the CSPDarknet53; Fusing the at least two feature maps to obtain a fused feature map; According to the fused feature map, the position information of the human target in the image frame is determined.
8. The method according to any one of claims 1 to 7, characterized in that: The first neural network is a neural network based on the yolov5 algorithm, and the second neural network is a neural network based on the OpenPose algorithm.
9. A behavior status detection device, characterized in that: including a data transmission unit and a processor; The data transmission unit is configured to perform: Acquire an image frame from an image detection device; The processor is configured to perform: If the image frame includes a human target, then obtaining coordinates of a plurality of key points of the human target included in the image frame; wherein the plurality of key points include a neck key point and a hip key point; Among the coordinates of the multiple key points, obtain a first coordinate of a neck key point of the human target in an image frame previous to the image frame, as well as a second coordinate of the neck key point and a third coordinate of a hip key point of the human target in the image frame; Determine the detection distance and detection angle of the image frame according to the first coordinate, the second coordinate and the third coordinate; wherein the detection angle is used to represent the movement direction of the human target in the image frame, and the detection distance is used to represent the movement distance of the human target in the image frame; The behavior state of the human target in the image frame is determined according to the detection angle and the detection distance.
10. The behavior state detection device according to claim 9, characterized in that: The processor is specifically configured to execute: Determine the difference between the ordinate of the second coordinate and the ordinate of the first coordinate as the detection distance; Determining the moving direction of the human target according to the second coordinate and the third coordinate; The detection angle is determined according to the movement direction and the gravity direction unit vector.
