Pedestrian turning abnormal behavior recognition method and device based on human joint sub-segment

Abstract

This invention relates to a method and apparatus for identifying abnormal pedestrian U-turn behavior based on human joint sub-segments. The identification method includes the following steps: acquiring image frames to be identified and constructing an image-based joint sub-segment model; extracting the dynamic features of the identified object in each image frame based on the joint sub-segment model; and obtaining the pedestrian U-turn abnormal behavior identification result according to a preset pedestrian U-turn abnormality criterion and the dynamic features. Compared with the prior art, this invention, by constructing a joint sub-segment model based on key human body nodes identified in images before the abnormal behavior occurs, can accurately identify and predict the abnormal behavior that is about to occur, with high reliability.

Description

Technical Field

[0001] This invention relates to the field of crowd flow stability analysis technology, and in particular to a method and device for identifying abnormal pedestrian U-turn behavior based on human joint sub-segments. Background Technology

[0002] In recent years, the coverage of network surveillance cameras in public places has increased, resulting in abundant video and image resources. Computer vision-based human behavior recognition technology has also been fully developed and widely applied. Foreign scholars began researching crowd evacuation and emergencies at the beginning of the last century, gradually transitioning from initial studies on crisis management systems and pedestrian organization characteristics to research on pedestrian flow theories, group evacuation behaviors and models, and pedestrian behavior simulation. This has formed a preliminary and relatively comprehensive research system.

[0003] However, current research on pedestrian abnormal U-turn behavior still has several shortcomings: 1) Most current research on the identification of pedestrian abnormal U-turn behavior focuses on the analysis of causes after an accident occurs, and there is a lack of research on predictive analysis before an accident occurs. 2) Existing research on crowd stability based on machine vision mostly focuses on crowd flow or density information extracted from image and video information, and there are few studies on stability analysis by extracting dynamic characteristics such as force, speed or acceleration from video image data. Summary of the Invention

[0004] The purpose of this invention is to overcome the defects of the prior art and provide a highly reliable method and device for identifying abnormal pedestrian turning behavior based on human joint sub-segments.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A method for identifying abnormal pedestrian U-turn behavior based on human joint sub-segments includes the following steps:

[0007] Obtain the image frame to be identified and construct an image-based joint sub-segment model;

[0008] Based on the joint sub-segment model, the dynamic features of the identified object in each image frame are extracted;

[0009] Based on the preset abnormal pedestrian U-turn criteria and the aforementioned dynamic characteristics, the abnormal pedestrian U-turn behavior identification results are obtained.

[0010] Furthermore, the steps for constructing the joint sub-segment model include:

[0011] Identify key human body nodes and extract their coordinates;

[0012] Based on the aforementioned key human nodes, body segments were re-divided;

[0013] Based on the body segments, calculate the segment mass ratio and segment centroid coefficient, and calculate the centroid coordinates of the body segments;

[0014] The total centroid coordinates of the human body are calculated using the centroid coordinates to form a joint sub-segment model.

[0015] Furthermore, the key human body nodes include: top of head, left ear, right ear, left eye, right eye, nose, left corner of mouth, right corner of mouth, left wrist, right wrist, left ankle, right ankle, left knee, right knee, left elbow, right elbow, left shoulder, right shoulder, neck, left hip, and right hip.

[0016] Furthermore, the segmentation of the body includes: head and neck, shoulders, upper body, upper arm, forearm, thigh, and lower leg.

[0017] Furthermore, the segment mass ratio = segment mass / human body mass, and the segment centroid coefficient = distance from segment centroid to endpoint / segment length.

[0018] Furthermore, the dynamic characteristics include the force at the center of mass of the motion and the angle between the shoulders and the velocity direction.

[0019] Furthermore, the abnormal pedestrian U-turn criterion is that the following two conditions are met simultaneously:

[0020] 1) The rate of change of centroid force in frame t is greater than the threshold.

[0021] 2) At frame t+a, the angle between the shoulders and the velocity direction changes by more than 180 degrees, and 'a' is less than one walking cycle.

[0022] Furthermore, the force at the total center of mass is a weighted sum of the forces at the center of mass positions of each body segment, expressed as:

[0023]

[0024] in, The force at the center of mass of the motion For the force at the center of mass of the k-th body segment, w k is the weight of the k-th body segment, and n is the number of body segments.

[0025] Furthermore, the weight of the kth body segment is proportional to the distance from the centroid of the kth body segment to the total centroid of the human body.

[0026] The present invention also provides an electronic device, including one or more processors, a memory, and one or more programs stored in the memory, said one or more programs including instructions for executing the pedestrian U-turn abnormal behavior recognition method based on human joint sub-segments as described above.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. This invention considers the mass distribution of different joints in the human body, constructs a sub-segment human skeleton model, builds a novel way of expressing the center of mass of motion, studies the dynamic characteristics of pedestrian trajectory, velocity, acceleration, etc., applies abnormal behavior criteria to the study of perturbation theory, and establishes a perturbation propagation model to address the lack of research on perturbation within a population. It explores in depth the dynamic impact of perturbation within a population on population stability. For key human nodes identified through images before abnormal behavior occurs, the invention constructs joint sub-segment models to accurately identify and predict impending abnormal behavior, effectively providing new ideas for subsequent research on population stability analysis.

[0029] 2. This invention divides the human body into 13 independent body segments, including the head and neck, shoulders, upper body, upper arm, forearm, thigh, and calf. The joint sub-segment model has high reliability and efficiency, thereby improving the recognition efficiency of abnormal pedestrian U-turn behavior.

[0030] 3. This invention uses machine vision technology to extract dynamic features such as force, velocity, or acceleration of an object based on video or image information. Based on these acquired dynamic feature data, it can accurately identify abnormal behavior and achieve early identification with a certain predictive lead time. It has high reliability and can be applied to subsequent population stability analysis research. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the process of the present invention;

[0032] Figure 2 This is a schematic diagram of key node comparison based on images in this invention;

[0033] Figure 3 This is a schematic diagram showing the angle between the two shoulders and the velocity direction of the center of mass in this invention;

[0034] Figure 4 This is a pedestrian trajectory diagram during an abnormal U-turn in an embodiment of the present invention;

[0035] Figure 5 This refers to the lead time for predicting abnormal U-turn behavior in embodiments of the present invention. Detailed Implementation

[0036] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0037] like Figure 1As shown, this embodiment provides a method for identifying abnormal pedestrian U-turn behavior based on human joint sub-segments, including the following steps:

[0038] Step S1: Obtain the image frame to be identified and construct an image-based joint sub-segment model.

[0039] The steps for constructing the joint sub-segment model include: identifying key human nodes and extracting their coordinates; re-dividing the body into segments based on the key human nodes; calculating the segment mass ratio and segment centroid coefficient based on the body segments, and calculating the centroid coordinates of the body segments; and using the centroid coordinates to calculate the total centroid coordinates of the human body, thus forming the joint sub-segment model.

[0040] In existing techniques for acquiring key human body nodes through image processing, the most mature and commonly used methods involve 15, 18, 21, or 25 key points. To improve recognition accuracy, additional facial key points are often added to ensure no individual pedestrian is lost. This embodiment uses 21 key points, including the top of the head, left ear, right ear, left eye, right eye, nose, left corner of the mouth, right corner of the mouth, left wrist, right wrist, left ankle, right ankle, left knee, right knee, left elbow, right elbow, left shoulder, right shoulder, neck, left hip, and right hip. Figure 2 As shown.

[0041] In this embodiment, drawing on the body segmentation method proposed by Winter et al., the human body is divided into 13 independent body segments: head and neck, shoulders (left and right), upper body (left and right), upper arm (left and right), forearm (left and right), thigh (left and right), and lower leg (left and right). Each body segment includes proximal and distal coordinates. The formula for calculating the centroid position of a body segment is defined as follows:

[0042] x k =x p l p +x d l d

[0043] y k =y p l p +y d l d

[0044] z k =z p l p +z d l d

[0045] Among them, (x k y k z k(x) represents the centroid coordinates of the segment; p y p z p (x) is the coordinate of the proximal end of the segment; (x) d y d z d ) represents the coordinates of the distal end of the segment; l p It is the percentage of segment length from proximal to distal; d This is the percentage of segment length from distal to distal, as detailed in Table 1.

[0046] Table 1. Parameters required for image-based centroid calculation

[0047]

[0048] Note: Segment mass ratio = segment mass / body mass; Segment centroid coefficient = distance from segment centroid to endpoint / segment length.

[0049] Finally, the position of the total center of mass of the human body is calculated using the positions of the centers of mass of each segment. The calculation formula is shown in the following formula:

[0050]

[0051] In the formula, (x c y c z c (x) represents the coordinates of the human body's center of mass; k y k z k ) represents the coordinates of the k-th segment; m k is the mass of the k-th segment; M is the total mass of the 13 body segments.

[0052] Step S2: Based on the joint sub-segment model, extract the dynamic features of the identified object in each image frame. In this embodiment, the extracted dynamic features include the force at the center of mass of motion and the angle between the shoulders and the velocity direction.

[0053] The force at the overall center of mass is derived from the forces in each segment. In pedestrian studies, the positions P of the center of mass of each segment are determined. k (x k y k z k This will be discussed separately. First, let's consider a mass of m. k The velocity of the particle at a certain moment is v k Then the momentum can be calculated as:

[0054]

[0055] According to Newton's second definition, the force at the center of mass of the k-th body segment is:

[0056]

[0057] In this embodiment, the forces at the centroid positions of the 13 sub-segments are weighted and summed to obtain the total centroid of motion (x). c y c z c The force of )

[0058]

[0059] In the above weighted summation, the weight is represented by w. k The contribution of geometric distance is determined. For a given pedestrian centroid, the farther the sub-segment centroid coordinates are (i.e., closer to the extremities), the greater their impact on overall pedestrian motion; for example, the greater the swing amplitude of the lower limbs when walking. Therefore, in this embodiment, the distance d from the sub-segment centroid to the pedestrian centroid is used to calculate the distance. k The dataset is then normalized to map the original data to a range of 0 to 1, which serves as the weight.

[0060]

[0061] Step S3: Based on the preset abnormal pedestrian U-turn criteria and the dynamic characteristics, obtain the abnormal pedestrian U-turn behavior identification result.

[0062] Pedestrian U-turn Abnormal Behavior Detection: When a pedestrian walks normally, the speed of each segment is relatively uniform, and the overall force value remains stable. If an abnormal U-turn occurs, the speed of each segment is no longer uniform, and the overall force value will show a significant change. When the rate of sudden change in force exceeds the safety threshold, it indicates that abnormal behavior is about to occur.

[0063] It is known that when a person turns around, the most obvious key change is the rotational movement of the shoulders. For example... Figure 3 As shown, the right shoulder is recorded as P. R The left shoulder is P L The angle θ between the shoulders and the velocity direction of the center of mass is determined by the relative positions of the centers of mass of the left and right shoulders. During normal walking, this angle is relatively small, but it increases when turning around, with the angle change exceeding 180°.

[0064] Therefore, the criterion for abnormal pedestrian U-turn behavior is: 1) The rate of change of centroid force in frame t is greater than the threshold. 2) At frame t+a, the angle between the shoulders and the velocity direction changes by more than 180 degrees, and 'a' is less than one walking cycle.

[0065]

[0066] In this embodiment, a real scene is captured at a certain location, and video frames are extracted at a frequency of f = 10 fps (Frames Per Second). Each frame has a resolution of 1920*1080 pixels, therefore the coordinate axes also use 1920*1080 values. The origin is set at the upper left corner of the image, with the x-axis horizontal and the y-axis vertical. A safe threshold for the rate of change of the centroid is obtained through repeated observations of pedestrians walking normally. Pedestrian walking cycle T cyc =0.6s.

[0067] Selecting a specific 25-frame abnormal turning motion process, the pedestrian trajectory diagram is as follows: Figure 4 As shown. To compare the recognition performance of the models, the upper curve is the centroid coordinate trajectory of the joint sub-segment model, and the lower curve is the pedestrian trajectory represented by the traditional rectangular box.

[0068] In the subsegment representation, the forward speed of the x-axis suddenly decreases between frames 9 and 10, and the rate of change of the centroid in frame 9 is much greater than... This can be seen as a buffer preparation for the turning behavior. In frame 11, the angle between the shoulders and the velocity direction changes by 210°, exceeding 180°. In the traditional representation method, the x-axis coordinate reaches its maximum in frame 13 and then gradually returns to normal.

[0069] The U-turn experiment was repeated 50 times, with the extraction frequency increased to f = 20 fps (Frames Per Second). Only the U-turn behavior portion was analyzed, with 20 frames per group, to determine the video frame moment when the U-turn occurred. First, to visually demonstrate the model's prediction of abnormal U-turn behavior, the lead time Δt was used. Second, to demonstrate the overall experimental effect of the model, the standard deviation (STD) was used as a measure, reflecting the spatial distribution of the samples.

[0070]

[0071]

[0072] Where N is the number of experiments, t n,d t is the moment when the centroid turns around, obtained from the criterion in the nth experiment. n,s It is the time when the conventional center of mass turns around, obtained from the nth experiment.

[0073] The results are as follows Figure 5 As shown, the prediction lead time is positive, indicating that the model can predict pedestrian U-turn behavior. 95% of the data points are 0.1s or longer, with a mean of 0.2570s and a standard deviation (STD) of 0.1460s, meaning the standard deviation is within 3 frames at this frame rate, thus validating the model's effectiveness.

[0074] If the above methods are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0075] In another embodiment, an electronic device is provided, including one or more processors, a memory, and one or more programs stored in the memory, the one or more programs including instructions for performing the pedestrian fall behavior recognition method based on the human dynamics center of mass model as described above.

[0076] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for identifying abnormal pedestrian U-turn behavior based on human joint sub-segments, characterized in that, Includes the following steps: Obtain the image frame to be identified and construct an image-based joint sub-segment model; Based on the joint sub-segment model, the dynamic features of the identified object in each image frame are extracted, including the force of the total center of mass and the angle between the shoulders and the velocity direction; Based on the preset abnormal pedestrian U-turn criteria and the aforementioned dynamic characteristics, the abnormal pedestrian U-turn behavior identification results are obtained. The abnormal pedestrian U-turn criterion is that the following two conditions must be met simultaneously: 1) No. t The rate of change of centroid force of the frame is greater than the threshold. φ thr ; 2) No. t+a At frame time, the change in the angle between the shoulders and the velocity direction is greater than 180 degrees. a Less than one walking cycle.

2. The method for identifying abnormal pedestrian U-turn behavior based on human joint sub-segments according to claim 1, characterized in that, The steps for constructing the joint sub-segment model include: Identify key human body nodes and extract their coordinates; Based on the aforementioned key human nodes, body segments were re-divided; Based on the body segments, calculate the segment mass ratio and segment centroid coefficient, and calculate the centroid coordinates of the body segments; The total centroid coordinates of the human body are calculated using the centroid coordinates to form a joint sub-segment model.

3. The method for identifying abnormal pedestrian U-turn behavior based on human joint sub-segments according to claim 2, characterized in that, The key points of the human body include: top of the head, left ear, right ear, left eye, right eye, nose, left corner of mouth, right corner of mouth, left wrist, right wrist, left ankle, right ankle, left knee, right knee, left elbow, right elbow, left shoulder, right shoulder, neck, left hip, and right hip.

4. The method for identifying abnormal pedestrian U-turn behavior based on human joint sub-segments according to claim 2, characterized in that, The resulting segmentation of the body includes: head and neck, shoulders, upper body, upper arms, forearms, thighs, and lower legs.

5. The method for identifying abnormal pedestrian U-turn behavior based on human joint sub-segments according to claim 2, characterized in that, The segment mass ratio = segment mass / human body mass, and the segment centroid coefficient = distance from segment centroid to endpoint / segment length.

6. The method for identifying abnormal pedestrian U-turn behavior based on human joint sub-segments according to claim 1, characterized in that, The force at the total center of mass of the motion is a weighted sum of the forces at the center of mass positions of each body segment, expressed as: in, The force at the center of mass of the motion For the first Force at the center of mass of each body segment For the first Weights of individual body segments The number of body segments to be divided.

7. The method for identifying abnormal pedestrian U-turn behavior based on human joint sub-segments according to claim 6, characterized in that, The first The weight of each body segment and the weight of the first segment The distance from the center of mass of each body segment to the total center of mass of the body is directly proportional.

8. An electronic device, characterized in that, It includes one or more processors, a memory, and one or more programs stored in the memory, said one or more programs including instructions for performing the pedestrian U-turn abnormal behavior recognition method based on human joint subsegments as described in any one of claims 1-7.

Images