A method for identifying and monitoring the status of luggage falling off in public places

Abstract

This invention proposes a method for identifying and monitoring the detachment of suitcases in public places. The method includes: setting up monitoring cameras and loudspeaker alerts in densely populated public areas; activating the monitoring cameras to capture video frames and providing them to a monitoring processor; the monitoring processor using image processing algorithms MMTrack and findContours to calculate the external force, center of gravity height, and rate of change of center of gravity height of the suitcase in the monitored area; and detecting the detachment state of the suitcase based on a suitcase detachment identification model. This invention identifies the motion characteristics of the suitcase throughout its entire movement and predicts its subsequent motion. Based on the identification results, timely warnings can be issued, effectively preventing potential hazards from detached suitcases. This method is of great significance for safety management in densely populated public places and can also be applied to logistics and freight fields, thereby reducing safety hazards and economic losses caused by detached suitcases or goods.

Description

TECHNICAL FIELD

[0001] The present application relates to a passenger luggage drop state recognition model considering combined motion and application, which is applied to public places with dense crowds. BACKGROUND

[0002] The luggage drop problem has always been one of the challenges faced by passengers in travel and travel, which often leads to chaos and disorder of the crowd in public places with dense crowds, and may cause major safety accidents such as crowd trampling. At present, although some researches have focused on the relationship between luggage and overall crowd evacuation, the research on the identification and detection of luggage drop state is still relatively less. Luggage drop may cause injury to passengers and surrounding people, so it is necessary to develop a luggage drop state recognition technology to improve the safety of people's travel.

[0003] At present, there are still some deficiencies in the research on the identification of luggage drop state: 1) Some researches have focused on the method of detecting luggage lifting, but the research on the luggage drop state, such as the collision between luggage and the ground or other objects, resulting in accidental drop, is still relatively less. 2) The research on luggage and passenger motion mainly focuses on the macroscopic angle, that is, the influence of passengers carrying luggage on the overall evacuation of the crowd. For example, Kitazono Y et al. proposed a method to detect the lifting of luggage, which can monitor the abnormal behavior of passengers discarding luggage and give a warning to this behavior. There is a lack of research on the luggage drop state from the microscopic angle. 3) The drop state of the luggage is closely related to the motion state of the passenger. At present, in the research on the identification of luggage drop state, there is a lack of in-depth research on the motion relationship between luggage and passengers. SUMMARY

[0004] In order to solve the above problems, the present application proposes a new luggage drop state recognition model considering combined motion. When applied, new technical means such as kinematics analysis and machine learning algorithm are adopted to propose a more comprehensive and accurate motion state monitoring method, so that the identification of luggage drop state is more accurate and fast.

[0005] The technical scheme of the present application is as follows:

[0006] The luggage drop state recognition model meets:

[0007]

[0008] That is: 1. From the t frame, the external force value F 外,t of the luggage is 0;

[0009] 2. From the t frame, the centroid distance change rate AT b,tgreater than 0.5 m / s;

[0010] 3. From the t frame, the height change rate of the center of mass of the luggage ΔH B,t greater than 0.2 m / s.

[0011] When the three determination conditions are met, it is considered that the luggage and the passenger are separated at time t, and the luggage falls off at time t.

[0012] A luggage falling state recognition and safety monitoring method in public places, comprising the following steps:

[0013] Step 1, setting up monitoring cameras, loudspeaker prompting equipment and the like in public places where people flow is dense, and connecting the monitoring cameras with a monitoring processor in a control room;

[0014] Step 2, starting the monitoring cameras, the photographing direction of which is tangent to the passenger movement direction, collecting video frames at a certain frequency, and providing video data to the monitoring processor;

[0015] Step 3, the monitoring processor calculates the external force F 外,t , the center of mass height H B and the center of mass height change rate ΔH B,t of the luggage in the monitoring place through image processing algorithms MMTrack and findContours respectively;

[0016] Step 4, detecting the luggage falling state based on a luggage falling state recognition model.

[0017] Specifically, the step 3 is implemented as follows:

[0018] (1) For the video frames collected by the camera, the luggage movement information is extracted to a four-parameter tuple R i through MMTrack, and the external force F 外,t of the luggage is calculated through equations (2)-(11);

[0019] (2) The outline picture of the luggage is extracted through findContours, and is converted into a gray scale picture;

[0020] (3) The origin moment of the gray scale picture can be calculated through equation (13):

[0021]

[0022] Wherein, M pq represents the origin moment, p and q represent the order of the moment, x and y represent the pixel coordinates in the image, and I(x, y) represents the pixel gray scale value at the coordinates (x, y).

[0023] The zeroth moment M of the luggage gray image is calculated by formula (13) B,00 and the first moment M B,01 , M B,10 The zeroth moment M of the passenger gray image is calculated by formula (14) C,00 and the first moment M C,01 , M C,10 .

[0024] The centroid height H of the luggage B can be obtained by formula (15):

[0025]

[0026] The centroid height change rate AH of the luggage B,t is:

[0027]

[0028] Where, AH B represents the centroid height difference between the ith frame and the Nth frame, and At represents the interval time between the ith frame and the Nth frame;

[0029] (4) The centroid coordinates of the luggage and the passenger are calculated by formula (16):

[0030]

[0031] In the formula, (B x , B y ) represents the centroid coordinates of the luggage, and (C x , C y ) represents the centroid coordinates of the passenger;

[0032] The centroid distance T between the luggage and the passenger is obtained by formula (17):

[0033]

[0034] The centroid distance change rate AT between the luggage and the passenger b,t is:

[0035]

[0036] Where, AT represents the centroid distance change between the luggage and the passenger between the ith frame and the Nth frame.

[0037] Specifically, the step 4, the luggage falling state recognition model is:

[0038]

[0039] That is:

[0040] 1. From the t frame, the value of the external force F experienced by the luggage 外,t is 0;

[0041] 2. From the t frame, the rate of change of the center of mass distance ΔT between the luggage and the passenger b,t is greater than 0.5 m / s;

[0042] 3. From the t frame, the rate of change of the center of mass height ΔH of the luggage B,t is suddenly greater than 0.2 m / s;

[0043] The luggage falling state is detected, that is, whether the three determination conditions are met.

[0044] Further, the luggage falling state is detected, and when the three determination conditions are met, it is determined that the luggage and the passenger are separated at time t, the luggage falls at time t, and the loudspeaker is prompted to play and prompt pedestrians to pay attention and warn in time in a public place with dense crowd until the normal state is restored.

[0045] Otherwise, it is determined that the luggage motion state is normal.

[0046] Beneficial effects:

[0047] Unlike other recognition methods, the present application recognizes the motion characteristics of the luggage in the entire motion state, and can predict the subsequent motion state of the luggage. According to the recognition result, a warning can be issued in time to effectively avoid the harm caused by the luggage falling, which is of great significance to the safety management of public places with dense crowds. In addition, the present application can also be applied to the fields of logistics and freight transportation, thereby reducing the safety hazards and economic losses caused by the falling of luggage or goods. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The turning force between the passenger and the luggage motion in the normal walking state;

[0049] Figure 2 The straight line force between the passenger and the luggage motion in the normal walking state;

[0050] Figure 3 The size of the motion resultant force between the luggage motion in the normal walking state;

[0051] Figure 4 The luggage falling top view of the embodiment;

[0052] Figure 5 The motion relationship diagram of the passenger and the luggage of the embodiment;

[0053] Figure 6 The speed relationship diagram of the passenger and the luggage of the embodiment;

[0054] Figure 7 Embodiment Public place luggage falling state recognition and safety monitoring method flow chart. DETAILED DESCRIPTION

[0055] The technical solutions provided by the present application will be further described below in combination with specific embodiments and their accompanying drawings. The advantages and features of the present application will be more apparent in combination with the following description.

[0056] The theoretical research process of luggage falling state recognition considering combined motion includes the following steps:

[0057] Step one: dynamics analysis of the luggage

[0058] The single target tracking algorithm MMTrack is used in the present application to track the luggage in the collected video [2] The relevant motion information of the luggage in the video is extracted into a four-parameter tuple, and the motion trajectory of the luggage is extracted frame by frame into a limited four-parameter tuple:

[0059]

[0060] R i represents the motion data of the luggage sequence of the i-th frame of video, wherein i represents the frame number of the video, and j represents the luggage number, and respectively represent the horizontal coordinate and the vertical coordinate of the luggage j at the i-th frame of the video; and respectively represent the horizontal velocity and the vertical velocity of the luggage j at the i-th frame of the video. The magnitude of the combined velocity of the luggage can be represented by The speed and position of the passenger can also be obtained by tracking the passenger by MMTrack.

[0061] The angle θ i,j between the luggage j at the i-th frame of the video and the horizontal positive direction satisfies:

[0062]

[0063] The motion angle Δθ j of the luggage j from the i-th frame to the N-th frame is:

[0064]

[0065] The angular acceleration of the luggage j can be obtained from Δθ j :

[0066]

[0067] Wherein, Δt is the time required for the luggage j from the i frame to the N frame.

[0068] In the experimental calculation process, the luggage is regarded as a cube, L is the edge length of the cube. The mass of the luggage is evenly distributed, and the mass is m i Then the moment of inertia of the luggage j is:

[0069]

[0070] The steering external force (steering force) received by the luggage j is:

[0071]

[0072] Wherein, d j is the moment length of the luggage j, usually the sum of the length of the luggage and the pull rod.

[0073] The horizontal and vertical linear acceleration a x , a y of the luggage j from the i frame to the N frame is:

[0074]

[0075]

[0076]

[0077] Then the motion driving external force (linear force) of the luggage j is:

[0078] F 直 = m i a (10)

[0079] In the analysis process, the luggage and the passenger move in the x, y plane without height difference. The force is converted to the x, y plane. The external force received by the luggage comes from the steering force provided by the passenger, the friction of the ground, and the disturbance force of the surrounding passengers. Under normal circumstances, the luggage is only affected by the external force of the carrier, and the luggage is normally moved. The external force acting on the luggage is:

[0080]

[0081] Therefore, the external force received by the luggage is also the internal force between the luggage and the carrier. When the luggage and the passenger run stably, the system composed of the luggage and the carrier is a stable system, and the combined external force acting on the luggage is 0.

[0082] Step two: kinematic analysis of passenger and luggage

[0083] In normal state, the distance between the passenger and the large suitcase is about 0.9m. Therefore, the moment of the suitcase is also 0.9m, and the mass of the suitcase is 15kg. The turning force of the passenger and the suitcase in the movement process can be calculated by formula (6) as shown in the following formula. Figure 1

[0084] When the passenger and the suitcase are in normal movement state, |Fturn|≤1N, and the maximum turning force F 转max turn is defined as 1N. In straight walking state, the movement speed of the large suitcase is between 0.8-0.9m / s. According to formula (7)-(10), the force of the suitcase in straight movement can be calculated as shown in the following formula. Figure 2

[0085] It can be seen that when the suitcase is in normal straight movement, the straight force |F 直 | of the suitcase is less than or equal to 1N, which is close to 0. That is, when the suitcase is in normal movement, the straight force of the suitcase is close to 0. The suitcase swings with the pace of the passenger and moves at a uniform speed. The speed of normal passenger walking is between 1.1-2m / s, and the speed of running is between 2-10m / s. The instantaneous acceleration range is 0-5m / s 2 . The acceleration of the suitcase lags behind the acceleration of the passenger, and it is difficult to detect when the instantaneous acceleration is too small. Therefore, when the acceleration of the suitcase is greater than 1m / s 2 , it is considered that the suitcase is in accelerated movement. In the above example of the suitcase, the critical straight force F 直max of the suitcase is 15N.

[0086] According to the turning force and the straight force, the force of the suitcase in the movement process is calculated according to formula (11). When the suitcase moves at a uniform speed along a straight line, the size of the movement resultant force of the suitcase is shown in the following formula. Figure 3

[0087] By comparing Figure 3 and Figure 2 , Figure 1 it can be seen that: the force of the suitcase in the movement process is mainly affected by the turning force of the suitcase; and the turning force of the suitcase is mainly related to the moment provided by the shoulder joint of the passenger.

[0088] Step three: suitcase falling state recognition model

[0089] The suitcase falling state refers to the state that the suitcase is out of control of the passenger due to various reasons during the process of carrying the suitcase by the passenger. In the crowded public place, the suitcase falling will disturb the stability of the crowd. When the disturbance continues to accumulate, it will cause the abnormality of the whole public place system, thereby threatening the public safety. The suitcase falling state recognition model is designed in the present application.​​​

[0090] When the luggage is separated from its carrier, the distance between the luggage and the original carrier will gradually increase (manifested as a sudden break in the distance between the original carrier and the stable centroid carrier, and the motion state of the luggage will suddenly stop (i.e. the luggage is no longer subject to force). When the luggage falls off, the centroid height of the luggage will suddenly drop. The centroid height of the luggage is H B A luggage fall-off state recognition model is established, that is, the following conditions are met:

[0091]

[0092] That is: 1. From the t frame, the external force value F 外,t experienced by the luggage is 0;

[0093] 2. From the t frame, the centroid distance change rate AT b,t between the luggage and the passenger is greater than 0.5 m / s;

[0094] 3. From the t frame, the centroid height change rate AH B,t of the luggage is suddenly greater than 0.2 m / s.

[0095] For luggage detection, when the above three determination conditions are met, it is considered that the luggage and the passenger are separated at time t, and the luggage falls off at time t.

[0096] Step four: application of the luggage fall-off state recognition model

[0097] The motion state of the luggage is predicted and safety monitored, and a warning is issued in a timely manner according to the recognition result.

[0098] Example 1 and verification

[0099] The present application takes a 28-inch luggage as an example and performs corresponding experiments in Shanghai Metro Line 6. Two shooting methods, namely, overhead shooting and side shooting, are adopted. The side shooting height is 1.2 meters from the ground, and the camera is placed at the center of the test section. The passengers move along a straight line, and the camera shooting direction is tangent to the passenger movement direction. The overhead shooting position is 5 meters from the ground on the second floor of the subway, and the anti-shake function of the camera is turned on during shooting to maintain the stability of the camera. The scene of the luggage falling off suddenly is selected to verify and discuss the proposed luggage motion model and luggage fall-off state recognition model.

[0100] The motion process of 20 frames is selected, and the passenger trajectory is shown in Figure 5 . Then the motion trajectory and centroid change of the passenger and the luggage are shown in Figure 5 (a) and (b).

[0101] From Figure 5As can be seen from the figure, when the luggage falls off, the center of mass of the luggage will quickly drop to a certain height and maintain a stable value. The luggage will stop in place and the distance between the luggage and the passenger will become farther and farther away. According to Figure 5 (a) The trajectory of the center of mass can be calculated, and the initial motion speed of the luggage is about 0.82 m / s, and the motion speed of the passenger is 1.1 m / s. According to the formula where V0 is the motion speed of the passenger, V1 is the motion speed of the luggage, and β can be calculated i = 26%. Taking the weight of the passenger as 65 kg, the weight of the luggage is about 16.9 kg. The weight of the luggage used in the experiment is 18 kg, and the weight error is 6.1%, which is within the required error of 15%, meeting the requirements of the algorithm research.

[0102] As can be seen from Figure 6 , the motion speed of the passenger and the luggage remains basically the same at the beginning of the motion. From the 4th frame, the motion speed of the luggage decreases rapidly, and the change rate of the center of mass distance between the passenger and the luggage instantaneously increases, reaching 0.5 m / s near the 6th frame. According to formula (11), it can be calculated that the external force F 外 experienced by the luggage starts to be 0 near the 8th frame. According to Figure 5 (b), the change rate of the center of mass height of the luggage suddenly becomes greater than 0.2 m / s from the 4th frame. Combining the above three conditions, that is, the luggage satisfies the falling state recognition model formula (12) from the 8th frame, the luggage and the passenger have separated, and the luggage falls off at this time. Therefore, the luggage falling state recognition model proposed by the present application is effective.

[0103] Embodiment 2 and a luggage falling state recognition and safety monitoring method in a public place

[0104] As shown in Figure 7 , a luggage falling state recognition and safety monitoring method in a public place includes the following steps:

[0105] Step 1, a monitoring camera, a loudspeaker prompting device and the like are arranged in a public place where people flow is dense, and the monitoring camera is connected to a monitoring processor in a control room;

[0106] Step 2, the monitoring camera is turned on, the photographing direction of the monitoring camera is tangent to the motion direction of the passenger, video frames are collected at a certain frequency, and video data are provided to the monitoring processor;

[0107] Step 3, the monitoring processor calculates the external force value F [3] , the center of mass height H 外,t , and the change rate of the center of mass height ΔH B of the luggage in the monitoring place by image processing algorithms MMTrack, findContours B,t, the rate of change of the center of mass distance between the luggage and the passenger ΔT b,t , the specific implementation is:

[0108] (1) For the video frames collected by the camera, the luggage motion information is extracted to a four-tuple R i by MMTrack 外,t ;

[0109] (2) The contour pictures of the luggage and the passenger are extracted by findContours, and are converted into gray scale pictures;

[0110] (3) The origin moment of the gray scale picture can be calculated by formula (13):

[0111]

[0112] Where, M pq represents the origin moment, p and q represent the order of the moment, x and y represent the pixel coordinates in the image, and I(x,y) represents the pixel gray value at the coordinates (x,y).

[0113] The zero-order moment M B,00 and the first-order moment M B,01 , M B,10 of the luggage gray scale picture, and the zero-order moment M C,00 and the first-order moment M C,01 , M C,10 of the passenger gray scale picture are calculated by formula (13) respectively.

[0114] Then the center of mass height H B of the luggage can be obtained by formula (14):

[0115]

[0116] Then the center of mass height change rate ΔH B,t of the luggage is:

[0117]

[0118] Where, ΔH B represents the center of mass height difference of the luggage between the i-th frame and the N-th frame, and Δt represents the interval time between the i-th frame and the N-th frame.

[0119] (4) The center of mass coordinates of the luggage and the passenger are calculated by formula (16):

[0120]

[0121] In the formula, (B x ,B y ) represents the center of mass coordinates of the luggage, and (Cx ,C y ) represents the center of mass coordinates of the passenger.

[0122] The center of mass distance T between the luggage and the passenger can be obtained by formula (17):

[0123]

[0124] The center of mass distance change rate ΔT between the luggage and the passenger can be obtained by formula (18): b,t

[0125]

[0126] Where ΔT represents the change amount of the center of mass distance between the luggage and the passenger between the ith frame and the Nth frame.

[0127] Step 4, detecting the luggage falling state based on the luggage falling state recognition model. Specifically, it is judged whether the following conditions are met:

[0128]

[0129] That is: 1. From the tth frame, the external force value F experienced by the luggage is 0; 外,t

[0130] 2. From the tth frame, the center of mass distance change rate ΔT between the luggage and the passenger is greater than 0.5 m / s; b,t

[0131] 3. From the tth frame, the center of mass height change rate ΔH of the luggage is suddenly greater than 0.2 m / s; B,t

[0132] When the luggage falling state is detected, when the above three judgment conditions are met, it can be determined that the luggage and the passenger are separated at time t, the luggage falls at time t, and the loudspeaker is prompted to play and prompt pedestrians to pay attention and warn in time in public places with dense crowd, until the normal state is restored;

[0133] Otherwise, it is determined that the luggage motion state is normal.

[0134] The above description is only a description of the preferred embodiments of the present application, and is not any limitation on the scope of the present application. Any modification or modification made by any person skilled in the art according to the above disclosed technical content shall be regarded as an equivalent effective embodiment, and shall fall within the scope of the technical scheme protected by the present application.

[0135] REFERENCES

[0136] ​​​​[1] Kitazono Y, Ihara I, Yoshida G, et al. Selective degradation of tetracycline antibiotics present in raw milk by electrochemical method. [J]. Journal of Hazardous Materials, 2012, 243(DEC.): 112-116.

[0137] [2] DONG Da-heng. Intelligent analysis of crowd evacuation stability based on video [D]. Tongji University, 2019.

[0138] [3] MAO Xing-yun. OpenCV Programming for Beginners [M]. Electronic Industry Press, 2015: 66-68.

Claims

1. A public place luggage off state recognition and safety monitoring method, characterized in that, The method comprises the steps of: Step 1, setting up a monitoring camera and a loudspeaker prompt device in a public place with dense passenger flow, and connecting the monitoring camera with a monitoring processor in a control room; Step 2, starting the monitoring camera, setting the photographing direction of the monitoring camera to be tangent to the passenger movement direction, collecting video frames, and providing video data to the monitoring processor; Step 3, the monitoring processor calculates the external force value of the luggage in the monitoring place through image processing algorithm MMTrack, findContours respectively , the centroid height , and the centroid height change rate , the centroid distance change rate between the luggage and the passenger ; Step 4, detecting the luggage falling state based on the luggage falling state recognition model; The step 3 is implemented as: Step 3.1, extracting the motion information of the luggage to a four-tuple by MMTrack for the video frames captured by the camera , calculating the external force on the luggage by equations (2)-(11) ; Step 3.2, extracting the contour picture of the luggage through findContours, and converting the contour picture into a gray-scale picture; Step 3.3, calculating the origin point of the gray-scale picture through formula (13): （13） wherein, represents the origin moment, p and q represent the order of the moment, x and y represent the pixel coordinates in the image, and I(x, y) represents the pixel intensity value at the coordinates (x, y); The zeroth and first moments of the luggage gray scale image are calculated by equation (13) and equation (14) , respectively. The zeroth and first moments of the passenger gray scale image are calculated by equation (13) and equation (14) , respectively. the height of the center of mass of the luggage may be obtained by equation (14) （14） the rate of change of the height of the center of mass of the luggage is: （15） wherein, represents the difference in height of the center of mass of the luggage between the i-th frame and the N-th frame, represents the time interval between the i-th frame and the N-th frame; Step 3.4, calculating the centroid coordinates of the luggage and the passenger through formula (16): （16） wherein denotes the center of mass coordinates of the luggage, denotes the center of mass coordinates of the passenger; Then the centroid distance T between the luggage and the passenger is obtained through formula (17): (17) the rate of change of the center of mass distance between the luggage and the passenger is: (18) wherein, represents the change in the distance between the center of mass of the luggage and the passenger between the i-th frame and the N-th frame; In the step 3.1, The four-tuple Is: （1） motion data of the luggage sequence in the i-th frame of the video, where i represents the frame number of the video, and j represents the luggage number, and respectively represent the horizontal coordinate and the vertical coordinate of the luggage j in the i-th frame of the video; and respectively represent the horizontal velocity and the vertical velocity of the luggage j in the i-th frame of the video; the speed magnitude of the luggage is represented by ; the speed and the position of the passenger are obtained by tracking the passenger by MMTrack; j angle of the i-th frame video with the horizontal positive direction satisfies: （2） then the motion angle of the luggage j from the i-th frame to the N-th frame is: （3） From The angular acceleration of the suitcase j is found to be: （4） wherein, is the time needed for the suitcase j to move from the i-th frame to the N-th frame; In the experimental calculation process, the luggage is regarded as a cube, L is the side length of the cube, and the mass of the luggage is evenly distributed, and the mass is The moment of inertia of the luggage j is: （5） The turning force received by the luggage j is: （6） wherein, is the moment length of the suitcase j, specifically the sum of the length of the suitcase and the length of the handle. horizontal and vertical linear acceleration of the luggage j from the i-th frame to the N-th frame , , the resultant acceleration a is respectively: （7） （8） （9） Then the motion driving external force of the luggage j, i.e. the linear force, is: （10） Under the normal motion condition of the luggage, the external force received by the luggage is: （11）。 2. The method of claim 1, wherein, The step 4 is a luggage falling state recognition model: （12） That is: The value of the external force received by the luggage from the t-th frame is 0; Rate of change of the center of mass distance between the luggage and the passenger from the tth frame greater than 0.5 m / s; from the t-th frame, the rate of change of the height of the center of mass of the luggage abruptly greater than 0.2 m / s; Detecting the luggage falling state, i.e. judging whether the luggage falling state recognition model is satisfied.

3. The method of claim 2, wherein, The step 4, Detecting the luggage falling state, when the three judgment conditions of the luggage falling state recognition model are satisfied, it is judged that the luggage and the passenger are separated at the time t, the luggage falls at the time t, and the loudspeaker prompt device is used to timely play and prompt the pedestrians to pay attention and give early warning in the public place with dense passenger flow until the normal state is restored; Otherwise, it is judged that the motion state of the luggage is normal.

Images