Escalator monitoring and early warning method and device based on target motion change

Abstract

The invention relates to an escalator monitoring and early warning method and device based on target motion changes. The method comprises the steps that bounding boxes, posture information and category information of all targets in a video stream are obtained; respectively acquiring image coordinate positions of the video stream targets; world coordinates of all the targets are obtained based on the camera and the image coordinate positions; first early warning information is sent based on bounding box changes of the same target in adjacent frames in the video stream; motion information of the target is acquired, and second early warning information is sent based on the motion information of the target; judging the relevancy between the passenger and the carried object, and sending third early warning information based on the relevancy between the passenger and the carried object; and performing an early warning action according to the first early warning information, the second early warning information or the third early warning information. According to the method, the target motion state on the escalator is obtained through the image information, differentiated early warning is performed based on the target motion state, automatic recognition and timely early warning of unforeseen circumstances on the escalator are realized, the escalator monitoring efficiency is improved, and the escalator operation safety is ensured.

Description

[0001] This application is a divisional application of a patent application with an application date of September 10, 2024, an application number of 202411260742.8, and an invention name of "A method and device for monitoring and early warning of escalator passenger accidents based on visual detection". Technical Field

[0002] The present application relates to the technical field of escalator safety monitoring, and in particular to an escalator monitoring and early warning method and device based on target motion changes. Background Art

[0003] As a common means of transportation in public places, escalators are becoming more and more common. Along with this, some safety issues and smooth operation issues of escalators are attracting more and more attention. At present, cameras are installed at the entrances and exits of escalators to observe the status of passengers or objects on the escalator, such as whether the passenger flow at the escalator entrance is congested, whether passengers have fallen, and whether large items are stranded or slipped. If the above accidents occur during the operation of the escalator, it may cause panic or injury to the passengers. At present, the accident monitoring of escalators mainly relies on integrated mechanical sensors, infrared sensors or visual monitoring, but these methods often need to be integrated and coupled with the escalator system, and the cost is uncontrollable. In addition, visual monitoring still relies on manual warnings, and the degree of automation is not high. Once manual participation is not timely, the degree of injury will be aggravated. Summary of the invention

[0004] In order to overcome the above-mentioned deficiencies of the prior art, the present application provides an escalator monitoring and early warning method and device based on target motion changes, which specifically adopts the following technical solutions:

[0005] An escalator monitoring and early warning method based on target motion changes comprises the following steps:

[0006] S1. Obtaining the bounding box, posture information and category information of each target in the video stream based on an image recognition algorithm; the category information of the target includes at least passengers and carried objects;

[0007] S2, obtaining the image coordinate position of the same target in the video stream based on the target tracking algorithm;

[0008] S3, obtaining the world coordinates of each target based on the camera with calibrated internal and external parameters and in combination with the image coordinate position of the target;

[0009] S4. issuing a first warning message based on the change of the bounding box of the same target in adjacent frames in the video stream:

[0010] S5, acquiring motion information of the corresponding target based on the change of the world coordinate position of each target in the video stream, and issuing a second warning information based on the target motion information;

[0011] When obtaining the motion information of the corresponding target, the world coordinates of the same target in adjacent video frames are used to calculate the motion speed of the target in the world coordinate system, and the calculation steps include:

[0012] Get the world coordinates of target j in video frame i and video frame i-1 respectively;

[0013] Calculate the linear motion distance of target j in adjacent video frames based on the world coordinates of target j in video frame i and video frame i-1:

[0014] D i,i-1 =F(P i ,P i-1 );

[0015] Where D i,i-1 is the linear motion distance of the target in video frame i and video frame i-1; F() is the calculation function of the linear distance between two points in the world coordinate system; P i is the world coordinate of target j in video frame i; P i-1 is the world coordinate of target j in video frame i-1;

[0016] Based on the linear motion distance D of target j in adjacent video frames i,i-1 And the frame interval time t is used to calculate the moving speed of target j in the world coordinate system;

[0017] S6. Determine the correlation between the passenger and the belongings based on the posture information and world coordinates of the passenger and the belongings, and issue a third warning message based on the correlation between the passenger and the belongings;

[0018] S7. Perform a warning action according to the first warning information, the second warning information or the third warning information, wherein the warning action at least includes a control room alarm, an escalator alarm or an escalator stop.

[0019] Optional: When obtaining the world coordinates of each target based on the camera with calibrated internal and external parameters combined with the image coordinate position of the target, the coordinate conversion method used is:

[0020]

[0021] Obtain the homogeneous equation corresponding to target j based on the coordinate transformation method of target j;

[0022]

[0023] With z j As a scaling factor for homogeneous coordinates, we solve for:

[0024]

[0025] The world coordinates of the target are:

[0026]

[0027] where u j and v j is the image coordinate position of target j in the video frame, u j is the column position of the target j pixel in the video frame, v j is the horizontal column position of the target j pixel in the video frame; K is the intrinsic parameter matrix of the camera, K -1 is the inverse matrix of the camera's intrinsic parameter matrix; R is the rotation matrix; t is the translation vector; X j , Y j and Z j is the world coordinate of target j, X j is the X-direction world coordinate of target j, Y j is the Y world coordinate of target j, Z j is the Z-direction world coordinate of target j, Z known is the known Z-direction world coordinate value of target j; x j ,y j and z j is the homogeneous coordinate of target j, that is, x j is the X-direction world coordinate X of target j j The homogeneous coordinates, y j is the Y-direction world coordinate Y of target j j The homogeneous coordinates of j is the Z-direction world coordinate Z of target j j The homogeneous coordinates of .

[0028] Optionally, the step of obtaining the first warning information based on the change of the bounding box of the same target in adjacent frames in the video stream includes:

[0029] Get at least one set of diagonal coordinates (A) of the bounding box of the target in video frame i i,cor ,B i,cor ) and (a i,cor ,b i,cor ); and calculate the bounding box area of ​​the target in video frame i according to the diagonal coordinates of the bounding box of the target in video frame i;

[0030] Get at least one set of diagonal coordinates (A) of the bounding box of the target in video frame i-1 i-1,cor ,B i-1,cor ) and (a i-1,cor ,b i-1,cor ), and calculate the bounding box area of ​​the target in the video frame i-1 according to the diagonal coordinates of the bounding box of the target in the video frame i-1;

[0031] Obtain the change rate of the bounding box area of the target based on the bounding box areas of the target in adjacent video frames:

[0032]

[0033] where ΔR Area is the change rate of the bounding box area of the target; Area i is the bounding box area of the target in video frame i; Area i-1 is the bounding box area of the target in video frame i - 1; Δt is the time interval between video frame i and video frame i - 1;

[0034] Judge the magnitude of the change rate ΔR of the target bounding box area in adjacent video frames Area and the preset threshold R0. If ΔR Area ≥R0, it indicates that the target on the escalator has fallen, and at this time, a first warning message is sent; if ΔR Area <R0, it indicates that there is no fall situation of the target on the escalator, and at this time, no first warning message is sent.

[0035] Optionally: The step of sending a second warning message based on the target motion information includes:

[0036] Successively obtain the motion speeds of target j between adjacent video frames;

[0037] Obtain the motion trend of target j based on the change in the motion speed of target j: If the difference in the motion speeds of target j at adjacent times is greater than or equal to the preset threshold, it is judged that the target has slipped on the escalator, and at this time, a second warning message is sent; if the difference in the motion speeds of target j at adjacent times is less than the preset threshold, it is judged that the target has not slipped on the escalator, and at this time, no second warning message is sent.

[0038] Optionally: The step of sending a second warning message based on the target motion information includes:

[0039] Successively obtain the motion speeds of target j between adjacent video frames and obtain the operating state of the escalator;

[0040] If the escalator operating state is in a stopped state, judge that the motion speed of target j is 0 or the residence time on the escalator exceeds 1 minute, then judge that the target is detained on the escalator, and at this time, a second warning message is sent;

[0041] If the escalator operating state is in a normal operating state, judge that the motion speed of target j is less than the normal operating speed of the escalator or the motion direction of target j is inconsistent with the normal operating direction of the escalator, then judge that the target is going against the flow on the escalator, and at this time, a second warning message is sent.

[0042] Optionally: The step of judging the relevance between the passenger and the carried object based on the posture information and world coordinates of the passenger and the carried object includes:

[0043] Get the world coordinates of passengers and objects respectively;

[0044] Taking the coordinates of the luggage as the center, obtain the straight-line distance between different passengers and the same luggage;

[0045] Setting the maximum safe distance for the luggage, which is the maximum distance range within which the luggage can be controlled by the passenger, and determining whether the straight-line distance between different passengers and the same luggage exceeds the maximum safe distance;

[0046] Selecting passengers whose straight-line distance from the same object does not exceed the maximum safe distance, and determining posture information of the selected passengers, the posture information including the frequency of contact between the passenger and the object and the duration of contact between the passenger and the object;

[0047] Based on the passenger's posture information, it is determined whether there is one or more passengers who meet the characteristics of the owner of the belongings within the maximum safe distance of the belongings. If so, it is determined that the passenger and the belongings are strongly correlated; if not, it is determined that the passenger and the belongings are weakly correlated.

[0048] Optionally, the step of issuing a third warning message based on the correlation between the passenger and the carried items includes:

[0049] When it is determined that the passenger and the carried item are strongly correlated, the carried item is under the control of at least one passenger, and there is no risk of the carried item being separated, and the third warning information is not issued at this time;

[0050] When it is determined that the passenger and the carried items are weakly correlated, the carried items are not under the control of the passenger and there is a risk of the carried items being out of the passenger's control, and at this time, a third warning message is issued.

[0051] Furthermore, the present application also discloses an escalator monitoring and early warning device based on target motion changes, the device comprising:

[0052] An image recognition module, used to obtain the bounding box, posture information and category information of each target in the video stream based on an image recognition algorithm; the category information of the target includes at least passengers and carried objects;

[0053] A target tracking module, used to obtain the image coordinate position of the same target in the video stream based on the target tracking algorithm;

[0054] A coordinate conversion module is used to obtain the world coordinates of each target based on the camera with calibrated internal and external parameters and in combination with the image coordinate position of the target;

[0055] A first warning module, configured to issue a first warning message based on a change in a bounding box of the same object in adjacent frames in a video stream;

[0056] A second warning module is used to obtain motion information of the corresponding target based on the change of the world coordinate position of each target in the video stream, and issue a second warning information based on the target motion information;

[0057] When obtaining the motion information of the corresponding target, the world coordinates of the same target in adjacent video frames are used to calculate the motion speed of the target in the world coordinate system, and the calculation steps include:

[0058] Get the world coordinates of target j in video frame i and video frame i-1 respectively;

[0059] Calculate the linear motion distance of target j in adjacent video frames based on the world coordinates of target j in video frame i and video frame i-1:

[0060] D i,i-1 =F(P i ,P i-1 );

[0061] Where D i,i-1 is the linear motion distance of the target in video frame i and video frame i-1; F() is the calculation function of the linear distance between two points in the world coordinate system; P i is the world coordinate of target j in video frame i; P i-1 is the world coordinate of target j in video frame i-1;

[0062] Based on the linear motion distance D of target j in adjacent video frames i,i-1 And the frame interval time t is used to calculate the moving speed of target j in the world coordinate system;

[0063] A third warning module, used to determine the relevance between the passenger and the belongings based on the passenger's posture information and the coordinate positions of the belongings, and issue a third warning message based on the relevance between the passenger and the belongings;

[0064] The warning execution module is used to perform warning actions according to the first warning information, the second warning information or the third warning information, and the warning actions at least include control room alarm, escalator alarm or escalator stop.

[0065] Beneficial Effects

[0066] The technical solution of this application has the following beneficial effects:

[0067] (1) The escalator monitoring and early warning method of the present application can obtain the target state on the escalator by using the image information collected by the camera, and distinguish and warn based on the target state, so as to realize automatic identification of passengers falling, objects being stranded or slipping, crowded conditions, etc. on the escalator, and timely warning, thereby improving the escalator monitoring efficiency and ensuring the safety of escalator operation. In addition, the method can perform corresponding early warning actions based on different early warning information, such as control room alarm, escalator alarm or escalator stop, etc., to realize intelligent monitoring.

[0068] (2) The escalator monitoring and early warning method of the present application utilizes existing surveillance cameras to monitor whether the escalator is operating abnormally when carrying people or goods, and does not require additional hardware integration, thereby greatly reducing equipment costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a flow chart of the escalator monitoring and early warning method in an embodiment of the present application.

[0070] Figure 2 It is a structural diagram of the escalator monitoring and early warning device in the embodiment of the present application.

[0071] Figure 3 This is a structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0072] The present application is further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and cannot be used to limit the scope of protection of the present application. It should be noted that the following detailed descriptions are all exemplary and are intended to provide further explanation of the present application.

[0073] At present, the common way to monitor escalators is to use visual sensors, such as cameras, by setting up multiple cameras at the entrance, exit and middle of the escalator to collect image data of passengers or objects on the escalator, and then transmit the image data to the central control room to manually monitor the status of each escalator. However, this method relies on manual accident identification and judgment. Once the control personnel are negligent and fail to find a certain monitoring image in time, serious accidents will occur. Therefore, this application is based on the image data obtained by the original camera for further processing, and part or all of the data that needs to be manually identified is converted to a computer for automatic processing. Without increasing the cost of equipment, the image data is processed by the computer's internal algorithm, escalator accidents are automatically identified, and warnings are issued to the central control or escalator passengers.

[0074] Combination Figure 1 As shown, the embodiment of the present application specifically discloses a method for monitoring and warning of escalator passenger accidents based on visual detection, which includes the following steps:

[0075] Step 1: Based on the image recognition algorithm, the bounding box, posture information and category information of each target in the video stream are obtained respectively; the category information of the target includes at least passengers and carrying objects; preferably, the embodiment of the present application adopts the NanoDet lightweight target detection network to detect targets on the escalator in real time to ensure fast and accurate target recognition.

[0076] Step 2: Based on the target tracking algorithm, the image coordinate positions of the same target in the video stream are obtained respectively; the embodiment of the present application preferably adopts the BoTSORT algorithm to continuously track the same target in the video stream, and obtains the coordinate positions of the same target in different video frames respectively.

[0077] Step 3: Based on the camera with calibrated internal and external parameters and combined with the image coordinate position of the target, the world coordinates of each target are obtained:

[0078] Specifically, in the step 3, when obtaining the world coordinates of each target based on the camera with calibrated internal and external parameters combined with the image coordinate position of the target, the coordinate conversion method used is:

[0079]

[0080] Obtain the homogeneous equation corresponding to target j based on the coordinate transformation method of target j;

[0081]

[0082] With z j As a scaling factor for homogeneous coordinates, we solve for:

[0083]

[0084] The world coordinates of the target are:

[0085]

[0086] where u j and v j is the image coordinate position of target j in the video frame, u j is the column position of the target j pixel in the video frame, v j is the horizontal column position of the target j pixel in the video frame; K is the intrinsic parameter matrix of the camera, K -1 is the inverse matrix of the camera's intrinsic parameter matrix; R is the rotation matrix; t is the translation vector; X j , Y j and Z j is the world coordinate of target j, X j is the X-direction world coordinate of target j, Y j is the Y world coordinate of target j, Z j is the Z-direction world coordinate of target j, Zknown is the known Z-direction world coordinate value of target j; x j ,y j and z j is the homogeneous coordinate of target j, that is, x j is the X-direction world coordinate X of target j j The homogeneous coordinates, y j is the Y-direction world coordinate Y of target j j The homogeneous coordinates of j is the Z-direction world coordinate Z of target j j The homogeneous coordinates of .

[0087] It should be noted that in the process of calculating the world coordinates of the target in step three, the longitudinal coordinate of the target, that is, the Z-direction world coordinate value is a known constant. Since the height of the escalator in the longitudinal direction is fixed, by determining the longitudinal position (Z direction) of the target on the escalator in advance, the coordinates in other directions can be obtained by reverse calculation based on the longitudinal position (Z direction).

[0088] Step 4: Issue a first warning message based on the change of the bounding box of the same target in adjacent frames in the video stream.

[0089] Specifically, the specific process of obtaining the first warning information based on the change of the bounding box of the same target in adjacent frames in the video stream in step 4 includes:

[0090] First, obtain at least one set of diagonal coordinates (A) of the bounding box of the target in video frame i. i,cor ,B i,cor ) and (a i,cor ,b i,cor ); and calculate the bounding box area of ​​the target in video frame i according to the diagonal coordinates of the bounding box of the target in video frame i;

[0091] Then, at least one set of diagonal coordinates (A) of the bounding box of the target in the video frame i-1 is obtained. i-1,cor ,B i-1,cor ) and (a i-1,cor ,b i-1,cor ), and calculate the bounding box area of ​​the target in the video frame i-1 according to the diagonal coordinates of the bounding box of the target in the video frame i-1;

[0092] The change rate of the target's bounding box area is obtained again based on the target's bounding box area in adjacent video frames:

[0093]

[0094] Where ΔR Area Area is the rate of change of the target's bounding box area; i is the bounding box area of ​​the target in video frame i; Area i-1is the bounding box area of the target in video frame i - 1; Δt is the time interval between video frame i and video frame i - 1;

[0095] Judge the change rate ΔR of the target bounding box area in adjacent video frames Area and the preset threshold R0. If ΔR Area ≥R0, it indicates that the target on the escalator has fallen, and at this time, a first warning message is sent; if ΔR Area <R0, it indicates that there is no fall of the target on the escalator, and at this time, the first warning message is not sent.

[0096] It should be noted that in step four, the change rate of the target bounding box is used to determine whether the target has fallen on the escalator. Generally, when ordinary passengers or carried objects are on the escalator, they are in a state of standing still or moving at a constant speed. At this time, the size of the bounding box of the target recognized by the image recognition algorithm does not change much, that is, the change of the area of the target bounding box fluctuates little or does not fluctuate; once a passenger or carried object has a fall accident, the area of the target bounding box will change greatly in a short time. Therefore, by monitoring the change rate of the target bounding box area, it can be judged whether a fall accident has occurred to the target on the escalator, and then the first warning message can be sent.

[0097] Step five: Obtain the motion information of the corresponding target based on the change of the world coordinate position of each target in the video stream, and send a second warning message based on the target motion information.

[0098] As an implementation, in step five, the Kalman filter algorithm is used to calculate the motion speed of the target in the vertical direction, and its calculation steps include:

[0099] Determine the initial state vector of target j. The state vector of target j includes the target position P k,j and the target speed V k,j , and establish a state transition equation based on the initial state vector of target j:

[0100]

[0101] Establish an observation equation based on the initial state vector of target j:

[0102]

[0103] where X k+1,j is the state vector of target j at time k + 1; X k,j is the state vector of target j at time k; Z k,j is the observation vector of target j at time k; A is the state transition matrix; H is the observation matrix; ω k,j is the process noise; υ k,j is the observation noise; ΔT is the state transition time interval of target j;

[0104] Predict the state vector of the target at the next moment based on the target's state transfer equation, and calculate the state vector of the target at the current moment based on the target's observation equation;

[0105] Using the state vector at the current moment to correct the predicted state vector at the next moment, to obtain a corrected state vector;

[0106] The target speed of the target at the next moment is obtained based on the corrected state vector.

[0107] Based on the above method, it is only necessary to analyze the change in the target's moving speed in the longitudinal direction to determine whether the current target is in a sliding state on the escalator. When the target is stationary or walking at a constant speed, the target speed obtained by adjacent video frames will not change significantly within a certain period of time; when the target has a sliding accident, the target speed will change significantly in a short period of time. Therefore, by analyzing the change in the target speed, it is possible to accurately determine whether the target on the escalator has a sliding accident. Once a sliding accident occurs, a second warning message is issued in time to remind the central control personnel or passengers on the escalator.

[0108] As another feasible implementation, in step 5, the world coordinates of the same target in adjacent video frames may be used to calculate the moving speed of the target in the world coordinate system, and the calculation steps include:

[0109] Get the world coordinates of target j in video frame i and video frame i-1 respectively;

[0110] Calculate the linear motion distance of target j in adjacent video frames based on the world coordinates of target j in video frame i and video frame i-1:

[0111] D i,i-1 =F(P i ,P i-1 );

[0112] Where D i,i-1 is the linear motion distance of the target in video frame i and video frame i-1; F() is the calculation function of the linear distance between two points in the world coordinate system; P i is the world coordinate of target j in video frame i; P i-1 is the world coordinate of target j in video frame i-1;

[0113] Based on the linear motion distance D of target j in adjacent video frames i,i-1 And the frame interval time t is used to calculate the movement speed of target j in the world coordinate system.

[0114] After obtaining the world coordinates of the same target in each video frame, the coordinate trajectory of the corresponding target in the real space is determined. Based on the world coordinates of the target in the video frame, the average movement speed of the target between adjacent video frames can be calculated. By analyzing the change in the average movement speed of the target in a certain number of video frames, for example, when the target is stationary or walking at a constant speed, the average movement speed of the target obtained by adjacent video frames will not change significantly within a certain period of time; when the target slips, the average movement speed of the target will change significantly in a short period of time. Therefore, by analyzing the change in the average movement speed, it is possible to accurately determine whether the target on the escalator has slipped. Once a slip occurs, a second warning message is issued in time to remind the personnel in the control center or the passengers on the escalator.

[0115] Furthermore, the specific steps of issuing the second warning information based on the target motion information in step 5 include:

[0116] Sequentially obtain the motion speed of target j between adjacent video frames;

[0117] The movement trend of target j is obtained based on the change in the movement speed of target j: if the difference in the movement speed of target j at adjacent moments is greater than or equal to the preset threshold, it is judged that the target has slipped on the escalator, and a second warning message is issued at this time; if the difference in the movement speed of target j at adjacent moments is less than the preset threshold, it is judged that the target has not slipped on the escalator, and no second warning message is issued at this time.

[0118] In order to further avoid misjudgment caused by human running or stopping on the escalator in the embodiment of the present application, a specific threshold can be pre-set when analyzing the target movement trend. When analyzing the change in the difference in the movement speed of the target at adjacent moments, generally when a person runs or stops, the movement speed change will have a certain time buffer (that is, the target speed does not suddenly rise or fall in a short period of time, but slowly rises or falls after a period of time), which is manifested in the small movement speed difference at adjacent moments. By setting a specific threshold, human factors can be excluded, and the situation where the movement speed changes due to unexpected factors is retained as much as possible, so as to correctly warn of escalator accidents.

[0119] Furthermore, the step of issuing the second warning information based on the target motion information in step 5 further includes:

[0120] Sequentially obtain the moving speed of target j between adjacent video frames and obtain the running status of the escalator;

[0121] If the escalator is in the stopped state, and the target j's moving speed is judged to be 0 or the target j's staying time on the escalator exceeds 1 minute, the target is judged to be stuck on the escalator, and a second warning message is issued;

[0122] If the escalator is in normal operation, if the target j's moving speed is less than the normal operating speed or the target j's moving direction is inconsistent with the normal operating direction, then the target is judged to be moving in the reverse direction on the escalator, and a second warning message is issued.

[0123] The above method can be used to judge the target that is stranded on the escalator or the target that is moving in the reverse direction, so as to increase the types of escalator accident warnings of the method, expand the warning range, and improve the warning efficiency.

[0124] Step 6: determining the correlation between the passenger and the belongings based on the posture information and world coordinates of the passenger and the belongings, and issuing a third warning message based on the correlation between the passenger and the belongings;

[0125] Specifically, the purpose of determining the correlation between the passenger and the object based on the posture information and world coordinates of the passenger and the object in step 6 is to further avoid the potential risk of objects on the escalator getting out of control and to issue an early warning for objects with a risk of getting out of control. The specific steps include:

[0126] According to the process of steps 1 to 3, the world coordinates of the passenger and the carried objects can be obtained respectively; it should be noted that at this time, the world coordinates of the passenger and the carried objects can use the center coordinates of the bounding box as the world coordinates of the corresponding target, ignoring the difference in volume.

[0127] At this time, taking the coordinates of each carried object as the center, the straight-line distances between different passengers and the same carried object can be obtained respectively, and the straight-line distances can be obtained from the world coordinates of the passenger and the carried object;

[0128] Setting the maximum safe distance for the luggage, which is the maximum distance range within which the luggage can be controlled by the passenger, and determining whether the straight-line distance between different passengers and the same luggage exceeds the maximum safe distance;

[0129] Selecting passengers whose straight-line distance from the same object does not exceed the maximum safe distance, and determining posture information of the selected passengers, the posture information including the frequency of contact between the passenger and the object and the duration of contact between the passenger and the object;

[0130] Based on the passenger's posture information, it is determined whether there is one or more passengers who meet the characteristics of the owner of the belongings within the maximum safe distance of the belongings. If so, it is determined that the passenger and the belongings are strongly correlated; if not, it is determined that the passenger and the belongings are weakly correlated.

[0131] Further, when it is judged that the passenger and the carried object are strongly correlated, the carried object is under the control of at least one passenger, and there is no risk of the carried object being separated, and the third warning information is not issued at this time; when it is judged that the passenger and the carried object are weakly correlated, the carried object is not under the control of the passenger, and there is a risk of the carried object being separated, and the third warning information is issued at this time.

[0132] Step 7: Perform an early warning action according to the first early warning information, the second early warning information or the third early warning information, wherein the early warning action at least includes a control room alarm, an escalator alarm or an escalator stop. It should be noted that the early warning actions executed by different early warning information can be automatically executed according to the pre-setting of the central control room personnel, or manually controlled by the central control room personnel.

[0133] In addition, combined Figure 2 As shown, the present application also discloses a method and device for monitoring and warning escalator passenger accidents based on visual detection, the device comprising:

[0134] An image recognition module, used to obtain the bounding box, posture information and category information of each target in the video stream based on an image recognition algorithm; the category information of the target includes at least passengers and carried objects;

[0135] A target tracking module, used to obtain the image coordinate position of the same target in the video stream based on the target tracking algorithm;

[0136] The coordinate conversion module is used to obtain the world coordinates of each target based on the camera with calibrated internal and external parameters and the image coordinate position of the target:

[0137] A first warning module, configured to issue a first warning message based on a change in a bounding box of the same object in adjacent frames in a video stream;

[0138] A second warning module is used to obtain motion information of the corresponding target based on the change of the world coordinate position of each target in the video stream, and issue a second warning information based on the target motion information;

[0139] A third warning module, used to determine the relevance between the passenger and the belongings based on the passenger's posture information and the coordinate position of the belongings, and issue a third warning message based on the relevance between the passenger and the belongings;

[0140] The warning execution module is used to perform warning actions according to the first warning information, the second warning information or the third warning information, and the warning actions at least include control room alarm, escalator alarm or escalator stop.

[0141] The device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment will not be described again here.

[0142] like Figure 3 As shown, the embodiment of the present application also provides an electronic device, including a processor and a memory, a program or instruction stored in the memory and executable on the processor, and when the program or instruction is executed by the processor, the following is achieved: Figure 1 The various processes of the method embodiment shown in the figure can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0143] The embodiment of the present application also provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the above Figure 1 The various processes of the method embodiments described above can achieve the same technical effect, and will not be described again here to avoid repetition.

[0144] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiments described above can achieve the same technical effect, and will not be described again here to avoid repetition.

[0145] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0146] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0147] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another device, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0148] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0149] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0150] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0151] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a device (which can be a terminal or a platform, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0152] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An escalator monitoring and early warning method based on target motion changes, characterized in that: The steps include: S1. Obtaining the bounding box, posture information and category information of each target in the video stream based on an image recognition algorithm; the category information of the target includes at least passengers and carried objects; S2, obtaining the image coordinate position of the same target in the video stream based on the target tracking algorithm; S3, obtaining the world coordinates of each target based on the camera with calibrated internal and external parameters and in combination with the image coordinate position of the target; S4. issuing a first warning message based on the change of the bounding box of the same target in adjacent frames in the video stream: S5, acquiring motion information of the corresponding target based on the change of the world coordinate position of each target in the video stream, and issuing a second warning information based on the target motion information; When obtaining the motion information of the corresponding target, the world coordinates of the same target in adjacent video frames are used to calculate the motion speed of the target in the world coordinate system, and the calculation steps include: Get the world coordinates of target j in video frame i and video frame i-1 respectively; Calculate the linear motion distance of target j in adjacent video frames based on the world coordinates of target j in video frame i and video frame i-1: D i,i-1 =F(P i ,P i-1 ); Where D i,i-1 is the linear motion distance of the target in video frame i and video frame i-1; F() is the calculation function of the linear distance between two points in the world coordinate system; P i is the world coordinate of target j in video frame i; P i-1 is the world coordinate of target j in video frame i-1; Based on the linear motion distance D of target j in adjacent video frames i,i-1 And the frame interval time t is used to calculate the moving speed of target j in the world coordinate system; S6. Determine the correlation between the passenger and the belongings based on the posture information and world coordinates of the passenger and the belongings, and issue a third warning message based on the correlation between the passenger and the belongings; S7. Perform a warning action according to the first warning information, the second warning information or the third warning information, wherein the warning action at least includes a control room alarm, an escalator alarm or an escalator stop.

2. The escalator monitoring and early warning method according to claim 1 is characterized in that: When the world coordinates of each target are obtained based on the camera with calibrated internal and external parameters combined with the image coordinate position of the target, the coordinate conversion method used is: The homogeneous equation corresponding to target j is obtained based on the coordinate transformation method of target j: With z j As a scaling factor for homogeneous coordinates, we solve for: The world coordinates of the target are: where u j and v j is the image coordinate position of target j in the video frame, u j is the column position of the target j pixel in the video frame, v j is the horizontal column position of the target j pixel in the video frame; K is the intrinsic parameter matrix of the camera, K -1 is the inverse matrix of the camera's intrinsic parameter matrix; R is the rotation matrix; t is the translation vector; X j , Y j and Z j is the world coordinate of target j, X j is the X-direction world coordinate of target j, Y j is the Y world coordinate of target j, Z j is the Z-direction world coordinate of target j, Z known is the known Z-direction world coordinate value of target j; x j ,y j and z j is the homogeneous coordinate of target j, that is, x j is the X-direction world coordinate X of target j j The homogeneous coordinates of j is the Y-direction world coordinate Y of target j j The homogeneous coordinates of j is the Z world coordinate Z of target j j The homogeneous coordinates of .

3. The escalator monitoring and early warning method according to claim 1 is characterized in that: The step of obtaining the first warning information based on the change of the bounding box of the same target in adjacent frames in the video stream includes: Get at least one set of diagonal coordinates (A) of the bounding box of the target in video frame i i,cor ,B i,cor ) and (a i,cor ,b i,cor ); and calculate the bounding box area of ​​the target in video frame i according to the diagonal coordinates of the bounding box of the target in video frame i; Get at least one set of diagonal coordinates (A) of the bounding box of the target in video frame i-1 i-1,cor ,B i-1,cor ) and (a i-1,cor ,b i-1,cor ), and calculate the bounding box area of ​​the target in the video frame i-1 according to the diagonal coordinates of the bounding box of the target in the video frame i-1; Get the rate of change of the target's bounding box area based on the target's bounding box area in adjacent video frames: Where ΔR Area Area is the rate of change of the target's bounding box area; i is the bounding box area of ​​the target in video frame i; Area i-1 is the bounding box area of ​​the target in video frame i-1; Δt is the time interval between video frame i and video frame i-1; Judge the change rate ΔR of the target bounding box area in adjacent video frames Area and the preset threshold R0. If ΔR Area ≥R0, it indicates that the target on the escalator has fallen, and at this time, the first warning message is sent; if ΔR Area <R0, it indicates that there is no fall of the target on the escalator, and at this time, the first warning message is not sent.

4. The escalator monitoring and early warning method according to claim 1 is characterized in that: The step of issuing the second warning information based on the target motion information comprises: Sequentially obtain the motion speed of target j between adjacent video frames; The movement trend of target j is obtained based on the change in the movement speed of target j: if the difference in the movement speed of target j at adjacent moments is greater than or equal to the preset threshold, it is judged that the target has slipped on the escalator, and a second warning message is issued at this time; if the difference in the movement speed of target j at adjacent moments is less than the preset threshold, it is judged that the target has not slipped on the escalator, and no second warning message is issued at this time.

5. The escalator monitoring and early warning method according to claim 1 is characterized in that: The step of issuing the second warning information based on the target motion information comprises: Sequentially obtain the moving speed of target j between adjacent video frames and obtain the running status of the escalator; If the escalator is in the stopped state, and the target j's moving speed is judged to be 0 or the target j's staying time on the escalator exceeds 1 minute, the target is judged to be stuck on the escalator, and a second warning message is issued; If the escalator is in normal operation, if the target j's moving speed is less than the normal operating speed or the target j's moving direction is inconsistent with the normal operating direction, then the target is judged to be moving in the reverse direction on the escalator, and a second warning message is issued.

6. The escalator monitoring and early warning method according to claim 1 is characterized in that: The step of determining the relevance between the passenger and the carried object based on the posture information and world coordinates of the passenger and the carried object comprises: Get the world coordinates of passengers and objects respectively; Taking the coordinates of the luggage as the center, obtain the straight-line distance between different passengers and the same luggage; Setting the maximum safe distance for the luggage, which is the maximum distance range within which the luggage can be controlled by the passenger, and determining whether the straight-line distance between different passengers and the same luggage exceeds the maximum safe distance; Selecting passengers whose straight-line distance from the same object does not exceed the maximum safe distance, and determining posture information of the selected passengers, the posture information including the frequency of contact between the passenger and the object and the duration of contact between the passenger and the object; Based on the passenger's posture information, it is determined whether there is one or more passengers who meet the characteristics of the owner of the belongings within the maximum safe distance of the belongings. If so, it is determined that the passenger and the belongings are strongly correlated; if not, it is determined that the passenger and the belongings are weakly correlated.

7. The escalator monitoring and early warning method according to claim 4 is characterized in that: The step of issuing the third warning information based on the correlation between the passenger and the carried objects comprises: When it is determined that the passenger and the carried item are strongly correlated, the carried item is under the control of at least one passenger, and there is no risk of the carried item being separated, and the third warning information is not issued at this time; When it is determined that the passenger and the carried items are weakly correlated, the carried items are not under the control of the passenger and there is a risk of the carried items being out of the passenger's control, and at this time, a third warning message is issued.

8. An escalator monitoring and early warning device based on target motion changes, characterized in that: The device comprises: An image recognition module, used to obtain the bounding box, posture information and category information of each target in the video stream based on an image recognition algorithm; the category information of the target includes at least passengers and carried objects; A target tracking module, used to obtain the image coordinate position of the same target in the video stream based on the target tracking algorithm; A coordinate conversion module is used to obtain the world coordinates of each target based on the camera with calibrated internal and external parameters and in combination with the image coordinate position of the target; A first warning module, configured to issue a first warning message based on a change in a bounding box of the same object in adjacent frames in a video stream; A second warning module is used to obtain motion information of the corresponding target based on the change of the world coordinate position of each target in the video stream, and issue a second warning information based on the target motion information; When obtaining the motion information of the corresponding target, the world coordinates of the same target in adjacent video frames are used to calculate the motion speed of the target in the world coordinate system, and the calculation steps include: Get the world coordinates of target j in video frame i and video frame i-1 respectively; Calculate the linear motion distance of target j in adjacent video frames based on the world coordinates of target j in video frame i and video frame i-1: D i,i-1 =F(P i ,P i-1 ); Where D i,i-1 is the linear motion distance of the target in video frame i and video frame i-1; F() is the calculation function of the linear distance between two points in the world coordinate system; P i is the world coordinate of target j in video frame i; P i-1 is the world coordinate of target j in video frame i-1; Based on the linear motion distance D of target j in adjacent video frames i,i-1 And the frame interval time t is used to calculate the moving speed of target j in the world coordinate system; A third warning module, used to determine the relevance between the passenger and the belongings based on the passenger's posture information and the coordinate position of the belongings, and issue a third warning message based on the relevance between the passenger and the belongings; The warning execution module is used to perform warning actions according to the first warning information, the second warning information or the third warning information, and the warning actions at least include control room alarm, escalator alarm or escalator stop.

Images