Escalator passenger accident monitoring and early warning method and device based on visual detection

Abstract

The application relates to an escalator passenger accident monitoring and early warning method and device based on visual detection, which comprises the following steps: obtaining the bounding box, posture information and category information of each target in a video stream; obtaining the image coordinate position of the target in the video stream; obtaining the world coordinate of each target based on the camera and the image coordinate position; issuing a first early warning information based on the bounding box change of the same target in adjacent frames in the video stream; obtaining the motion information of the target and issuing a second early warning information based on the target motion information; judging the correlation between the passenger and the carried object and issuing a third early warning information based on the correlation between the passenger and the carried object; and performing a warning action according to the first early warning information, the second early warning information or the third early warning information. The method obtains the motion state of the target on the escalator through image information, and performs differentiated early warning based on the target motion state, realizes automatic identification of the unexpected situation on the escalator and timely early warning, improves the escalator monitoring efficiency, and ensures the safe operation of the escalator.

Description

[0001] This application is a divisional application of the patent application filed on September 10, 2024, with application number 202411260742.8 and invention title "A method and device for monitoring and early warning of escalator passenger accidents based on visual detection". Technical Field

[0002] This application relates to the field of escalator safety monitoring technology, specifically to a method and device for monitoring and early warning of escalator passenger accidents based on visual detection. Background Technology

[0003] Escalators, as a common mode of transportation in public places, are becoming increasingly prevalent. Consequently, safety and smooth operation issues related to escalators are attracting growing attention. Currently, cameras are installed at escalator entrances and exits to monitor the status of passengers and objects on the escalator, such as whether there is congestion at the escalator entrance, whether any passengers have fallen, or whether any large items have become stuck or slipped. If any of these accidents occur during operation, they could cause panic or injury to passengers. Currently, escalator accident monitoring mainly relies on integrated mechanical sensors, infrared sensors, or visual monitoring. However, these methods often require integration and coupling with the escalator system, leading to uncontrollable costs. Furthermore, visual monitoring still relies on manual warnings, resulting in a low level of automation. If human intervention is not timely, the severity of the injury can be exacerbated. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, this application provides a method and device for monitoring and early warning of passenger accidents on escalators based on visual detection, specifically adopting the following technical solution:

[0005] A method for monitoring and early warning of passenger accidents on escalators based on visual detection, comprising the following steps:

[0006] S1. Obtain the bounding box, pose information, and category information of each target in the video stream based on an image recognition algorithm; the category information of the target includes at least passengers and carried items;

[0007] S2. Based on the target tracking algorithm, obtain the image coordinates of the same target in the video stream;

[0008] S3. Based on the calibrated internal and external parameters of the camera and combined with the image coordinates of the target, obtain the world coordinates of each target;

[0009] S4. Issue the first warning message based on the bounding box changes of the same target in adjacent frames of the video stream:

[0010] S5. Obtain motion information of the corresponding target based on the changes in the world coordinate position of each target in the video stream, and issue a second warning message based on the target motion information;

[0011] When acquiring the motion information of the corresponding target, the motion velocity of the target in the world coordinate system is calculated using the world coordinates of the same target in adjacent video frames. The calculation steps include:

[0012] Obtain video frames respectively i and video frames i-1 Medium target j World coordinates;

[0013] Based on video frames i and video frames i-1 Medium target j Calculate the linear motion distance of target j in adjacent video frames using world coordinates:

[0014] ;

[0015] in For video frames i and video frames i-1 The straight-line distance of the target; This is a function for calculating the straight-line distance between two points in the world coordinate system. For video frames i Medium target j World coordinates; For video frames i-1 Medium target j World coordinates;

[0016] Based on targets in adjacent video frames j linear motion distance and frame interval time Calculate to obtain the target j The velocity of motion in the world coordinate system;

[0017] S6. Determine the correlation between the passenger and the carried items based on the posture information and world coordinates of the passenger and the carried items, and issue a third early warning message based on the correlation between the passenger and the carried items;

[0018] S7. Take a warning action based on the first warning information, the second warning information or the third warning information, wherein the warning action includes at least a control room alarm, an escalator alarm or an escalator stop.

[0019] Optional: When obtaining the world coordinates of each target based on the image coordinates of the target using a camera with calibrated intrinsic and extrinsic parameters, the coordinate transformation method used is as follows:

[0020] ;

[0021] The homogeneous equation of target j is obtained based on the coordinate transformation method of target j;

[0022] ;

[0023] by As a scaling factor for homogeneous coordinates, we can solve for:

[0024] ;

[0025] The world coordinates of the target are:

[0026] ;

[0027] in and For the target in the video frame j The image coordinates position, For the target in the video frame j The column position of the pixel For the target in the video frame j The horizontal row position of the pixel; K Let be the intrinsic parameter matrix of the camera. K -1 This is the inverse of the camera's intrinsic parameter matrix; R It is a rotation matrix; t It is a translation vector; , and For the goal j World coordinates For the goal j of X To the world coordinates, For the goal j of Y To the world coordinates, For the goal j of Z To the world coordinates, For the goal j The known Z To world coordinates; , and For the goal j homogeneous coordinates, i.e. For the goal j of X To the world coordinates homogeneous coordinates For the goal j of Y To the world coordinates homogeneous coordinates For the goal j of Z To the world coordinates The homogeneous coordinates.

[0028] Optionally: The step of obtaining the first warning information based on the bounding box changes of the same target in adjacent frames of the video stream includes:

[0029] Acquire video frames i At least one set of diagonal coordinates of the bounding box of the target ( A i,cor , B i,cor )and( a i,cor , b i,cor ); and based on video frames i Calculate the diagonal coordinates of the bounding box of the target to obtain video frames. i The area of ​​the bounding box of the target;

[0030] Acquire video frames i-1 At least one set of diagonal coordinates of the bounding box of the target ( A i-1,cor , B i-1,cor )and( a i-1,cor , b i-1,cor ), and based on video frames i-1 Calculate the diagonal coordinates of the bounding box of the target to obtain video frames. i-1 The area of ​​the bounding box of the target;

[0031] The rate of change of the target's bounding box area is obtained from the bounding box areas of the target in adjacent video frames:

[0032] ;

[0033] in The rate of change of the bounding box area of ​​the target; For video frames i The area of ​​the bounding box of the target; For video frames i-1 The area of ​​the bounding box of the target; For video frames i With video frames i-1 The time interval between;

[0034] Determine the rate of change of the area of ​​the target bounding box in adjacent video frames With preset threshold Size, if This indicates that a person has fallen on the escalator, at which point the first warning message is issued; if If the signal is clear, it indicates that there is no risk of the person falling on the escalator, and no first warning message will be issued.

[0035] Optionally: The step of issuing the second warning information based on the target motion information includes:

[0036] Obtain the target in sequence j Motion speed between adjacent video frames;

[0037] Based on the goal j Obtaining the target's movement speed change j The motion trend: If the difference in the motion speed of target j between adjacent moments is greater than or equal to a preset threshold, it is determined that the target has slipped on the escalator, and a second warning message is issued at this time; if the difference in the motion speed of target j between adjacent moments is less than the preset threshold, it is determined that the target has not slipped on the escalator, and a second warning message is not issued at this time.

[0038] Optionally: The step of issuing the second warning information based on the target motion information includes:

[0039] Obtain the target in sequence j The motion speed between adjacent video frames and the escalator's operating status are obtained;

[0040] If the escalator is in a stopped state, determine the target. j If the target's movement speed is 0 or the time spent on the escalator exceeds 1 minute, it is determined that the target is lingering on the escalator, and a second warning message is issued.

[0041] If the escalator is in normal operating condition, determine the target. j The movement speed is less than the normal operating speed of the escalator or the target speed. j If the direction of movement of the target is inconsistent with the normal operating direction of the escalator, it is determined that the target is moving in the wrong direction on the escalator, and a second warning message is issued.

[0042] Optionally: The step of determining the correlation between the passenger and the carried item based on the passenger's and the carried item's posture information and world coordinates includes:

[0043] Obtain the world coordinates of the passenger and their belongings respectively;

[0044] Using the coordinates of the object as the center, obtain the straight-line distance between different passengers and the same object;

[0045] Set a maximum safe distance for carried items, where the maximum safe distance is the maximum distance range that the carried items can be controlled by the passenger, and determine whether the straight-line distance between different passengers and the same carried items exceeds the maximum safe distance;

[0046] Select passengers whose straight-line distance from the same object does not exceed the maximum safe distance, and determine the posture information of the selected passengers, including the frequency of contact between the passenger and the object and the duration of contact between the passenger and the object.

[0047] Based on the passenger's posture information, determine whether there is one or more passengers within the maximum safe distance of the carried item who match the characteristics of the owner of the carried item. If so, it is determined that the passenger and the carried item are strongly correlated; if not, it is determined that the passenger and the carried item are weakly correlated.

[0048] Optionally: The step of issuing third warning information based on the correlation between passengers and their belongings includes:

[0049] When it is determined that the passenger and the carried item are strongly related, the carried item is under the control of at least one passenger, and there is no risk of the carried item falling out. In this case, no third warning information is issued.

[0050] If it is determined that the passenger and the carried item are in a weak relationship, then the carried item is not under the passenger's control and there is a risk that the carried item will be lost. At this time, a third warning message will be issued.

[0051] Furthermore, this application also discloses a visual detection-based escalator passenger accident monitoring and early warning device, the device comprising:

[0052] The image recognition module is used to acquire the bounding box, pose information, and category information of each target in the video stream based on the image recognition algorithm; the category information of the target includes at least passengers and carried items;

[0053] The target tracking module is used to obtain the image coordinates of the same target in the video stream based on the target tracking algorithm.

[0054] The coordinate transformation module is used to obtain the world coordinates of each target based on the calibrated intrinsic and extrinsic parameters of the camera and the image coordinate position of the target;

[0055] The first early warning module is used to issue a first early warning message based on the change of the bounding box of the same target in adjacent frames of the video stream;

[0056] The second early warning module is used to obtain the motion information of the corresponding target based on the changes in the world coordinate position of each target in the video stream, and issue a second early warning message based on the target motion information;

[0057] When acquiring the motion information of the corresponding target, the motion velocity of the target in the world coordinate system is calculated using the world coordinates of the same target in adjacent video frames. The calculation steps include:

[0058] Obtain video frames respectively i and video frames i-1 Medium target j World coordinates;

[0059] Based on video frames i and video frames i-1 Medium target j Calculate the linear motion distance of target j in adjacent video frames using world coordinates:

[0060] ;

[0061] in For video frames i and video frames i-1 The straight-line distance of the target; This is a function for calculating the straight-line distance between two points in the world coordinate system. For video frames i Medium target j World coordinates; For video frames i-1 Medium target j World coordinates;

[0062] Based on targets in adjacent video frames j linear motion distance and frame interval time Calculate to obtain the target j The velocity of motion in the world coordinate system;

[0063] The third early warning module is used to determine the correlation between the passenger and the carried item based on the passenger's posture information and the coordinate position of the carried item, and to issue a third early warning message based on the correlation between the passenger and the carried item.

[0064] The early warning execution module is used to perform early warning actions based on the first early warning information, the second early warning information, or the third early warning information. The early warning actions include at least control room alarm, escalator alarm, or escalator stop.

[0065] Beneficial effects

[0066] The technical solution of this application achieves the following beneficial effects:

[0067] (1) The escalator monitoring and early warning method of this application can obtain the target status on the escalator by using image information collected by a camera, and make different warnings based on the target status, so as to realize the automatic identification of situations such as passengers falling, items being stuck or slipping, and crowding on the escalator, and timely warnings, thereby improving the escalator monitoring efficiency and ensuring the safety of escalator operation. In addition, the method can execute corresponding early warning actions based on different early warning information, such as control room alarm, escalator alarm, or escalator stop, to achieve intelligent monitoring.

[0068] (2) The escalator monitoring and early warning method of this application uses existing surveillance cameras to monitor whether the escalator is operating abnormally when carrying people or goods, and does not require additional hardware integration, which greatly reduces equipment costs. Attached Figure Description

[0069] Figure 1 This is a flowchart of the escalator monitoring and early warning method in the embodiments of this application.

[0070] Figure 2 This is a structural diagram of the escalator monitoring and early warning device in the embodiments of this application.

[0071] Figure 3 This is a structural diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0072] The present application will now be further described with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application and should not be construed as limiting the scope of protection of the present application. It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present application.

[0073] Currently, the common method for escalator monitoring is to use visual sensors, such as cameras. Multiple cameras are installed at the escalator entrance, exit, and midway to collect image data of passengers or their belongings. This image data is then transmitted to the control room for manual monitoring. However, this method relies on manual accident identification and judgment; if control personnel are negligent and fail to detect a monitored image in time, it can lead to a serious accident. Therefore, this application further processes the image data obtained from existing cameras, converting some or all of the data requiring manual identification into automated computer processing. Without increasing equipment costs, the computer uses internal algorithms to process the image data, automatically identifying escalator accidents and issuing warnings to the control room or escalator passengers.

[0074] Combination Figure 1 As shown in the embodiments, this application specifically discloses a method for monitoring and early warning of escalator passenger accidents based on visual detection, which includes the following steps:

[0075] Step 1: Based on image recognition algorithms, obtain the bounding boxes, pose information, and category information of each target in the video stream; the category information of the target includes at least passengers and carried items; preferably, in this embodiment, the NanoDet lightweight target detection network is used to detect targets on the escalator in real time to ensure fast and accurate target recognition.

[0076] Step 2: Obtain the image coordinates of the same target in the video stream based on the target tracking algorithm; in this embodiment, the BoTSORT algorithm is preferably used to continuously track the same target in the video stream and obtain the coordinates of the same target in different video frames.

[0077] Step 3: Based on the calibrated intrinsic and extrinsic parameters of the camera and the image coordinates of the target, obtain the world coordinates of each target:

[0078] Specifically, in step three, when obtaining the world coordinates of each target based on the image coordinates of the target using the calibrated intrinsic and extrinsic parameters of the camera, the coordinate transformation method used is as follows:

[0079] ;

[0080] The homogeneous equation of target j is obtained based on the coordinate transformation method of target j;

[0081] ;

[0082] by As a scaling factor for homogeneous coordinates, we can solve for:

[0083] ;

[0084] The world coordinates of the target are:

[0085] ;

[0086] in and For the target in the video frame j The image coordinates position, For the target in the video frame j The column position of the pixel For the target in the video frame j The horizontal row position of the pixel; K Let be the intrinsic parameter matrix of the camera. K -1 This is the inverse of the camera's intrinsic parameter matrix; R It is a rotation matrix; t It is a translation vector; , and For the goal j World coordinates For the goal j of X To the world coordinates, For the goal j of Y To the world coordinates, For the goal j of Z To the world coordinates, For the goal j The known Z To world coordinates; , and For the goal j homogeneous coordinates, i.e. For the goal j ofX To the world coordinates homogeneous coordinates For the goal j of Y To the world coordinates homogeneous coordinates For the goal j of Z To the world coordinates The homogeneous coordinates.

[0087] It should be noted that in step three, the longitudinal coordinates of the target, i.e. the Z-direction world coordinates, are known constants during the calculation of the target's world coordinates. Since the height of the escalator in the longitudinal direction is fixed, the coordinates in other directions can be obtained by pre-determining the target's longitudinal position (Z-direction) on the escalator.

[0088] Step 4: Issue the first warning message based on the changes in the bounding box of the same target in adjacent frames of the video stream.

[0089] Specifically, the process of obtaining the first warning information based on the bounding box changes of the same target in adjacent frames of the video stream in step four includes:

[0090] First, obtain video frames. i At least one set of diagonal coordinates of the bounding box of the target ( A i,cor , B i,cor )and( a i,cor , b i,cor ); and based on video frames i Calculate the diagonal coordinates of the bounding box of the target to obtain video frames. i The area of ​​the bounding box of the target;

[0091] Then video frames are acquired. i-1 At least one set of diagonal coordinates of the bounding box of the target ( A i-1,cor , B i-1,cor )and( a i-1,cor , b i-1,cor ), and based on video frames i-1 Calculate the diagonal coordinates of the bounding box of the target to obtain video frames. i-1 The area of ​​the bounding box of the target;

[0092] Again, obtain the rate of change of the target's bounding box area based on the target's bounding box area in adjacent video frames:

[0093] ;

[0094] in The rate of change of the bounding box area of ​​the target; For video frames i The area of ​​the bounding box of the target; For video frames i-1 The area of ​​the bounding box of the target; For video frames i With video frames i-1 The time interval between;

[0095] Determine the rate of change of the area of ​​the target bounding box in adjacent video frames With preset threshold Size, if This indicates that a person has fallen on the escalator, at which point the first warning message is issued; if If the signal is clear, it indicates that there is no risk of the person falling on the escalator, and no first warning message will be issued.

[0096] It should be noted that step four uses the rate of change of the target bounding box to determine whether the target has fallen on the escalator. Generally, when passengers or their belongings are riding an escalator, they are either standing still or moving at a constant speed. At this time, the size of the target's bounding box identified by the image recognition algorithm does not change much, that is, the area of ​​the target's bounding box fluctuates little or not at all. Once a passenger or their belongings fall, the area of ​​the target's bounding box will change significantly in a short period of time. Therefore, by monitoring the rate of change of the target's bounding box area, it can be determined whether the target on the escalator has fallen, and thus a first warning message can be issued.

[0097] Step 5: Obtain the motion information of the corresponding target based on the changes in the world coordinate position of each target in the video stream, and issue a second warning message based on the target motion information.

[0098] In one implementation, step five involves calculating the target's vertical velocity using a Kalman filter algorithm. The calculation steps include:

[0099] Define the goal j The initial state vector of the target j The state vector includes the target position and target speed and based on the goal j The initial state vector is used to establish the state transition equation:

[0100] ;

[0101] The observation equation is established based on the initial state vector of target j:

[0102] ;

[0103] in For the goal j exist k+1 The state vector at any given time; For the goal j exist k The state vector at any given time; For the goal j exist k The observation vector at time step A; A is the state transition matrix; H is the observation matrix; This is process noise; To observe noise; For the goal j State transition time interval;

[0104] Predict the target's state vector at the next moment based on the target's state transition equation, and calculate the target's state vector at the current moment based on the target's observation equation;

[0105] The predicted state vector for the next time step is corrected using the current state vector to obtain the corrected state vector.

[0106] The target velocity at the next moment is obtained based on the corrected state vector.

[0107] Based on the above method, it is only necessary to analyze the change in the target's velocity in the longitudinal direction to determine whether the target is currently slipping on the escalator. When the target is stationary or moving at a constant speed, the target velocity obtained from adjacent video frames will not change significantly within a certain period of time. When the target slips, the target velocity will change significantly in a short period of time. Therefore, by analyzing the change in target velocity, it is possible to accurately determine whether a target on the escalator has slipped. Once a slip occurs, a second warning message is promptly issued to alert the personnel in control or passengers on the escalator.

[0108] As another feasible implementation, step five can also use the world coordinates of the same target in adjacent video frames to calculate the target's motion velocity in the world coordinate system. The calculation steps include:

[0109] Obtain video frames respectively i and video frames i-1 Medium target j World coordinates;

[0110] Based on video frames i and video frames i-1 Medium target j Calculate the linear motion distance of target j in adjacent video frames using world coordinates:

[0111] ;

[0112] in For video frames i and video frames i-1 The straight-line distance of the target; This is a function for calculating the straight-line distance between two points in the world coordinate system. For video frames i Medium target j World coordinates; For video frames i-1 Medium target j World coordinates;

[0113] Based on targets in adjacent video frames j linear motion distance and frame interval time Calculate to obtain the target j The velocity of motion in the world coordinate system.

[0114] After obtaining the world coordinates of the same target in each video frame, the corresponding target's coordinate trajectory in real space is determined. Based on the world coordinates of the target in the video frame, the average speed of the target between adjacent video frames can be calculated. By analyzing the changes in the average speed of the target within a certain number of video frames, for example, when the target is stationary or moving at a constant speed, the average speed of the target obtained from adjacent video frames does not change significantly within a certain period of time; when the target experiences a slip, the average speed of the target will change significantly in a short period of time. Therefore, by analyzing the changes in the average speed, it is possible to accurately determine whether a target on the escalator has experienced a slip. Once a slip occurs, a second warning message is promptly issued to alert personnel in the control room or passengers on the escalator.

[0115] Furthermore, the specific steps in step five of issuing the second warning information based on the target motion information include:

[0116] Obtain the target in sequence j Motion speed between adjacent video frames;

[0117] Based on the goal j Obtaining the target's movement speed change j Movement trend: If the target j If the difference in motion speed between adjacent moments is greater than or equal to a preset threshold, it is determined that the target has slipped on the escalator, and a second warning message is issued; if the target j If the difference in motion speed between adjacent moments is less than a preset threshold, it is determined that the target has not slipped off the escalator, and no second warning message is issued at this time.

[0118] In this embodiment of the application, to further avoid misjudgments caused by human running or stopping on the escalator, a specific threshold can be preset when analyzing the target's movement trend. When analyzing the change in the difference in the target's movement speed between adjacent moments, the change in movement speed will generally have a certain buffer time when a person runs or stops (that is, the target speed does not suddenly increase or decrease in a short period of time, but slowly increases or decreases over a period of time). This is reflected in the small difference in movement speed between adjacent moments. By setting a specific threshold, human factors can be eliminated, and the situation where unexpected factors cause changes in movement speed can be preserved as much as possible, so as to correctly warn of escalator accidents.

[0119] Furthermore, step five, which involves issuing a second warning message based on the target motion information, also includes:

[0120] Obtain the target in sequence j The motion speed between adjacent video frames and the escalator's operating status are obtained;

[0121] If the escalator is in a stopped state, determine the target. j If the target's movement speed is 0 or the time spent on the escalator exceeds 1 minute, it is determined that the target is lingering on the escalator, and a second warning message is issued.

[0122] If the escalator is in normal operating condition, determine the target. j The movement speed is less than the normal operating speed of the escalator or the target speed. j If the direction of movement of the target is inconsistent with the normal operating direction of the escalator, it is determined that the target is moving in the wrong direction on the escalator, and a second warning message is issued.

[0123] The above methods can be used to identify lingering or reversing targets on escalators, thereby increasing the types of escalator accident warnings, expanding the warning scope, and improving warning efficiency.

[0124] Step 6: Determine the correlation between the passenger and their belongings based on the passenger's and their belongings' posture information and world coordinates, and issue a third early warning message based on the correlation between the passenger and their belongings;

[0125] Specifically, the purpose of determining the correlation between passengers and their belongings based on the posture information and world coordinates in step six is ​​to further avoid potential dangers of objects going out of control on escalators and to provide early warnings for belongings at risk of going out of control. The specific steps include:

[0126] The world coordinates of the passenger and the carried items can be obtained according to the process of steps one to three. It should be noted that the world coordinates of the passenger and the carried items can be the center coordinates of the bounding box as the world coordinates of the corresponding targets, ignoring the difference in volume.

[0127] At this point, taking the coordinates of each item as the center, the straight-line distance between different passengers and the same item can be obtained, and the straight-line distance can be obtained from the world coordinates of the passenger and the item.

[0128] Set a maximum safe distance for carried items, where the maximum safe distance is the maximum distance range that the carried items can be controlled by the passenger, and determine whether the straight-line distance between different passengers and the same carried items exceeds the maximum safe distance;

[0129] Select passengers whose straight-line distance from the same object does not exceed the maximum safe distance, and determine the posture information of the selected passengers, including the frequency of contact between the passenger and the object and the duration of contact between the passenger and the object.

[0130] Based on the passenger's posture information, determine whether there is one or more passengers within the maximum safe distance of the carried item who match the characteristics of the owner of the carried item. If so, it is determined that the passenger and the carried item are strongly correlated; if not, it is determined that the passenger and the carried item are weakly correlated.

[0131] Furthermore, when it is determined that the passenger and the carried item are strongly correlated, the carried item is under the control of at least one passenger, and there is no risk of the carried item being lost. In this case, no third warning information is issued. When it is determined that the passenger and the carried item are weakly correlated, the carried item is not under the control of the passenger, and there is a risk of the carried item being lost. In this case, a third warning information is issued.

[0132] Step Seven: Execute warning actions based on the first, second, or third warning information. These actions include at least a control room alarm, an escalator alarm, or an escalator stop. It should be noted that the warning actions for different warning information can be automatically executed according to pre-set parameters by control room personnel, or manually controlled by control room personnel.

[0133] In addition, combined Figure 2 As shown, this application also discloses a method and apparatus for monitoring and early warning of escalator passenger accidents based on visual detection, the apparatus comprising:

[0134] The image recognition module is used to acquire the bounding box, pose information, and category information of each target in the video stream based on the image recognition algorithm; the category information of the target includes at least passengers and carried items;

[0135] The target tracking module is used to obtain the image coordinates of the same target in the video stream based on the target tracking algorithm.

[0136] The coordinate transformation module is used to obtain the world coordinates of each target based on the calibrated intrinsic and extrinsic parameters of the camera and the image coordinates of the target:

[0137] The first early warning module is used to issue a first early warning message based on the change of the bounding box of the same target in adjacent frames of the video stream;

[0138] The second early warning module is used to obtain the motion information of the corresponding target based on the changes in the world coordinate position of each target in the video stream, and issue a second early warning message based on the target motion information;

[0139] The third early warning module is used to determine the correlation between the passenger and the carried item based on the passenger's posture information and the coordinate position of the carried item, and to issue a third early warning message based on the correlation between the passenger and the carried item.

[0140] The early warning execution module is used to perform early warning actions based on the first early warning information, the second early warning information, or the third early warning information. The early warning actions include at least control room alarm, escalator alarm, or escalator stop.

[0141] The apparatus provided in this application embodiment can achieve... Figure 1 To avoid repetition, the various processes implemented in the method embodiments will not be described again here.

[0142] like Figure 3 As shown in the illustration, this application also provides an electronic device, including a processor and a memory, and a program or instructions stored in the memory and executable on the processor, which, when executed by the processor, implement as follows: Figure 1 The various processes of the method embodiments shown are all capable of achieving the same technical effect, and will not be described again here to avoid repetition.

[0143] This application embodiment also provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the above-described functionality. Figure 1 The various processes described in the embodiments of the method described herein can achieve the same technical effect, and will not be repeated here to avoid repetition.

[0144] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described... Figure 1 The various processes described in the embodiments of the method described herein can achieve the same technical effect, and will not be repeated here to avoid repetition.

[0145] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0146] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another device, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0148] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0149] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0150] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0151] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a device (which may be a terminal or platform, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0152] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for monitoring and early warning of escalator passenger accident based on visual detection, characterized in that, The method comprises the following steps: S1, obtaining the bounding box, pose information and category information of each target in the video stream based on an image recognition algorithm; the category information of the target at least includes passengers and carry-on articles; S2, obtaining the image coordinate position of the same target in the video stream based on a target tracking algorithm; S3, obtaining the world coordinate of each target based on the camera with calibrated internal and external parameters and in combination with the image coordinate position of the target; S4, issuing a first warning information based on the change of the bounding box of the same target in adjacent frames in the video stream; S5, obtaining the motion information of the corresponding target based on the change of the world coordinate position of each target in the video stream, and issuing a second warning information based on the target motion information; Wherein, when obtaining the motion information of the corresponding target, the world coordinate of the same target in adjacent video frames is used to calculate the motion speed of the target in the world coordinate system, and the calculation steps include: Respectively acquiring world coordinates of a target in a video frame and a video frame i i-1 j ​​​ Based on video frames i And video frames i-1 World coordinates of the target j Calculate the straight-line motion distance of target j in adjacent video frames: ; wherein is the video frame i and the video frame i-1 target straight line motion distance; is the world coordinate system under two points straight line distance calculation function; is the video frame i target j world coordinates; is the video frame i-1 target j world coordinates; Based on targets in adjacent video frames j linear motion distance and frame interval time Calculate to obtain the target j The velocity of motion in the world coordinate system; S6, judging the correlation of the passengers and the carry-on articles based on the pose information and the world coordinate of the passengers and the carry-on articles, and issuing a third warning information based on the correlation of the passengers and the carry-on articles; Wherein, the step of judging the correlation of the passengers and the carry-on articles based on the pose information and the world coordinate of the passengers and the carry-on articles includes: obtaining the world coordinate of the passengers and the carry-on articles respectively; obtaining the straight-line distance between different passengers and the same carry-on article with the coordinate where the carry-on article is located as the center; setting the maximum safe distance of the carry-on article, the maximum safe distance being the maximum distance range in which the carry-on article is controlled by the passengers, and judging whether the straight-line distance between different passengers and the same carry-on article exceeds the maximum safe distance; selecting the passengers whose straight-line distance to the same carry-on article does not exceed the maximum safe distance, and judging the pose information of the selected passengers, the pose information including the contact frequency of the passengers and the carry-on article and the duration that the passengers contact the carry-on article; judging whether there is one or more passengers who meet the characteristics of the owner of the carry-on article within the maximum safe distance of the carry-on article according to the pose information of the passengers, if there is, judging that the passengers and the carry-on article are in strong correlation, and if not, judging that the passengers and the carry-on article are in weak correlation; S7, performing a warning action according to the first warning information, the second warning information or the third warning information, the warning action at least including controlling the alarm of the room, the alarm of the escalator or stopping the escalator.

2. The escalator passenger accident monitoring and early warning method according to claim 1, characterized in that, When obtaining the world coordinate of each target based on the camera with calibrated internal and external parameters in combination with the image coordinate position of the target, the coordinate conversion method used is: ; Based on the coordinate transformation method of the target j obtained corresponding target j homogeneous equation: ； With As a scaling factor for homogeneous coordinates, we find ； Then the world coordinate of the target is: ； wherein and is the image coordinate position of the target j in the video frame, is the vertical column position of the target j pixel point in the video frame, is the horizontal column position of the target j pixel point in the video frame; K is the intrinsic matrix of the camera, K -1 is the inverse matrix of the intrinsic matrix of the camera; R is the rotation matrix; t is the translation vector; , and is the world coordinate of the target j , is the world coordinate of the target j , X is the world coordinate of the target , j is the world coordinate of the target Y , is the world coordinate of the target j , Z is the world coordinate of the target , j is the known world coordinate value of the target Z , , and is the homogeneous coordinate of the target j , is the homogeneous coordinate of the target j , X is the homogeneous coordinate of the target , is the homogeneous coordinate of the target j , Y is the homogeneous coordinate of the target , is the homogeneous coordinate of the target j , Z is the homogeneous coordinate of the target .

3. The escalator passenger accident monitoring and early warning method according to claim 1, characterized in that, The step of obtaining the first warning information based on the change of the bounding box of the same target in adjacent frames in the video stream includes: acquiring a video frame i at least one set of diagonal coordinates of a bounding box of the target in the video frame A i,cor , B i,cor ) and ( a i,cor , b i,cor ); and calculating an area of the bounding box of the target in the video frame i based on the diagonal coordinates of the bounding box of the target in the video frame i ​ Acquire video frames i-1 At least one set of diagonal coordinates of the bounding box of the target ( A i-1,cor , B i-1,cor )and( a i-1,cor , b i-1,cor ), and based on video frames i-1 Calculate the diagonal coordinates of the bounding box of the target to obtain video frames. i-1 The area of ​​the bounding box of the target; obtaining the change rate of the area of the bounding box of the target according to the area of the bounding box of the target in adjacent video frames: ; wherein a rate of change of the area of the bounding box of the target; is a video frame i the area of the bounding box of the target in the video frame is a video frame i-1 the area of the bounding box of the target in the video frame is a video frame i the time interval between the video frame i-1 the time interval between the video frame Judging area change rate of target bounding box in adjacent video frames a preset threshold value size, if , it indicates that the target on the escalator falls, at which time the first early warning information is sent; if , it indicates that the target on the escalator does not fall, at which time the first early warning information is not sent.

4. The escalator passenger accident monitoring and early warning method according to claim 1, characterized in that, The step of issuing the second warning information based on the target motion information includes: Sequentially acquiring targets j Motion velocity between adjacent video frames; Based on the target j of the movement speed change obtaining target j of the movement trend: if the movement speed difference of the target j at adjacent time is greater than or equal to a preset threshold value, it is judged that the target slips on the escalator, at which time the second warning information is issued; if the movement speed difference of the target j at adjacent time is less than the preset threshold value, it is judged that the target does not slip on the escalator, at which time the second warning information is not issued.

5. The escalator passenger accident monitoring and early warning method according to claim 1, characterized in that, The step of issuing the second warning information based on the target motion information includes: Sequentially acquiring target j The motion speed between adjacent video frames is acquired and the escalator running state is acquired; If the escalator running state is in the stop running state, it is judged that the movement speed of the target is 0 or the escalator staying time is more than 1 minute, and it is judged that the target is staying on the escalator, and at this time, the second early warning information is sent out. j If the escalator running state is in the stop running state, it is judged that the movement speed of the target is 0 or the escalator staying time is more If the escalator is in normal operating condition, determine the target. j The movement speed is less than the normal operating speed of the escalator or the target speed. j If the direction of movement of the target is inconsistent with the normal operating direction of the escalator, it is determined that the target is moving in the wrong direction on the escalator, and a second warning message is issued.

6. The escalator passenger accident monitoring and early warning method according to claim 4, characterized in that, The step of issuing the third warning information based on the correlation of the passengers and the carry-on articles includes: When judging that the passengers and the carry-on article are in strong correlation, the carry-on article is controlled by at least one passenger, the carry-on article has no risk of separation, and no third warning information is issued at this time; When judging that the passengers and the carry-on article are in weak correlation, the carry-on article is not controlled by the passengers, the carry-on article has a risk of separation, and the third warning information is issued at this time.

Images