Escalator passenger abnormal behavior detection method, system, electronic device and storage medium

Abstract

The present invention discloses a method, system, electronic device and storage medium for detecting abnormal behavior of escalator passengers, wherein the method comprises: obtaining an image frame sequence of the area where the escalator is located; extracting and tracking the passengers and markers from the image frame sequence to obtain passenger trajectories and marker trajectories, wherein the markers are set on the escalator and move synchronously with the escalator; and determining the abnormal behavior detection result of the passenger based on the similarity between the passenger trajectory and the marker trajectory. Through the above scheme, the passenger trajectory and the marker trajectory can be obtained respectively according to the image frame sequence of the area where the escalator is located, and then the abnormal behavior detection result of the passenger can be obtained based on the similarity between the passenger trajectory and the marker trajectory, thereby improving the accuracy of abnormal behavior detection in crowded situations.

Description

Technical Field

[0001] The present application relates to the technical field of video surveillance and image processing, and in particular to a method, system, electronic device, and storage medium for detecting abnormal behavior of escalator passengers. Background Art

[0002] With the increasing maturity and popularization of information technology and the widespread installation of convenience facilities, security measures for convenience facilities have received increasing attention, especially security measures for escalators. Current technology for escalator security measures usually includes key point detection of personnel through monitoring equipment, obtaining personnel trajectory, and then obtaining abnormal behavior of personnel.

[0003] During the research and practice of current technology, the inventors of this application found that the current technology will not obtain enough key points in crowded situations, resulting in unsatisfactory personnel trajectories and a decrease in the accuracy of detecting abnormal personnel behavior. Summary of the Invention

[0004] The main technical problem solved by the present application is to provide a method, system, electronic device and storage medium for detecting abnormal behavior of escalator passengers, which can obtain passenger trajectories and marker trajectories associated with the passenger trajectories, and obtain the abnormal behavior detection results of the passengers based on the similarity between the passenger trajectories and the marker trajectories.

[0005] In order to solve the above technical problems, a technical solution adopted in this application is: to provide a method for detecting abnormal behavior of escalator passengers, the method comprising: obtaining an image frame sequence of the area where the escalator is located; extracting and tracking passengers and markers from the image frame sequence respectively to obtain passenger trajectories and marker trajectories, wherein the markers are set on the escalator; and determining the abnormal behavior detection result of the passenger based on the similarity between the passenger trajectory and the marker trajectory.

[0006] In one embodiment of the present application, the target extraction and tracking of the passenger and the marker from the image frame sequence respectively includes: target extraction of the passenger using the passenger's head and / or shoulder as the target.

[0007] In one embodiment of the present application, there are at least two markers and the marker trajectories corresponding to the markers; before obtaining the abnormal behavior detection result of the passenger based on the similarity between the passenger trajectory and the marker trajectory, it also includes: screening out the marker trajectory associated with the passenger from at least two marker trajectories.

[0008] In one embodiment of the present application, filtering out the marker trajectory associated with the passenger from at least two marker trajectories includes: expanding the detection frame of the passenger; and selecting the marker trajectory corresponding to the marker located within the expanded detection frame as the marker trajectory associated with the passenger.

[0009] In one embodiment of the present application, the selecting the marker trajectory corresponding to the marker located in the expanded detection frame as the marker trajectory associated with the passenger includes: in response to the number of the markers in the expanded detection frame being at least two, selecting the marker trajectory corresponding to the marker with the longest duration located in the expanded detection frame or the marker closest to the passenger as the marker trajectory associated with the passenger.

[0010] In one embodiment of the present application, the passenger trajectory is a discrete sequence of passenger positions, and the marker trajectory is a discrete sequence of marker positions.

[0011] In one embodiment of the present application, obtaining the abnormal behavior detection result of the passenger based on the similarity between the passenger trajectory and the marker trajectory includes: calculating the similarity between the passenger trajectory and the marker trajectory based on the passenger position sequence and the marker position sequence; the similarity is represented by the trajectory similarity distance between the passenger position sequence and the marker position sequence; wherein, the trajectory similarity distance and the similarity are negatively correlated, and the trajectory similarity distance is calculated and obtained by a trajectory similarity measurement method.

[0012] In one embodiment of the present application, obtaining the abnormal behavior detection result of the passenger based on the similarity between the passenger trajectory and the marker trajectory includes: in response to the similarity between the passenger trajectory and the marker trajectory being less than or equal to a preset similarity threshold, determining that the passenger has abnormal behavior.

[0013] In one embodiment of the present application, the obtaining of the abnormal behavior detection result of the passenger based on the similarity between the passenger trajectory and the marker trajectory further includes: extracting a regional image of the area where the passenger with the abnormal behavior is located from the image frame sequence; and classifying the regional image using a pre-trained classification network to obtain a final abnormal behavior detection result.

[0014] In order to solve the above technical problems, another technical solution adopted in this application is: to provide an escalator passenger abnormal behavior detection system, the system including: a sequence acquisition module, used to obtain an image frame sequence of the area where the escalator is located; a trajectory acquisition module, used to extract and track passengers and markers from the image frame sequence respectively to obtain passenger trajectories and marker trajectories, wherein the marker is set on the escalator and moves synchronously with the escalator; a detection result acquisition module, which obtains the passenger's abnormal behavior detection result based on the similarity between the passenger trajectory and the marker trajectory.

[0015] In one embodiment of the present application, the system further includes: a classification module, configured to classify the regional image of the area where the passenger with abnormal behavior is located, so as to obtain a final abnormal behavior detection result.

[0016] In order to solve the above technical problems, another technical solution adopted in this application is: to provide an electronic device, which includes a memory and a processor coupled to the memory, and the memory stores at least one computer program. When the at least one computer program is loaded and executed by the processor, it is used to implement the above-mentioned escalator passenger abnormal behavior detection method.

[0017] In order to solve the above technical problems, another technical solution adopted in this application is: providing a computer-readable storage medium, which stores at least one program. When the at least one program is loaded and executed by the processor, it is used to implement the above-mentioned escalator passenger abnormal behavior detection method.

[0018] Different from the prior art, the method for detecting abnormal behavior of escalator passengers provided by the present application includes: obtaining an image frame sequence of the area where the escalator is located; extracting and tracking the passengers and markers from the image frame sequence to obtain passenger trajectories and marker trajectories, wherein the markers are set on the escalator and move synchronously with the escalator; obtaining the abnormal behavior detection result of the passenger based on the similarity between the passenger trajectory and the marker trajectory. That is, the present application obtains an image frame sequence of the escalator area, thereby determining the passenger trajectory and the marker trajectory respectively according to the discrete position sequence corresponding to the passengers and the markers, obtains the marker trajectory associated with the passenger based on the passenger's detection frame, and then determines the marker trajectory associated with the passenger trajectory, and calculates the trajectory similarity based on the passenger trajectory and the marker trajectory to obtain the passenger abnormal behavior detection result. By comparing different trajectories, the accuracy of abnormal behavior detection in crowded situations is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of an embodiment of a method for detecting abnormal behavior of escalator passengers according to the present invention;

[0020] Figure 2 This is a flow chart of an embodiment of step S1 of the present invention;

[0021] Figure 3 This is a flow chart of an embodiment of step S2 of the present invention;

[0022] Figure 4 This is a structural diagram of an embodiment of an image acquisition device in the present invention for acquiring a marker on an escalator;

[0023] Figure 5 This is a flow chart of an embodiment of step S23 of the present invention;

[0024] Figure 6 This is a flow chart of an embodiment of step S232 of the present invention;

[0025] Figure 7 This is a flow chart of an embodiment of step S3 of the present invention;

[0026] Figure 8 This is a flow chart of an embodiment of the present invention for classifying abnormal behaviors;

[0027] Figure 9 1 is a schematic structural diagram of an embodiment of an escalator passenger abnormal behavior detection system according to the present invention;

[0028] Figure 10 2 is a schematic structural diagram of another embodiment of the escalator passenger abnormal behavior detection system of the present invention;

[0029] Figure 11 It is a structural diagram of an embodiment of an electronic device of the present invention;

[0030] Figure 12 It is a structural diagram of an embodiment of a computer-readable storage medium of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It is particularly noted that the following examples are intended only to illustrate the present invention and are not intended to limit the scope of the present invention. Similarly, the following examples are only some embodiments of the present invention and are not intended to be all embodiments. All other embodiments obtained by those of ordinary skill in the art without creative effort are intended to fall within the scope of protection of the present invention.

[0032] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0033] Existing methods for detecting abnormal escalator passenger behavior typically use key point detection to obtain passenger trajectories, thereby determining abnormal passenger behavior. However, in crowded environments, occlusions between people can easily occur, resulting in suboptimal key point acquisition results. This in turn affects the accuracy of subsequent abnormal behavior detection.

[0034] During the research, the applicant found that for detection in crowded situations, the passengers and markers on the escalator can be extracted and tracked based on the image frame sequence of the escalator area to obtain the discrete position sequence corresponding to the passenger trajectory and the marker trajectory, determine the corresponding passenger trajectory and marker trajectory, and then obtain the passenger's abnormal behavior detection results based on the similarity between the passenger trajectory and the marker trajectory, which can effectively improve the accuracy of the abnormal behavior detection results.

[0035] Therefore, a method for detecting abnormal behavior of escalator passengers is proposed. The method obtains an image frame sequence of the area where the escalator is located; extracts and tracks passengers and markers from the image frame sequence respectively to obtain passenger trajectories and marker trajectories, wherein the markers are set on the escalator and move synchronously with the escalator; and obtains the abnormal behavior detection results of passengers based on the similarity between the passenger trajectory and the marker trajectory.

[0036] See also Figure 1 , Figure 1 is a flow chart of an embodiment of the method for detecting abnormal behavior of escalator passengers of the present invention; it should be noted that if there are substantially the same results, the method of the present invention is not based on Figure 1 The process sequence shown is limited, such as Figure 1 As shown, the method includes the following steps:

[0037] S1. Acquire a sequence of image frames of the area where the escalator is located;

[0038] The area where the escalator is located is the target area for detection, and the image frame is the smallest unit that constitutes a video. Therefore, a sequence of image frames can constitute a series of image data, such as a video.

[0039] Specifically, the target area where the escalator is located is imaged by the image acquisition device to obtain the corresponding image data of the target area, that is, the image frame sequence, thereby reducing the interference of complex background, such as removing the background corresponding to scenes such as supermarkets and ground drops.

[0040] See Figure 2 , Figure 2 1 is a flow chart of an embodiment of step S1, wherein step S1 includes:

[0041] S11. Install an image acquisition device above the escalator;

[0042] Among them, the image acquisition device can be a color camera, a black and white camera, a binocular camera, a network camera, etc., which can obtain the image frame sequence of the area where the escalator is located. During use, there is no need to adjust the installation angle of the image acquisition device.

[0043] Specifically, the image acquisition device is arranged above the escalator so that the image acquisition device can cover the entire escalator area.

[0044] In some embodiments, the image acquisition device can be set directly above the escalator or diagonally above the escalator, and it only needs to be able to capture detection targets such as the passenger's head and / or shoulders.

[0045] In some embodiments, if the escalator is too long and one image acquisition device cannot fully cover the area where the escalator is located, multiple image acquisition devices can be set up in sections, with each image acquisition device being responsible for its respective area.

[0046] S12, using an image acquisition device above the escalator to capture an image frame sequence in a bird's-eye view;

[0047] Among them, the image frame sequence is a sequence composed of image frames, which includes multiple image frames, and each image frame corresponds to an image; the image acquisition device can obtain the positions of all passengers on the escalator in a bird's-eye view, which can effectively reduce occlusion between passengers.

[0048] Specifically, after the installation position of the image acquisition device is determined, the image frame sequence is obtained by shooting in a bird's-eye view by the image acquisition device arranged above the escalator.

[0049] In some embodiments, it is possible to detect whether the area where the escalator is located is moving. If there is movement in the area where the escalator is located, that is, the escalator is in operation, then a sequence of image frames of the area is acquired; if there is no movement in the area where the escalator is located, that is, when the escalator stops running, then no corresponding processing is performed.

[0050] Optionally, whether the escalator is in motion can be detected by using motion area detection methods, such as regional light flow method and background difference method, which can effectively reduce the background interference of complex backgrounds in scenes such as supermarkets and subways on the entire solution, especially the impact on false alarms of detection targets.

[0051] In some embodiments, it is possible to detect whether there are passengers in the area where the escalator is located. If there are passengers in the area where the escalator is located, an image frame sequence of the area is obtained; if there are no passengers in the area where the escalator is located, no corresponding processing is performed.

[0052] Optionally, corresponding detection of passengers and markers can be performed through a target detection network. The target detection network is trained based on the markers on the escalator until the network converges and can detect targets normally. The target detection network can be a one-stage method YOLO series, a two-stage method Faster RCNN series, an anchor-free method CenterNet series, etc.

[0053] In some embodiments, if the escalator is not in operation, but there are passengers in the area where the escalator is located, it is also possible to select to obtain a sequence of image frames in the area.

[0054] S2. Extracting and tracking the passenger and the marker from the image frame sequence to obtain the passenger trajectory and the marker trajectory; wherein the marker is set on the escalator and moves synchronously with the escalator;

[0055] Among them, target extraction refers to the operation of separating the target of interest from the background in a single image or a sequence of images, identifying and interpreting meaningful object entities from the image, and extracting different image features; target tracking refers to giving the target of interest an initial state, and then obtaining the state of the target at each moment in the sequence of images.

[0056] Specifically, the passengers and the background corresponding to the passengers are segmented in the image frame sequence obtained from the area where the escalator is located to obtain the image features of the passengers and the status of the passengers at each moment in the image frame sequence; and the markers on the escalator and the background corresponding to the markers are segmented in the image frame sequence obtained from the area where the escalator is located to obtain the image features of the markers and the status of the markers at each moment in the image frame sequence, thereby obtaining the corresponding passenger trajectories and marker trajectories.

[0057] See Figure 3 , Figure 3 2 is a flow chart of an embodiment of step S2, wherein step S2 includes:

[0058] S21, extracting and tracking the passenger from the image frame sequence to obtain the passenger trajectory;

[0059] Specifically, the passenger and the corresponding background are segmented from the image frame sequence obtained from the escalator area to obtain the passenger's image features; and the target is tracked to obtain the passenger's state at each moment in the image frame sequence, thereby obtaining the passenger's trajectory;

[0060] In some embodiments, the passenger's head and / or shoulders are used as detection targets for passenger extraction; for example, in a sequence of image frames obtained from the area where the escalator is located, the passenger's head or shoulders or head and shoulders are segmented from the rest of the background, and then image features representing the passenger are obtained based on the passenger's head or shoulders or head and shoulders, and the state of the passenger at each moment in the image frame sequence is obtained for the passenger's head or shoulders or head and shoulders. Using human body information with strong structures such as the head and / or shoulders, the position of the passenger in the image frame sequence can be effectively obtained, thereby effectively solving the problem of unstable pedestrian detection in crowded areas. And using the same method, the image features corresponding to the markers on the escalator are obtained, and the state of the markers at each moment in the image frame sequence is correspondingly obtained.

[0061] S22, extracting and tracking the marker from the image frame sequence to obtain a trajectory of the marker;

[0062] Specifically, the markers on the escalator and the background corresponding to the markers are segmented from the image frame sequence obtained from the area where the escalator is located to obtain the image features of the markers, and the state of the markers at each moment in the image frame sequence is obtained to obtain the trajectory of the markers.

[0063] In some embodiments, there are two markers and marker trajectories corresponding to the markers; that is, there can be two markers on the escalator, and there are also two marker trajectories corresponding to the markers. The two markers here refer to the presence of two markers in the image captured by the image acquisition device; because the escalator is moving, and the markers move synchronously with the escalator, during the image acquisition process, there are always two markers on the escalator surface corresponding to the image acquisition device. In this way, in the case of large-scale activities of passengers in the area where the escalator is located, a small number of markers and the marker trajectories corresponding to the markers can be used to associate with the passengers in the area; for example: most of the passengers on the escalator are moving at an accelerated speed, then a small number of markers and their corresponding marker trajectories can be used to perform subsequent trajectory identification on most of the passengers, thereby correspondingly reducing the time it takes to acquire the markers and their marker trajectories.

[0064] In some embodiments, there are multiple markers and marker trajectories corresponding to the markers; the multiple number can be three, five, ten, etc., which can be set according to actual conditions; if the escalator is ten meters long, the markers on the escalator can be set to one marker per meter or one marker per half a meter, and multiple markers can be arranged on the escalator at the same intervals; the multiple markers referred to here mean that there are multiple markers in the image captured by the image acquisition device; because the escalator is moving, and the markers move synchronously with the escalator, there are always multiple markers on the escalator surface corresponding to the image acquisition device during the image acquisition process, so that in the case of small-scale activities of passengers in the area where the escalator is located, the detection accuracy can be improved by associating the passengers in the area with the markers closest to the passengers and the marker trajectories corresponding to the markers; for example: if a small number of passengers on the escalator are moving at an accelerated speed, the subsequent trajectory of the passengers can be identified through the markers closest to the passengers and their corresponding marker trajectories, thereby correspondingly improving the accuracy of associating passengers with markers.

[0065] See Figure 4 , Figure 4 This is a structural diagram of an embodiment of an image acquisition device capturing markers on an escalator, wherein the markers clearly indicate the direction of travel of the escalator, and a passenger's detection frame may include multiple markers.

[0066] Optionally, the marker may be an additional printed marker on the escalator. The marker may have obvious features such as color features or travel direction features, or may be a combination of multiple features.

[0067] It is understandable that the number of markers is not limited to a specific number and can be set according to the actual situation on site as long as the technical solution of this application can be implemented and the corresponding technical effects can be achieved.

[0068] S23. Filter out a marker trajectory associated with the passenger from at least two marker trajectories.

[0069] There are passengers and corresponding markers in the detection frame, and there may be two or more markers in the detection frame. Therefore, it is necessary to screen out a marker that is most associated with the passenger and its corresponding marker trajectory.

[0070] See Figure 5 , Figure 5 This is a flow chart of an embodiment of step S23, where step S23 includes:

[0071] S231. Expand the passenger detection frame;

[0072] The passenger detection frame is used to obtain an image representing the passenger detection target.

[0073] Specifically, a detection frame corresponding to the passenger position in the escalator area is obtained to obtain an image of the passenger detection target. On this basis, the passenger detection frame is expanded to obtain the markers on the escalator that appear in the passenger detection frame and their corresponding images.

[0074] In some embodiments, each passenger has a corresponding detection frame, corresponding to the situation where there are multiple markers on the escalator, to determine the marker corresponding to each passenger.

[0075] In some embodiments, each detection frame may also have multiple passengers. When there are fewer markers on the escalator, the markers corresponding to the passengers within a small range in the escalator can be determined.

[0076] S232. Select the marker track corresponding to the marker located in the expanded detection frame as the marker track associated with the passenger.

[0077] See Figure 6 , Figure 6 This is a flow chart of an embodiment of step S232 of the present invention. Step S232 includes:

[0078] S2321, in response to there being at least two markers in the expanded detection frame;

[0079] Among them, the expanded detection frame contains corresponding passengers and signs.

[0080] Specifically, after the passenger detection frame is expanded, the corresponding passenger and identifiers will be included in the detection frame, and there will be at least two identifiers, in preparation for the subsequent selection of the passenger-related identifiers.

[0081] In some embodiments, there may be two markers within the expanded detection frame. For example, if there are only two markers on the escalator, the expanded detection frame should cover the entire escalator, corresponding to the situation where passengers on the escalator have large-scale activities.

[0082] In some embodiments, there can be multiple markers in the expanded detection frame, such as three, five, ten, etc. For the situation where there are many markers on the escalator, such as there are ten markers on the escalator, the expanded detection frame only needs to cover a small area of the escalator, corresponding to the situation where passengers on the escalator have small-scale activities.

[0083] S2322. Select the marker trajectory corresponding to the marker with the longest duration in the expanded detection frame or the marker closest to the passenger as the marker trajectory associated with the passenger.

[0084] Among them, the marker with the longest duration in the detection frame refers to the marker with the longest duration from the time it appears to the time it disappears in the detection frame, and the marker closest to the passenger refers to the marker closest to the passenger on the escalator in the image frame sequence.

[0085] Specifically, the marker that appears in the detection frame for the longest time is obtained, or the marker closest to the passenger on the escalator in the image frame sequence is obtained, and the marker trajectory corresponding to the obtained marker is used as the marker trajectory associated with the passenger.

[0086] In some embodiments, the detection frame contains a small number of markers, such as two markers. When a passenger accelerates or moves backward on the escalator, the markers will disappear from the detection frame because they move synchronously with the escalator. The longer the appearance time, the more relevant the obtained marker trajectory is, and the higher the accuracy of the subsequent detection results. Therefore, it is necessary to obtain the marker with the longest appearance time in the detection frame.

[0087] In some embodiments, the detection frame includes multiple markers, such as three, five, etc. When a passenger falls on the escalator or climbs over the escalator, the marker will always be in the detection frame because the marker moves synchronously with the escalator. When there are multiple markers, the marker closest to the passenger is most relevant to the passenger, and the more relevant the marker trajectory is, the higher the accuracy of the subsequent detection results will be; therefore, it is necessary to select the marker closest to the passenger.

[0088] S3. Determine the abnormal behavior detection result of the passenger based on the similarity between the passenger trajectory and the marker trajectory.

[0089] The similarity refers to the degree of similarity between trajectories.

[0090] Specifically, the marker trajectory associated with the passenger trajectory is determined, and then the similarity between the passenger trajectory and the marker trajectory associated with the passenger trajectory is judged, and the abnormal behavior detection result of the passenger is obtained according to the similarity judgment result.

[0091] See Figure 7 , Figure 7 1 is a flow chart of an embodiment of step S3, wherein step S3 includes:

[0092] S31, obtaining a discrete passenger position sequence corresponding to the passenger trajectory and a discrete marker position sequence corresponding to the marker trajectory;

[0093] The passenger trajectory is composed of a discrete sequence of passenger positions, and the marker trajectory is composed of a discrete sequence of marker positions.

[0094] In some embodiments, a passenger trajectory is a discrete sequence of passenger positions, and a marker trajectory is a discrete sequence of marker positions. For a passenger trajectory, the image features of the passenger corresponding to the detection target acquired at each moment correspond to the passenger position sequence, while the image features acquired at all moments correspond to the discrete passenger position sequence. Therefore, the passenger trajectory is composed of a discrete sequence of passenger positions. For a marker trajectory, the image features of the escalator markers acquired at each moment correspond to the marker position sequence, while the image features acquired at all moments correspond to the discrete marker position sequence. Therefore, the marker trajectory is a discrete sequence of marker positions.

[0095] Optionally, a discrete passenger position sequence is obtained by counting the coordinate center points of the passengers in the image frame sequence as a discontinuous discrete sequence; a discrete marker position sequence is obtained by counting the coordinate center points of the markers associated with the passengers as a discontinuous discrete sequence.

[0096] S32. Calculating the similarity between the passenger trajectory and the marker trajectory based on the passenger position sequence and the marker position sequence;

[0097] The similarity refers to the degree of similarity between trajectories.

[0098] Specifically, the marker trajectory associated with the passenger trajectory is determined, and then the similarity between the passenger trajectory and the marker trajectory associated with the passenger trajectory is judged.

[0099] The similarity is represented by the trajectory similarity distance between the discrete passenger position sequence and the discrete marker position sequence. The trajectory similarity distance and similarity are negatively correlated, that is, the larger the trajectory similarity distance, the smaller the similarity, and conversely, the smaller the trajectory similarity distance, the greater the similarity. The trajectory similarity distance is calculated using the trajectory similarity measurement method.

[0100] In some embodiments, the similarity between the passenger trajectory and the marker trajectory can be calculated based on the passenger position sequence and the marker position sequence, that is, the discrete passenger position sequence and the discrete marker position sequence in the image frame sequence are determined, and the similarity of the discrete passenger position sequence and the discrete marker position is judged, thereby obtaining the abnormal behavior detection result of the passenger.

[0101] In some embodiments, the similarity can be calculated using a shape-based method, such as Frechet, Hausdorff, etc.; wherein, Frechet corresponds to Fréchet distance (Fréchet Distance), Hausdorff corresponds to Hausdorff distance (Hausdorff Distance), that is, the similarity can be characterized by the Hausdorff distance between the passenger position sequence and the marker position sequence; wherein, the larger the Hausdorff distance, the smaller the similarity. The Hausdorff distance is a distance defined between any two sets in a metric space, which refers to the longest distance that an opponent must travel from one point in one of the two groups to the other group. In other words, the Hausdorff distance is the largest of all distances from a point in one set to the nearest point in the other set;

[0102] For example, let the discrete passenger position sequence corresponding to the passenger trajectory be X = (x1,…,x N ), and set the discrete marker position sequence corresponding to the marker trajectory associated with the passenger trajectory to be Y = (y1,…,y N ), then the Hausdorff distance between two sequences is:

[0103] H(X,Y)=max(h(X,Y),h(Y,X)),

[0104] h(X,Y)=max(x∈X)min(y∈Y)||xy||,

[0105] h(Y,X)=max(x∈Y)min(y∈X)||xy||,

[0106] Where h(X,Y) is the one-way Hausdorff distance from the passenger position sequence X to the marker position sequence Y, h(Y,X) is the one-way Hausdorff distance from the marker position sequence Y to the passenger position sequence X, and ||*|| is the distance of the calculation set, which can optionally be the Euclidean distance.

[0107] Then, if the Hausdorff distance is smaller, the two trajectories are more similar, that is, the passenger has not behaved abnormally; if the Hausdorff distance is larger, the two trajectories are more dissimilar, that is, the passenger has behaved abnormally.

[0108] In some embodiments, similarity can also be calculated using point-based methods such as EDR, LCSS, and DTW. EDR stands for Edit Distance on Real Sequence (EDR), and Longest Common Sub-Sequence (LCS) can be used. DTW (Dynamic Time Warping) can calculate the similarity between two time series and is particularly suitable for time series of varying lengths and tempos. DTW automatically warps the time series, performing local scaling on the time axis to make the two sequences as consistent as possible, achieving the greatest possible similarity.

[0109] In some embodiments, similarity may also be calculated using a segmentation-based approach, such as One Way Distance, LIP distance, etc., wherein One Way Distance is a one-way distance, and LIP distance is a locality in-between polylines (LIP).

[0110] In some embodiments, similarity calculation can also adopt task-specific methods: TRACLUS, RoadNetwork, grid, etc.

[0111] In some embodiments, the similarity calculation may further include calculating whether two discrete trajectories conform to the same Gaussian distribution to determine whether they have the same trajectories.

[0112] In some embodiments, the similarity can also be determined by the area enclosed by the passenger trajectory and the marker trajectory of the same length, that is, the larger the area, the greater the distance between the trajectories, and the lower the similarity; conversely, the smaller the area, the closer the distance between the trajectories, and the higher the similarity.

[0113] S33. Obtain the abnormal behavior detection result of the passenger based on the similarity.

[0114] Among them, after obtaining the similarity between the passenger trajectory and the marker trajectory associated with the passenger trajectory, it is possible to determine whether abnormal behavior has occurred based on the trajectory similarity. For example, the smaller the similarity, the greater the probability of abnormal behavior, and the greater the similarity, the smaller the probability of abnormal behavior.

[0115] In some embodiments, in response to the similarity between the passenger trajectory and the marker trajectory being less than or equal to a preset similarity threshold, it is determined that the passenger has abnormal behavior.

[0116] Among them, the preset similarity threshold is used to judge the similarity between the passenger trajectory and the marker trajectory.

[0117] Specifically, a similarity threshold is preset. When the similarity obtained between the passenger trajectory and the marker trajectory is less than or equal to the preset similarity threshold, it can be determined that the passenger trajectory and the marker trajectory have little similarity, that is, the passenger has engaged in abnormal behavior.

[0118] See Figure 8 , Figure 8 This is a flow chart of an embodiment of the present invention for classifying abnormal behaviors, including:

[0119] Y1. Extracting a regional image of the area where the passenger with abnormal behavior is located from the image frame sequence;

[0120] Among them, the area where passengers with abnormal behavior are located is the abnormal area.

[0121] Specifically, the area where the passengers with abnormal behaviors are located is first determined as the abnormal area, and then the regional image of the abnormal area is extracted from the image frame sequence.

[0122] Y2. Use the pre-trained classification network to classify the regional image to obtain the final abnormal behavior detection result.

[0123] Among them, the pre-trained classification network is trained by pre-defined abnormal behavior images and negative sample images, and is a multi-classification network.

[0124] Specifically, the abnormal behavior type is pre-set, and a classification network is obtained by training according to the pre-set abnormal behavior type. The regional image of the acquired abnormal area is input into the classification network for classification to obtain the corresponding abnormal behavior classification, that is, the final abnormal behavior detection result is obtained.

[0125] In some embodiments, the pre-set abnormal behavior types may include falling, bending over, climbing over the escalator, running in the direction of the escalator, running in the opposite direction of the escalator, and other abnormal behaviors.

[0126] Different from the existing technology, in this embodiment, a sequence of image frames of the area where the escalator is located is obtained; the passenger and the marker are respectively extracted and tracked from the image frame sequence to obtain the passenger trajectory and the marker trajectory, wherein the marker is set on the escalator and moves synchronously with the escalator; the abnormal behavior detection result of the passenger is obtained based on the similarity between the passenger trajectory and the marker trajectory. That is, the present application obtains the passenger trajectory and the marker trajectory respectively through the image frame sequence obtained in the area where the escalator is located, and calculates the similarity between the passenger trajectory and the marker trajectory, and obtains the abnormal behavior detection result of the passenger based on the similarity calculation result between the passenger trajectory and the marker trajectory, thereby improving the detection accuracy.

[0127] See also Figure 9 , Figure 9This is a schematic diagram of the structure of an embodiment of the escalator passenger abnormal behavior detection system of the present invention. This system can execute the steps of the above-mentioned escalator passenger abnormal behavior detection method. For relevant details, please refer to the detailed description of the above-mentioned method and will not be repeated here.

[0128] The escalator passenger abnormal behavior detection system 200 includes: a sequence acquisition module 210, a trajectory acquisition module 220, and a detection result acquisition module 230, wherein the sequence acquisition module 210 is used to obtain an image frame sequence of the area where the escalator is located; the trajectory acquisition module 220 is used to extract and track passengers and markers from the image frame sequence respectively to obtain passenger trajectories and marker trajectories, wherein the markers are set on the escalator and move synchronously with the escalator; the detection result acquisition module 230 obtains the passenger abnormal behavior detection result based on the similarity between the passenger trajectory and the marker trajectory.

[0129] See Figure 10 , Figure 10 FIG2 is a schematic diagram of another embodiment of the escalator passenger abnormal behavior detection system of the present invention. In this embodiment, the escalator passenger abnormal behavior detection system 200 further includes a classification module 240, which is configured to classify the regional image of the area where the passengers exhibiting abnormal behavior are located to obtain a final abnormal behavior detection result.

[0130] See also Figure 11 , Figure 11 This is a schematic diagram of the structure of an embodiment of an electronic device according to the present invention. This electronic device can execute the steps of the aforementioned method for detecting abnormal escalator passenger behavior. The electronic device 300 includes a memory 310 and a processor 320 coupled to the memory. The memory 310 stores at least one computer program. When the at least one computer program is loaded and executed by the processor, it implements the aforementioned method for detecting abnormal escalator passenger behavior.

[0131] See also Figure 12 , Figure 12 The computer-readable storage medium 400 stores at least one program 410, which, when loaded and executed by a processor, is used to implement the above-mentioned method for detecting abnormal behavior of escalator passengers.

[0132] The above scheme can reduce background interference, especially the impact of false target alarms, by obtaining a sequence of image frames of the area where the escalator is located; the passengers and markers are extracted and tracked from the image frame sequence respectively to obtain passenger trajectories and marker trajectories. Among them, the markers are set on the escalator and move synchronously with the escalator, which can enhance the continuity of the target trajectory and weaken the impact of missed target detection; the abnormal behavior detection results of passengers are obtained based on the similarity between the passenger trajectory and the marker trajectory, and the accuracy of detection can be improved based on the similarity calculation.

[0133] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0134] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of this embodiment.

[0135] In addition, the functional units in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0136] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0137] The above description is only an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for detecting abnormal behavior of escalator passengers, characterized in that: The method comprises: Obtain a sequence of image frames of the area where the escalator is located; Extracting and tracking the passenger and the marker from the image frame sequence to obtain a passenger trajectory and a marker trajectory, wherein the marker is set on the escalator; Based on the similarity between the passenger trajectory and the marker trajectory, the abnormal behavior detection result of the passenger is determined; wherein, the passenger trajectory is a discrete passenger position sequence, and the marker trajectory is a discrete marker position sequence. The similarity between the passenger trajectory and the marker trajectory is calculated based on the discrete passenger position sequence and the discrete marker position sequence, and the similarity is characterized by the trajectory similarity distance between the discrete passenger position sequence and the discrete marker position sequence; the trajectory similarity distance and the similarity are negatively correlated; the trajectory similarity distance is calculated and obtained by a trajectory similarity measurement method.

2. The method according to claim 1, characterized in that The extracting and tracking the passenger and the marker from the image frame sequence respectively includes: Target extraction is performed on the passenger using the passenger's head and / or shoulders as targets.

3. The method according to claim 2, characterized in that There are at least two markers and marker tracks corresponding to the markers; Before obtaining the abnormal behavior detection result of the passenger based on the similarity between the passenger trajectory and the marker trajectory, the method further includes: The marker trajectory associated with the passenger is filtered out from at least two of the marker trajectories.

4. The method according to claim 3, characterized in that The step of selecting the marker track associated with the passenger from the at least two marker tracks includes: Expanding the passenger's detection frame; The marker track corresponding to the marker located within the expanded detection frame is selected as the marker track associated with the passenger.

5. The method according to claim 4, characterized in that The selecting the marker track corresponding to the marker located within the expanded detection frame as the marker track associated with the passenger includes: In response to there being at least two markers within the expanded detection frame, the marker trajectory corresponding to the marker that has been within the expanded detection frame for the longest time or the marker that is closest to the passenger is selected as the marker trajectory associated with the passenger.

6. The method according to claim 1, characterized in that The determining of the abnormal behavior detection result of the passenger based on the similarity between the passenger trajectory and the marker trajectory includes: In response to the similarity between the passenger trajectory and the marker trajectory being less than or equal to a preset similarity threshold, it is determined that the passenger has abnormal behavior.

7. The method according to claim 6, characterized in that The obtaining of the abnormal behavior detection result of the passenger based on the similarity between the passenger trajectory and the marker trajectory further includes: extracting, from the image frame sequence, a region image of the region where the passenger exhibiting the abnormal behavior is located; The region image is classified using a pre-trained classification network to obtain a final abnormal behavior detection result.

8. An escalator passenger abnormal behavior detection system, characterized in that: The system comprises: A sequence acquisition module is used to acquire a sequence of image frames of the area where the escalator is located; a trajectory acquisition module, configured to extract and track the passenger and the marker from the image frame sequence to obtain the passenger trajectory and the marker trajectory, wherein the marker is provided on the escalator and moves synchronously with the escalator; A detection result acquisition module obtains the abnormal behavior detection result of the passenger based on the similarity between the passenger trajectory and the marker trajectory; wherein the passenger trajectory is a discrete passenger position sequence, and the marker trajectory is a discrete marker position sequence, and the similarity between the passenger trajectory and the marker trajectory is calculated based on the discrete passenger position sequence and the discrete marker position sequence, and the similarity is characterized by the trajectory similarity distance between the discrete passenger position sequence and the discrete marker position sequence; the trajectory similarity distance and the similarity are negatively correlated; the trajectory similarity distance is calculated and obtained by a trajectory similarity measurement method.

9. The system according to claim 8, characterized in that The system further comprises: The classification module is used to classify the regional image of the area where the passenger with abnormal behavior is located to obtain the final abnormal behavior detection result.

10. An electronic device, characterized in that: The electronic device includes a memory and a processor coupled to the memory, the memory stores at least one computer program, and when the at least one computer program is loaded and executed by the processor, it is used to implement the method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that The storage medium stores at least one program, and when the at least one program is loaded and executed by the processor, it is used to implement the method according to any one of claims 1 to 7.

Images