Method and electronic device for detecting the presence of large luggage on an escalator

Abstract

The application discloses a method and electronic equipment for detecting large luggage on escalators, which detects pedestrians and luggage in a warning area through a target detection model, matches the detected pedestrians and luggage, judges the size of handbags and backpacks to determine whether they belong to the category of large luggage, tracks the matched large luggage, delimits an initial triangular area to determine the motion trajectory of the luggage, excludes the case that pedestrians pass through or stay in the warning area, and only alarms when the pedestrian has the intention to get on the escalator, so as to guide the pedestrian to take the elevator and reduce false alarms. The application can be deployed in multiple cameras as required, reduce the burden of monitoring personnel, and strengthen the management of potential dangerous behaviors.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, in particular to a method for detecting a person carrying a large piece of luggage riding an escalator and an electronic device. BACKGROUND

[0002] When a person carrying a large piece of luggage or a baby carriage rides an escalator, the luggage may fall or roll due to inertia if the person does not hold the luggage steadily, and the luggage may also easily hit other passengers on the escalator. In addition, when the person reaches the entrance of the escalator, the luggage is difficult to move out of the escalator, which blocks the following passengers and causes congestion. Therefore, the person carrying a large piece of luggage riding an escalator may cause a safety hazard. Therefore, some stations or shopping malls do not allow a person to carry a large piece of luggage to ride an escalator.

[0003] The current method mainly relies on security personnel to observe and remind in real time, or set up monitoring personnel to monitor in real time, and issue a reminder broadcast after the human discovers an abnormality. The above safety management method needs to consume a large amount of human and material resources, and is prone to the problem of untimely reminding. SUMMARY

[0004] To solve the above technical problems, the present application provides a method for detecting a person carrying a large piece of luggage riding an escalator and an electronic device. The purpose of the present application is achieved by the following technical solutions.

[0005] A method for detecting a person carrying a large piece of luggage riding an escalator, comprising the following steps:

[0006] Obtaining an image of an entrance range of an escalator;

[0007] Detecting a person and luggage in the image respectively, and judging a minimum distance between the luggage and the person in the image; if the minimum distance is less than a preset distance, the luggage is paired with the closest person, and if not, detecting a subsequent image until the luggage is successfully paired;

[0008] Judging whether the successfully paired luggage is a large piece of luggage; if not, filtering the luggage, and if it is a large piece of luggage, taking the image in which the large piece of luggage is successfully paired with the person as an initial judgment image, then forming a triangular detection area by connecting the center point of the large piece of luggage with both ends of the entrance of the escalator, and continuously tracking the trajectory of the center point of the large piece of luggage; if the center point of the large piece of luggage is detected in the triangular detection area in a consecutive a frames of images, and the distance between the center points of the person in the first frame and the a-th frame of images is greater than a preset threshold, it is judged that the person wants to carry a large piece of luggage to ride an escalator, and an alarm is given, otherwise no alarm is given; a>2.

[0009] Further improvement, a=3.

[0010] Further improvement, if the distance between the center points of the person in the first frame and the third frame of images is greater than a preset threshold, an alarm is given.

[0011] Further improvement, the method for judging whether the luggage is large luggage is as follows:

[0012] If the luggage type is detected as a suitcase, it is considered as large luggage; if the luggage type is not a suitcase, it is judged whether the maximum value of the length and width of the luggage detection frame is greater than the maximum value of the height of the pedestrian detection frame, b is an adjustment coefficient, b < 1, if yes, it is considered as large luggage, otherwise it is not considered as large luggage.

[0013] Further improvement, b = 1 / 3.

[0014] Further improvement, the distance between the large luggage and the pedestrian in the image is the distance from the center point of the large luggage to the center point of the pedestrian; the setting range of the preset distance is 0.2-1m.

[0015] Further improvement, set an alarm interval, that is, at least after the set alarm interval after the alarm, the next alarm is performed.

[0016] Further improvement, the alarm interval includes t1 and t2, when the large luggage type is detected as a suitcase, the alarm interval is t1; when the large luggage type is detected as not a suitcase, the alarm interval is t2; t1 > t2.

[0017] Further improvement, the pedestrians and luggage in the image are detected by a yolov5 model.

[0018] An electronic device, a memory and a processor, the memory stores a computer program, the computer program is executed by the processor, so that the processor executes the steps of the method as described above.

[0019] The beneficial effects of the present application are:

[0020] The present application can detect pedestrians and luggage in the warning area through the target detection model, and improve the positive and negative sample selection rules of the detection model. The detected pedestrians and luggage are paired, and the size of the handbag and backpack is judged to determine whether it belongs to the category of large luggage. The paired large luggage is tracked, and an initial triangular area is drawn to determine the motion trajectory of the suitcase, to exclude the case that pedestrians pass through or stay in the warning area. Only when the pedestrian has the intention to get on the escalator, the alarm is timely, the pedestrian is guided to take the straight elevator, and the false alarm is reduced. The algorithm can be deployed in multiple cameras as needed, reducing the burden of monitoring personnel and strengthening the management of potential dangerous behaviors. BRIEF DESCRIPTION OF DRAWINGS

[0021] The present application is further illustrated by the accompanying drawings, but the contents in the drawings do not constitute any limitation on the present application.

[0022] Figure 1 It is a flowchart of the present application.

[0023] Figure 2a Sample diagram for original yolov5 model;

[0024] Figure 2b Sample diagram for yolov5 model of the application;

[0025] Figure 3 Diagram for pairing of pedestrians and luggage;

[0026] Figure 4a Diagram for setting initial judgment triangular region for large luggage;

[0027] Figure 4b Diagram for detecting continuous movement of large luggage within the set initial judgment triangular region;

[0028] Figure 4c Diagram for detecting continuous movement of large luggage only through the initial judgment triangular region;

[0029] Figure 4d Diagram for detecting large luggage as a suitcase and to take the escalator in the application;

[0030] Figure 4e Diagram for detecting large luggage as a backpack and not taking the escalator in the application;

[0031] Figure 5 Diagram for alarm process. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below in combination with the drawings and examples.

[0033] The technical solutions in the embodiments of the application will be described clearly and completely below in combination with the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0034] In the description of the present application, it needs to be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can be explicitly or implicitly included one or more features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise explicitly specified.

[0035] In the present application, the word "exemplary" is used to mean "serving as an example, instance, or illustration." Any implementation described as "exemplary" in the present application is not necessarily to be construed as preferred or advantageous over other implementations. The following description is presented to enable any person skilled in the art to make and use the present application. In the following description, for the purposes of explanation, details are set forth in order to provide a thorough understanding of the present application. It will be apparent to one skilled in the art that the present application can be practiced without using these specific details. In other instances, well-known structures and processes are not described in detail in order to avoid obscuring the description of the present application. Thus, the present application is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0036] The present application provides a detection method for a passenger carrying a large piece of luggage on an escalator in a designated area. According to the detection result, the voice prompt is triggered in time and pushed to the monitoring platform according to the alarm interval, so as to improve the real-time and accuracy of the safety monitoring in the station. The overall flowchart of the present application is shown in Figure 1

[0037] The method for detecting a passenger carrying a large piece of luggage on an escalator comprises the following steps:

[0038] Step S101, every set frame number, the target image in the monitoring video stream is acquired.

[0039] First, the video is shot by a fixedly installed camera. The camera needs to face the escalator entrance and exit, and the warning area needs to be divided on the flat ground at the escalator entrance and exit.

[0040] ​Then the image to be detected is extracted from the monitored video stream. In this step, the image of each frame can be extracted for real-time detection, or a certain frame interval can be set for image extraction operation of the video stream, and the frame interval should not exceed 5 frames.

[0041] In step S012, the pedestrian detection model and the luggage detection model detect pedestrians and luggage respectively.

[0042] First, a large number of pedestrian samples and luggage samples of track scenes are taken as training samples to complete the training of the neural network model, wherein the luggage samples include three categories of luggage, such as suitcases, handbags and backpacks, in various colors, shapes and sizes. Among them, the original positive and negative sample assignment rule of yolov5 is improved. In the original yolov5, two grid cells in addition to the center point grid cell are selected according to the coordinates of the center point of the real data. Figure 2a As shown in the following formula (1), if the coordinates of the center point are more biased towards the upper and left sides, the preset boxes of the upper and left grid cells are selected as positive samples.

[0043] However, the contribution of the additional selected grid cells should not be completely equal, but related to the distance. Generally, the closer the distance, the greater the contribution. Here, the distance of the center point in the yolov5 detection grid in four directions (i.e. the vertical distance of four changes) is set as l top , l down , l left , l right As shown in the following formula (2), when the direction distance is greater than 0.5, the grid cell in the direction is selected, and the direction distance value is taken as the weight of the positive sample classification loss in the current grid cell. Figure 2b For example Figure 2b The classification loss corresponding to the upper left grid cell is calculated as:

[0044] loss=l top (y top ×lna top +(1-y top )×ln(1-a top ))

[0045] Wherein, a top is the prediction value of the network, and y top is the true value. That is, the classification loss of the existing yolov5 model is improved to obtain more accurate detection boxes.

[0046] The above modified yolov5 is used for model training, and the trained pedestrian detection model and luggage detection model perform target detection on the obtained images to obtain the detection results of pedestrians and luggage, including detection box size information and detection box center point coordinate information.

[0047] S103, the pedestrians and the luggage are paired by the shortest principle through the center point, and the pedestrians carrying the luggage and the respective detection frame sizes are determined:

[0048] Pairing of pedestrians and luggage:

[0049] The detection results outside the region are deleted according to the range of the warning region. For each luggage ID newly generated in the region and the luggage ID that is not successfully paired in the last frame, pairing with a newly generated pedestrian ID is needed, the distance between it and the center point of the pedestrian detection frame is calculated, the shortest distance is taken as the matching distance, and when the matching distance is greater than the set distance threshold, it is considered that the luggage is carried by the pedestrian corresponding to the matching distance. That is:

[0050]

[0051] Wherein, L is the ID set of all luggage to be paired, P is the ID set of all pedestrians to be paired, represents the distance between the luggage with ID i and the pedestrian with ID j. l1 is the set distance threshold, when the matching distance of the luggage is greater than l1, it belongs to the luggage without claiming. According to the above formula, the pedestrian matched by each luggage is found, and the luggage without claiming waits for the matching of the next frame. The luggage ID and the pedestrian ID of the successful pairing are stored.

[0052] The specific steps of pairing are as shown in Figure 3

[0053] Step S1031, new luggage in the warning region and luggage i=1 that is not paired in the last frame.

[0054] Step S1032, center point distance calculation is performed with all pedestrians in the warning region, and the minimum distance between the pedestrian j is obtained.

[0055] Step S1033, it is judged whether the minimum distance is less than the distance threshold, if yes, step S1033 is performed, otherwise, the luggage is determined as the luggage without claiming, and is added to the luggage set to be matched in the next frame, and continues to be paired in the next frame.

[0056] Step S1034, it is determined that the luggage i is carried by the pedestrian j.

[0057] Step S1035, the luggage ID and the pedestrian ID information of the successful pairing are stored.

[0058] Step S1036, i=i+1, and the above steps are repeated.

[0059] Step S104, it is judged whether the luggage is a luggage box, that is, whether the luggage detection category obtained by yolov5 model detection determines whether the luggage is a luggage box:

[0060] ​Based on the inspection results, it will be determined whether the luggage qualifies as a suitcase. All suitcases are considered large luggage and should not be taken onto escalators. For handbags and backpacks, their size needs to be assessed to confirm whether they qualify as large luggage.

[0061] Step S105: For handbags and backpacks, compare the dimensions of the baggage inspection frame and the pedestrian inspection frame. Baggage whose length or width is greater than 1 / 3 of the person's length is considered large baggage; baggage smaller than 1 / 3 can be carried onto the escalator.

[0062] This involves determining the size of luggage. For handbags and backpacks successfully paired with pedestrians within the warning area, the dimensions of their detection frames and the dimensions of the paired pedestrian detection frames are confirmed. If the maximum length and width of the luggage detection frame are greater than one-third of the maximum value of the pedestrian detection frame, the luggage is considered large.

[0063]

[0064] Among them, L w and L h P represents the width and length of the inspection frame for handbags and backpacks. w and P h This indicates the width and length of the corresponding pedestrian detection box. 1 / 3 is an empirical value and can be adjusted according to the requirements of different locations, such as setting it to 1 / 2.

[0065] After the above category and size determination, the baggage ID and pedestrian ID of all large baggage are stored, and no further determination will be made.

[0066] Step S106: Filter out large luggage items that need to be tracked and their matching pedestrians by size comparison, record their initial center point coordinates, and form a judgment triangle for the luggage.

[0067] For each newly matched large piece of luggage and pedestrian in each frame, an initial judgment triangle region is set for each large piece of luggage, formed by connecting the center point of the large piece of luggage detection box in the current frame to the two ends of the escalator entrance, such as... Figure 4a The gray triangle area is shown in the image. Due to situations where pedestrians carrying large luggage pass through the detection area, such as... Figure 4c As shown. In order to avoid frequent and false alarms, it is necessary to roughly determine the movement trajectory of large luggage. An alarm should be triggered for large luggage moving in the direction of the escalator, i.e., step S107.

[0068] Step S107: For the same large piece of luggage, if the center point of three consecutive frames is within the triangular area during the tracking process, and the distance between the center points of the first and third frames is greater than a certain threshold (this is used to filter out pedestrians carrying large pieces of luggage who are only temporarily stopping in front of the escalator and do not intend to ride the escalator), it is considered that the pedestrian intends to ride the escalator, and an alarm is generated.

[0069] It should be noted here that, because it is necessary to exclude the situation that pedestrians carrying luggage wait near the escalator entrance, in addition to judging whether the center point of the detection box falls inside the initial judgment triangle, the distance l of the center points of the three frames of detection boxes also needs to be considered, as shown in the following formula: Figure 4b When the distance between the center points of the first frame and the third frame of detection boxes is greater than l2, an alarm is generated to remind passengers in time to avoid potential dangers. Otherwise, it is considered that the pedestrian is only staying in the warning area and does not have the intention to take the escalator at the moment, and no alarm is generated. The threshold value l2 is related to the frequency of extracting detection frames and the speed of pedestrians carrying luggage moving according to experience. The threshold value is usually set to be in the range of 0.3-1.2 m / s, preferably 0.3 m / s, of the moving speed of pedestrians carrying luggage per unit time. Specifically, if the frequency of extracting detection frames is 2 frames / s, the threshold value can be set to 0.3 m. If the frequency of extracting detection frames is 5 frames / s, the threshold value of adjacent detection frames is set to 0.12 m.

[0070] Specifically, the tracking algorithm is used to track large luggage and corresponding pedestrians. When the center points of the detection boxes of the same large luggage are located inside the initial judgment triangle for three consecutive frames, it is considered that the pedestrian has the intention to take the escalator with the large luggage, as shown in the following formula: Figure 4b

[0071] The tracking algorithm can use the ByteTrack target tracker to track pedestrians. The ByteTrack target tracker is a multi-target tracker that matches low-confidence detection bbox with tracklets. It can effectively reduce the problem of target missing caused by target occlusion, motion blur, etc., or the problem of being eliminated due to too low detection confidence, thereby reducing the Fregament of IDs and tracklets and improving the performance of the MOT tracker

[0072] Step S108, after the algorithm starts the initial alarm, different types of large luggage are alarmed at corresponding time intervals, that is, different types of luggage are set different alarm intervals:

[0073] Considering the differences between the suitcase, handbag, and backpack, a group of people carrying suitcases will usually queue up to take the escalator, especially at the escalators of subways and high-speed rails. Therefore, in order to avoid too frequent alarms in this case, a longer alarm interval t1 is set for the suitcase. The large handbag and backpack appear less frequently, and if a longer alarm interval is set, it is easy to cause a false negative. Therefore, a shorter alarm interval t2 is set for these two types. When a large piece of luggage that meets the alarm condition is detected in the warning area and the alarm interval of the corresponding type is met, the system will push the alarm information and play a prompt broadcast in time. As shown in the following formula: Figure 5 ​t1 and t2 are set according to the actual flow of people in the place, for example, in a high-speed rail station, there are usually more luggage cases, in order to avoid frequent alarms, t1 can be set to be slightly larger, such as 1-3 min, and t2 is set to 10 s. While in the mall, there are fewer luggage cases, t1 can be set to be slightly smaller, such as 0.5-1 min, and t2 can be set to be slightly larger, such as 20 s, etc.

[0074] That is, in addition to setting the alarm interval according to the category, the alarm interval can also be set according to the escalator port at different points in different environments.

[0075] The specific steps are as shown in Figure 5 :

[0076] Step S1081, detecting the large piece of luggage to be alarmed.

[0077] Step S1082, judging whether the category of the large piece of luggage is a luggage case, if yes, executing step S1083, otherwise executing step S1084.

[0078] Step S1083, judging whether the time difference from the last luggage case alarm is greater than the alarm interval t1, if no, executing step S1085, if yes, executing step S1086.

[0079] Step S1084, judging whether the time difference from the last handbag or backpack alarm is greater than the alarm interval t2, if no, executing step S1085, if yes, executing step S1086; wherein, the large piece of luggage is not a luggage case, but a handbag or a backpack, so the alarm interval t2 is used for judgment.

[0080] Step S1085, ending, that is, not alarming;

[0081] Step S1086, alarming, recording the alarm time and the alarm category, that is, judging whether the large piece of luggage is a luggage case or a handbag or a backpack when alarming.

[0082] Although the embodiments of the present application have been shown and described, it can be understood by those skilled in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for detecting carrying large luggage on an escalator, characterized in that: Includes the following steps: Obtain an image of the area around the escalator entrance; The system detects pedestrians and luggage in the images separately and determines the minimum distance between the luggage and the pedestrians in the image. If the minimum distance is less than a preset distance, the luggage is paired with the nearest pedestrian. Otherwise, the system detects the luggage in the subsequent images until the luggage is successfully paired. Determine whether the successfully paired luggage is large luggage; otherwise, filter the luggage; if it is large luggage, use the image of the large luggage successfully paired with the pedestrian as the initial judgment image, then connect the center point of the large luggage with the two ends of the escalator entrance to form a triangular detection area, and continuously track the trajectory of the center point of the successfully paired large luggage. If the center point of a large piece of luggage is detected in all consecutive a-frame images within the triangular detection area, and the distance between the pedestrian center points in the first and a-th frames of the a-frame images is greater than a preset threshold, then it is determined that the pedestrian is carrying a large piece of luggage on the escalator and an alarm is triggered; otherwise, no alarm is triggered; a≥2.

2. The method for detecting carrying large luggage on an escalator as described in claim 1, characterized in that: a=3。 3. The method for detecting carrying large luggage on an escalator as described in claim 2, characterized in that: An alarm will be triggered if the distance between the center points of pedestrians in the first and third frames of the image is greater than a preset threshold.

4. The method for detecting carrying large luggage on an escalator as described in claim 1, characterized in that: Here are some methods to determine if luggage is large or bulky: If the detected luggage type is a suitcase, it is considered large luggage; if the luggage type is not a suitcase, it is determined whether the maximum value of the length and width of the luggage detection frame is greater than the maximum value of b * the height of the pedestrian detection frame, where b is an adjustment coefficient and b < 1. If so, it is considered large luggage; otherwise, it is not considered large luggage.

5. The method for detecting carrying large luggage on an escalator as described in claim 4, characterized in that: b=1 / 3。 6. The method for detecting carrying large luggage on an escalator as described in claim 4, characterized in that: The distance between large luggage and pedestrians in the image is the distance from the center point of the large luggage to the center point of the pedestrian; the preset distance setting range is 0.2-1m.

7. The method for detecting carrying large luggage on an escalator as described in claim 1, characterized in that: Set an alarm interval, meaning that at least the set alarm interval will elapse after an alarm is triggered before the next alarm is triggered.

8. The method for detecting carrying large luggage on an escalator as described in claim 7, characterized in that: The alarm intervals include t1 and t2. When a large piece of luggage is detected as a suitcase, the alarm interval is t. 1； When the type of large luggage detected is not a suitcase, the alarm interval is t2; t1 > t 2。 9. The method for detecting carrying large luggage on an escalator as described in claim 7, characterized in that: Pedestrians and luggage in the images were detected using the YOLOv5 model.

10. An electronic device, characterized in that, A memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1-9.

Images