Depth learning-based armrest-unholding behavior detection system

Abstract

The invention provides a system for detecting a behavior of not holding a handrail based on deep learning, and the system comprises a deep learning calculation module which is used for carrying out the detection of a person from an image transmitted by a camera based on a pre-trained person detection model, and carrying out the multi-dimensional feature analysis of the person based on a person detection result; the handrail state analysis module is used for carrying out handrail judgment on the personnel based on the multi-dimensional characteristic analysis result of the personnel in combination with a preset stair area and a handrail area; and the alarm reminding module is used for carrying out targeted voice broadcast reminding on the personnel on the current stairs when the personnel are determined not to hold the handrail based on the handrail holding judgment result. The use safety of the stairs is effectively improved, the falling accident risk is reduced, automatic and real-time monitoring and intervention are achieved, manual guarding is not needed, and the intelligent management efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a handrail non-holding behavior detection system based on deep learning. BACKGROUND

[0002] At present, the handrail of a step ladder or an escalator is a necessary component for assisting pedestrians to keep their bodies stable in the ladder area and reducing the probability of accidents, especially in the old people activity area, resident building stairs, hospital stairs, subway station, high-speed rail station and other scenes with long escalators. Pedestrians need to hold the handrail well when walking or standing in the ladder area to stabilize the body center of gravity and reduce the possibility of personal injury caused by falling, crowding and other factors. At present, most of the stair scenes do not have effective handrail monitoring and reminding means. In a few key areas, specific staff are arranged for supervision, or loudspeakers are placed to play reminder sounds to remind pedestrians. Arranging staff for supervision has problems such as wasting manpower and being unable to focus on supervision for a long time. The problem of playing reminder sounds is that it lacks pertinence, is not easy to attract the attention of pedestrians, and the reminding strength is insufficient. At the same time, hand detection and posture estimation are not comprehensively considered, and analysis errors caused by inaccurate posture estimation cannot be reduced. Therefore, in order to overcome the above defects, the present application provides a handrail non-holding behavior detection system based on deep learning. SUMMARY

[0003] The present application provides a handrail non-holding behavior detection system based on deep learning, which intelligently analyzes the images captured by the camera in real time through deep learning, can accurately identify the behavior state of the personnel in the stair area, and judges whether the handrail is held according to the obtained behavior state of the personnel. When it is detected that the personnel do not hold the handrail, a voice reminder is triggered immediately, thereby effectively improving the safety of stair use, reducing the risk of falling accidents, realizing automatic and real-time monitoring and intervention, and improving the intelligent management efficiency without manual attendance.

[0004] The present application provides a handrail non-holding behavior detection system based on deep learning, which intelligently analyzes the images captured by the camera in real time through deep learning, can accurately identify the behavior state of the personnel in the stair area, and judges whether the handrail is held according to the obtained behavior state of the personnel. When it is detected that the personnel do not hold the handrail, a voice reminder is triggered immediately, thereby effectively improving the safety of stair use, reducing the risk of falling accidents, realizing automatic and real-time monitoring and intervention, and improving the intelligent management efficiency without manual attendance. The deep learning computing module is used for personnel detection based on a pre-trained personnel detection model and personnel multi-dimensional feature analysis based on the personnel detection result. The handrail holding state analysis module is used for handrail holding judgment of the personnel based on the personnel multi-dimensional feature analysis result and the pre-set stair area and handrail area. The alarm reminding module is used for targeted voice broadcast reminding of the personnel on the current stairs when it is determined that the personnel do not hold the handrail based on the handrail holding judgment result.

[0005] Preferably, a deep learning-based system for detecting unattended handrail behavior includes a deep learning computing module comprising: The image acquisition unit is used to dock with the camera, receive the images transmitted from the camera in real time based on the docking result, and divide the images transmitted from the camera into frames based on the time series to obtain the image sequence; The personnel detection unit is used to sequentially input the images from the camera into a pre-trained personnel detection model based on the image sequence, and to extract features from the images based on the pre-trained personnel detection model to determine the target area where the personnel in the image are located. The personnel location determination unit is used to lock personnel in the target area based on the minimum bounding rectangle, and to determine the coordinate information [x1, y1, x2, y2] of the minimum bounding rectangle based on the locking result and the coordinate system to which the image belongs. Here, x1 is the x-coordinate of the upper left corner of the minimum bounding rectangle, y1 is the y-coordinate of the upper left corner of the minimum bounding rectangle, x2 is the x-coordinate of the lower right corner of the minimum bounding rectangle, and y2 is the y-coordinate of the lower right corner of the minimum bounding rectangle.

[0006] Preferably, a deep learning-based system for detecting unattended handrail behavior includes a deep learning computing module comprising: The result acquisition unit is used for: The personnel detection results are obtained, and when personnel are present, the detected personnel are tracked based on a preset algorithm to obtain the position coordinates of the personnel in different image frames; A unique index number is assigned to each person based on the location tracking results, and an information storage queue is assigned to each person based on the unique index number. Store the location coordinates of personnel in different image frames into the information storage queue under the corresponding unique index number; The multi-dimensional feature analysis unit is used for: Based on the personnel location tracking results, a local image of the personnel's location is captured from the current camera feed. Then, pre-trained personnel hand detection models, pre-trained pose estimation models, pre-trained face detection models, and pre-trained personnel attribute analysis models are retrieved to perform corresponding business feature analysis on the local image of the personnel's location, resulting in multi-dimensional features. These multi-dimensional features include the coordinates of the personnel's hand location, the position information of human skeleton points, the position coordinates of the face, and the personnel attribute information. The information storage unit is used to store the obtained multi-dimensional features into the information storage queue of the corresponding personnel.

[0007] Preferably, a deep learning-based system for detecting unhandrail behavior includes a handrail status analysis module, comprising: The object locking unit is used to acquire the image transmitted from the camera and perform object locking on the image transmitted from the camera based on the preset appearance characteristics of the handrail and stairs to obtain the handrail and stairs; Regional delineation units are used for: Based on the object locking result, the four corner points of the handrails on both sides in the image transmitted by the camera are firstly calibrated, and the four corner points of each side are sequentially connected based on the first calibration result to obtain the first quadrilateral, which is the handrail area on both sides. Meanwhile, based on the object locking result, the corner points on both sides of the top step and the corner points on both sides of the bottom step of the stairs are second-calibrated, and the four corner points are sequentially connected based on the second calibration result to obtain a second quadrilateral, which is the staircase area.

[0008] Preferably, a deep learning-based system for detecting unattended handrail behavior includes a region delineation unit comprising: The result acquisition sub-unit is used to acquire the obtained handrail area, and select the center points A and B of the handrail areas on both sides respectively, and connect the center points A and B to obtain the line segment AB; Centerline determination unit, used for: Determine the center point of each side of the four sides of the staircase area, and connect the center points of opposite sides of the quadrilateral to obtain the center lines CD and EF of the staircase area; Calculate the angle values ​​of AB and CD using the following formula: ; Calculate the angle values ​​of AB and EF based on the following formula; ; If the calculation yields Less than If so, CD will be used as the center line of the staircase; otherwise, EF will be used as the center line of the staircase. The direction perpendicular to the center line of the staircase is determined as the direction of pedestrian movement.

[0009] Preferably, a deep learning-based system for detecting unhandrail behavior includes a handrail status analysis module, comprising: The hand detection and judgment unit is used for: Obtain the results of multi-dimensional feature analysis of people in the image input from the camera, and when a person's hand is detected in the multi-dimensional feature analysis results, determine the coordinates of the rectangular frame of the person's hand based on the multi-dimensional feature analysis results; The interaction ratio between the hand rectangle and the handrail areas on both sides is calculated based on the coordinates of the hand rectangle. The formula for calculating the interaction ratio between a single hand and a single handrail is as follows: ; When any IOU value is greater than the preset interaction threshold, it is determined that the person in the current frame image has held the handrail, and the determination result and the current frame image are stored in the person's information storage queue. The attitude estimation and judgment unit is used for: When no hand is detected in the multi-dimensional feature analysis of personnel or the interaction ratio is less than or equal to the preset interaction threshold, extract the position information of the personnel skeleton points in the multi-dimensional feature analysis of personnel. Based on the location information of the person's skeletal points, determine whether the representative points of the hand extending from the key points of the person's elbow and wrist are located within the handrail area to determine whether the person is holding the handrail.

[0010] Preferably, a deep learning-based system for detecting unattended handrail behavior includes a posture estimation and judgment unit comprising: The hand coordinate determination sub-unit is used for: The obtained skeletal point position information of the person is used, and the elbow point on the same side as the handrail area is set as G, and the wrist point is set as H. The coordinates of G are determined based on the image coordinates. The coordinates of H and H ]; The coordinates of the extended hand representative point K are determined based on the preset extension coefficient obtained from the management terminal, combined with the elbow point G and the wrist point H. The specific calculation formula is as follows, where the preset extension coefficient is 0.4: ; ; based on and Obtain the coordinates of the extended hand representative point K, and determine the positional relationship between the coordinates of K and the handrail area on the same side; If the coordinates of K are within the same side of the handrail area, it is determined that the person in the current frame image has held the handrail, and the determination result is stored in the current person's information storage queue. Otherwise, it is determined that the current person is not holding the handrail with their current side hand. At the same time, if it is determined that the current person is not holding the handrail with either side hand, it is finally determined that the person in the current frame image is not holding the handrail, and the determination result is stored in the current person's information storage queue.

[0011] Preferably, a deep learning-based system for detecting unhandrail behavior includes a centerline determination unit, comprising: The result determines the sub-unit, used for: People are tracked based on images transmitted from cameras, and when a person crosses the center line of the stairwell area, the information storage queue of the current person is accessed. Based on the access results, the number of people who are judged not to be holding the handrail in different frame images is determined. At the same time, based on the access results, the total number of status records is determined. When the proportion of the number of people who are not holding the handrail in the total number of status records exceeds a preset threshold, it is finally determined that the current person is not holding the handrail; otherwise, it is finally determined that the current person is holding the handrail.

[0012] Preferably, a deep learning-based system for detecting unhandrail behavior includes a result determination subunit, comprising: Information access subunit, used for: Access the information storage queue of the current person and extract the face detection results of the current person based on the access results; If the face detection results contain frontal facial information of the current person, it is determined that the current person is walking in the direction of the camera; otherwise, it is determined that the current person is walking with their back to the direction of the camera. The parameter adjustment subunit is used for: When it is determined that a person is walking with their back to the camera's shooting direction, the degree of interference between walking with their back to the camera and the determination of the behavior of not holding the handrail is determined based on historical observation data. The correction parameters for the preset threshold are determined based on the degree of interference, and the preset threshold is adjusted based on the correction parameters.

[0013] Preferably, a deep learning-based system for detecting unattended handrail behavior includes an alarm reminder module comprising: The status determination unit is used to obtain the judgment result of the person holding the handrail. When it is determined that the person is not holding the handrail, the personnel attribute information of the current person is retrieved based on the current person's information storage queue. The reminder scheme generation unit is used for: The personnel attribute information is parsed to extract the personalized attribute features of the current personnel, and the personalized attribute features of the current personnel are sorted by voice broadcast based on the preset feature sorting rules. Based on the ranking results of voice broadcasts, the voice conversion program is invoked to convert personalized attribute features into broadcast language, and the voice of the broadcast object is obtained based on the conversion results. At the same time, the preset broadcast voice library is accessed, the target reminder voice is retrieved, and the broadcast target voice is combined with the target reminder voice to obtain the target reminder scheme; Alarm notification unit, used for: The system tracks the location of the person in real time based on the image transmitted from the camera, and when the person enters the reminder area of ​​the voice broadcasting device, the system plays the target reminder plan to the person based on the voice broadcasting device. Based on the playback results, targeted voice reminders are given to people on the stairs who are not holding the handrail.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: By using deep learning to intelligently analyze images captured in real time by cameras, the system can accurately identify the behavioral status of people in the stairwell area and determine whether they are holding the handrail based on their behavior. When a person is detected not holding the handrail, a voice reminder is immediately triggered, thereby effectively improving the safety of stairwell use, reducing the risk of falls, and achieving automated, real-time monitoring and intervention without the need for manual supervision, thus improving the efficiency of intelligent management.

[0015] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in this application.

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a structural diagram of a deep learning-based system for detecting unhandled handrail behavior in an embodiment of the present invention. Figure 2 This is a flowchart of a deep learning-based system for detecting unhandrail behavior in an embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the principle of determining the centerline of a staircase in a deep learning-based system for detecting unhandrailed behavior, as described in an embodiment of the present invention. Figure 4 This is a flowchart illustrating the overall process of handrail status analysis in a deep learning-based system for detecting unhandrail behavior in an embodiment of the present invention. Figure 5 This is a schematic diagram of skeletal key points in a deep learning-based system for detecting unhandled handrail behavior, as described in an embodiment of the present invention. Detailed Implementation

[0018] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0019] Example 1: This example provides a deep learning-based system for detecting the behavior of not holding onto a handrail, such as... Figure 1 As shown, it includes: The deep learning computing module is used to detect people in images input from the camera based on a pre-trained people detection model, and to perform multi-dimensional feature analysis of people based on the detection results. The handrail status analysis module is used to determine whether a person is holding a handrail based on the results of multi-dimensional feature analysis of people and the preset stair area and handrail area. The alarm and reminder module is used to provide targeted voice reminders to people on the stairs when it is determined from the handrail judgment results that a person is not holding the handrail.

[0020] In this embodiment, the personnel detection model refers to a pre-trained deep learning model used to identify and locate personnel from images.

[0021] In this embodiment, multi-dimensional feature analysis of personnel refers to determining the position of a person's hands, posture, face detection results, and personnel attribute information based on the personnel detection results, in order to support the judgment of behavioral status.

[0022] In this embodiment, the staircase area refers to a preset image area, corresponding to the actual structural range of the staircase, used to define the space for behavior analysis.

[0023] In this embodiment, the handrail area refers to a preset image area that corresponds to the actual handrail position and is used to determine whether a person is touching or approaching the handrail.

[0024] In this embodiment, the handrail judgment refers to the comprehensive analysis of whether a person is holding onto the handrail based on their characteristics and the area information.

[0025] In this embodiment, targeted voice broadcast reminders refer to issuing personalized voice warnings or prompts based on the specific location and situation of the person not holding the handrail, such as "Gentleman in white shirt, please hold the handrail and pay attention to safety."

[0026] In this embodiment, a general flowchart of a deep learning-based method for detecting unattended handrail behavior is shown below. Figure 2 As shown.

[0027] The beneficial effects of the above technical solution are as follows: by using deep learning to intelligently analyze the images captured in real time by the camera, it can accurately identify the behavioral status of people in the stairwell area and determine whether or not to hold the handrail based on the behavioral status of the people. When it is detected that a person is not holding the handrail, a voice reminder is immediately triggered, thereby effectively improving the safety of stairwell use, reducing the risk of falls, realizing automated and real-time monitoring and intervention, eliminating the need for manual duty, and improving the efficiency of intelligent management.

[0028] Example 2: Based on Example 1, this example provides a deep learning-based system for detecting unattended handrail behavior, characterized in that the deep learning computing module includes: The image acquisition unit is used to dock with the camera, receive the images transmitted from the camera in real time based on the docking result, and divide the images transmitted from the camera into frames based on the time series to obtain the image sequence; The personnel detection unit is used to sequentially input the images from the camera into a pre-trained personnel detection model based on the image sequence, and to extract features from the images based on the pre-trained personnel detection model to determine the target area where the personnel in the image are located. The personnel location determination unit is used to lock personnel in the target area based on the minimum bounding rectangle, and to determine the coordinate information [x1, y1, x2, y2] of the minimum bounding rectangle based on the locking result and the coordinate system to which the image belongs. Here, x1 is the x-coordinate of the upper left corner of the minimum bounding rectangle, y1 is the y-coordinate of the upper left corner of the minimum bounding rectangle, x2 is the x-coordinate of the lower right corner of the minimum bounding rectangle, and y2 is the y-coordinate of the lower right corner of the minimum bounding rectangle.

[0029] In this embodiment, the image sequence refers to a collection of single-frame images captured continuously by a camera, arranged in chronological order.

[0030] In this embodiment, the pre-trained person detection model refers to a deep learning model that has been trained with a large amount of data and is capable of identifying and locating the area where a person is located from an input image.

[0031] In this embodiment, the target region refers to the portion of the image that contains a complete human subject, as identified by the model.

[0032] In this embodiment, the minimum bounding rectangle refers to the smallest rectangular frame that can completely enclose the "target area" (i.e., the main body of the person) identified in the image.

[0033] In this embodiment, the coordinate information [x1, y1, x2, y2] refers to a set of data used to accurately describe the position of the "minimum bounding rectangle" in the image coordinate system, representing the pixel coordinates of the upper left and lower right corners of the rectangle, respectively.

[0034] The beneficial effects of the above technical solution are as follows: by receiving and serializing camera images in real time, a continuous and orderly data foundation can be provided for subsequent analysis. Then, the pre-trained model is used to accurately locate the personnel area in the image, and the personnel position is accurately defined by geometric methods to obtain their standardized coordinate information. This achieves stable and automatic identification and positioning of personnel targets in dynamic scenes, providing accurate and reliable data support for subsequent behavior state analysis and improving the processing efficiency and detection accuracy of the entire system.

[0035] Example 3: Based on Example 1, this example provides a deep learning-based system for detecting unattended handrail behavior. The deep learning computing module includes: The result acquisition unit is used for: The personnel detection results are obtained, and when personnel are present, the detected personnel are tracked based on a preset algorithm to obtain the position coordinates of the personnel in different image frames; A unique index number is assigned to each person based on the location tracking results, and an information storage queue is assigned to each person based on the unique index number. Store the location coordinates of personnel in different image frames into the information storage queue under the corresponding unique index number; The multi-dimensional feature analysis unit is used for: Based on the personnel location tracking results, a local image of the personnel's location is captured from the current camera feed. Then, pre-trained personnel hand detection models, pre-trained pose estimation models, pre-trained face detection models, and pre-trained personnel attribute analysis models are retrieved to perform corresponding business feature analysis on the local image of the personnel's location, resulting in multi-dimensional features. These multi-dimensional features include the coordinates of the personnel's hand location, the position information of human skeleton points, the position coordinates of the face, and the personnel attribute information. The information storage unit is used to store the obtained multi-dimensional features into the information storage queue of the corresponding personnel.

[0036] In this embodiment, the preset algorithm refers to the ByteTrack algorithm.

[0037] In this embodiment, the personnel attribute information includes personnel gender, clothing color, style, etc.

[0038] In this embodiment, the unique index number refers to a unique and unchanging number assigned to each tracked person, used to uniquely identify and distinguish different individuals in the system.

[0039] In this embodiment, the information storage queue refers to a data storage structure created for each person identified by a unique index number, used to store the person's continuous location coordinates and subsequent multi-dimensional feature data in chronological order.

[0040] In this embodiment, the partial image of a person's location refers to a partial image area that is cropped from the complete camera view based on the person's position coordinates and contains only that specific person.

[0041] In this embodiment, multi-dimensional features refer to a set of feature information about attributes such as the position of a person's hands, body posture, face position, and clothing, obtained by analyzing local images of a person's location using multiple specialized models.

[0042] The beneficial effects of the above technical solution are as follows: by continuously tracking the detected personnel and establishing an independent identity for each target, coherent analysis of the behavior of specific individuals is achieved. At the same time, by comprehensively utilizing multiple feature models such as hand, posture, face and attributes for collaborative analysis, the detailed state of personnel can be captured comprehensively and accurately. Finally, all analysis results are systematically associated and stored under the corresponding identity, providing sufficient and reliable basis for accurately judging specific behaviors such as whether to hold a handrail, significantly improving the accuracy of behavior state recognition and the intelligence level of the system.

[0043] Example 4: Based on Example 1, this example provides a deep learning-based system for detecting unhandrail behavior, including a handrail status analysis module, comprising: The object locking unit is used to acquire the image transmitted from the camera and perform object locking on the image transmitted from the camera based on the preset appearance characteristics of the handrail and stairs to obtain the handrail and stairs; Regional delineation units are used for: Based on the object locking result, the four corner points of the handrails on both sides in the image transmitted by the camera are firstly calibrated, and the four corner points of each side are sequentially connected based on the first calibration result to obtain the first quadrilateral, which is the handrail area on both sides. Meanwhile, based on the object locking result, the corner points on both sides of the top step and the corner points on both sides of the bottom step of the stairs are second-calibrated, and the four corner points are sequentially connected based on the second calibration result to obtain a second quadrilateral, which is the staircase area.

[0044] In this embodiment, the object locking result refers to the location information obtained after the image recognition algorithm accurately identifies the handrail and stairs as two specific objects in the camera image.

[0045] In this embodiment, the first calibration refers to the operation of marking the positions of four key corner points in the image to define the handrail area.

[0046] In this embodiment, the first quadrilateral refers to the quadrilateral image range enclosed by four sequentially connected corner points, used to represent the area of ​​a single-sided handrail.

[0047] In this embodiment, the second calibration refers to the operation of marking the positions of four key corner points in the image to define the staircase area.

[0048] In this embodiment, the second quadrilateral refers to the quadrilateral image range enclosed by four sequentially connected corner points, used to represent the entire staircase area.

[0049] The beneficial effects of the above technical solution are: by automatically identifying and accurately calibrating key structures in the stairwell environment, the boundary range of handrails and stairs can be accurately defined, providing a reliable spatial benchmark for subsequent judgment of whether people are in the stairwell area and whether they have contacted the handrail, effectively improving the accuracy and systematicness of behavior analysis.

[0050] Example 5: Based on Example 4, this example provides a deep learning-based system for detecting unattended handrail behavior, such as... Figure 3 As shown, the area delineation unit includes: The result acquisition sub-unit is used to acquire the obtained handrail area, and select the center points A and B of the handrail areas on both sides respectively, and connect the center points A and B to obtain the line segment AB; Centerline determination unit, used for: Determine the center point of each side of the four sides of the staircase area, and connect the center points of opposite sides of the quadrilateral to obtain the center lines CD and EF of the staircase area; Calculate the angle values ​​of AB and CD using the following formula: ; Calculate the angle values ​​of AB and EF based on the following formula; ; If the calculation yields Less than If so, CD will be used as the center line of the staircase; otherwise, EF will be used as the center line of the staircase. The direction perpendicular to the center line of the staircase is determined as the direction of pedestrian movement.

[0051] In this embodiment, line segment AB refers to a virtual straight line formed by connecting the center points of the left and right handrail areas in the image.

[0052] In this embodiment, the center lines CD and EF in the stair area refer to two intersecting virtual straight lines obtained by connecting the center points of the two pairs of opposite sides of the stair area quadrilateral.

[0053] In this embodiment, since the relative positions of the camera and the stair area and handrail area remain unchanged, there is no need to use algorithms to identify the handrail area and stair area in the image and to make a preset delineation; The staircase area and handrail area are defined by a preset quadrilateral, that is, based on the location of the staircase and handrail in the video surveillance footage. Based on the actual situation on site, the area to be analyzed is selected. Here, "based on the actual situation, the area to be analyzed is selected" means that in some scenarios, due to the camera angle, camera parameters, the camera's field of view is limited, or the staircase is too long for the camera to encompass the entire staircase and handrail area. In such cases, the area that can be captured is selected.

[0054] The beneficial effects of the above technical solution are: by automatically calculating and comparing the angle between the handrail line and the potential centerline of the stairs, the correct stair orientation and pedestrian direction can be determined intelligently, providing a key spatial reference for subsequent accurate analysis of people's behavior on the stairs (such as whether they hold the handrail), and improving the adaptability of the entire system to real-world scenarios and the accuracy of judgment.

[0055] Example 6: Based on Example 1, this example provides a deep learning-based system for detecting unattended handrail behavior, such as... Figure 4 As shown, the handrail status analysis module includes: The hand detection and judgment unit is used for: Obtain the results of multi-dimensional feature analysis of people in the image input from the camera, and when a person's hand is detected in the multi-dimensional feature analysis results, determine the coordinates of the rectangular frame of the person's hand based on the multi-dimensional feature analysis results; The interaction ratio between the hand rectangle and the handrail areas on both sides is calculated based on the coordinates of the hand rectangle. The formula for calculating the interaction ratio between a single hand and a single handrail is as follows: ; When any IOU value is greater than the preset interaction threshold, it is determined that the person in the current frame image has held the handrail, and the determination result and the current frame image are stored in the person's information storage queue. The attitude estimation and judgment unit is used for: When no hand is detected in the multi-dimensional feature analysis of personnel or the interaction ratio is less than or equal to the preset interaction threshold, extract the position information of the personnel skeleton points in the multi-dimensional feature analysis of personnel. Based on the location information of the person's skeletal points, determine whether the representative points of the hand extending from the key points of the person's elbow and wrist are located within the handrail area to determine whether the person is holding the handrail.

[0056] In this embodiment, the Interaction Ratio (IOU) refers to the ratio of the overlapping area of ​​the hand detection rectangle and the single-side handrail area rectangle to the union area of ​​the two, which is used to quantify the degree of contact between the hand and the handrail.

[0057] In this embodiment, the preset interaction threshold refers to a pre-set numerical threshold. When the calculated interaction ratio (IOU) is higher than this value, it is determined that the handrail has been held.

[0058] In this embodiment, the hand representative point refers to the coordinate point used to represent the approximate position of the hand, calculated through geometric relationships based on the positions of the two key skeletal points of the elbow and wrist of the person being detected when the hand cannot be directly detected.

[0059] The beneficial effects of the above technical solution are: by combining hand position detection and human posture analysis, it is possible to flexibly judge the handrail behavior in different situations. When the hand can be clearly detected, it can be directly judged by calculating the spatial overlap relationship between the handrail and the handrail. When the hand is obscured or not clearly detected, the possible position of the hand can be inferred through human key points for auxiliary judgment, which significantly improves the accuracy and robustness of behavior recognition in different scenarios.

[0060] Example 7: Based on Example 6, this example provides a deep learning-based system for detecting unattended handrail behavior, such as... Figure 5 As shown, the attitude estimation and determination unit includes: The hand coordinate determination sub-unit is used for: The obtained skeletal point position information of the person is used, and the elbow point on the same side as the handrail area is set as G, and the wrist point is set as H. The coordinates of G are determined based on the image coordinates. The coordinates of H and H ]; The coordinates of the extended hand representative point K are determined based on the preset extension coefficient obtained from the management terminal, combined with the elbow point G and the wrist point H. The specific calculation formula is as follows, where the preset extension coefficient is 0.4: ; ; based on and Obtain the coordinates of the extended hand representative point K, and determine the positional relationship between the coordinates of K and the handrail area on the same side; If the coordinates of K are within the same side of the handrail area, it is determined that the person in the current frame image has held the handrail, and the determination result is stored in the current person's information storage queue. Otherwise, it is determined that the current person is not holding the handrail with their current side hand. At the same time, if it is determined that the current person is not holding the handrail with either side hand, it is finally determined that the person in the current frame image is not holding the handrail, and the determination result is stored in the current person's information storage queue.

[0061] In this embodiment, the preset extension coefficient refers to a scaling factor used to extrapolate the coordinates of the representative hand point from the known coordinates of the elbow and wrist key points. Here, it is set to 0.4.

[0062] In this embodiment, the coordinates of the hand representative point K refer to a coordinate point in the image that is calculated using a preset formula based on the positions of the elbow point G and the wrist point H, and is used to represent the position of the end of the hand.

[0063] The beneficial effects of the above technical solution are: by calculating the estimated position of the hand through the coordinates of key points of the human body, even when the hand is directly obscured, it can effectively determine the relative positional relationship between the hand and the handrail, which significantly improves the accuracy of handrail behavior detection and its adaptability in complex scenarios.

[0064] Example 8: Based on Example 5, this example provides a deep learning-based system for detecting unattended handrail behavior, including a centerline determination unit, comprising: The result determines the sub-unit, used for: People are tracked based on images transmitted from cameras, and when a person crosses the center line of the stairwell area, the information storage queue of the current person is accessed. Based on the access results, the number of people who are judged not to be holding the handrail in different frame images is determined. At the same time, based on the access results, the total number of status records is determined. When the proportion of the number of people who are not holding the handrail in the total number of status records exceeds a preset threshold, it is finally determined that the current person is not holding the handrail; otherwise, it is finally determined that the current person is holding the handrail.

[0065] In this embodiment, the total number of status records refers to the total number of all image frames located in the stairwell area that are recorded for a specific person in the information storage queue.

[0066] In this embodiment, the preset threshold refers to a pre-set proportional threshold used for the final judgment. When the proportion of the number of people not holding the handrail to the total number of status records exceeds this value, the system will make the final judgment that no handrail is being held.

[0067] The beneficial effects of the above technical solution are: by comprehensively analyzing the continuous behavioral records of personnel during their movement on the stairs, and making a final judgment based on statistical proportions rather than a single detection, it can effectively avoid misjudgments caused by instantaneous actions or detection errors, and significantly improve the accuracy and reliability of behavioral status determination.

[0068] Example 9: Based on Example 1, this example provides a deep learning-based system for detecting unattended handrail behavior. The result determination subunit includes: Information access subunit, used for: Access the information storage queue of the current person and extract the face detection results of the current person based on the access results; If the face detection results contain frontal facial information of the current person, it is determined that the current person is walking in the direction of the camera; otherwise, it is determined that the current person is walking with their back to the direction of the camera. The parameter adjustment subunit is used for: When it is determined that a person is walking with their back to the camera's shooting direction, the degree of interference between walking with their back to the camera and the determination of the behavior of not holding the handrail is determined based on historical observation data. The correction parameters for the preset threshold are determined based on the degree of interference, and the preset threshold is adjusted based on the correction parameters.

[0069] In this embodiment, when walking with one's back to the camera's shooting direction, the estimation results of the person's hands and posture may be inaccurate due to factors such as the camera angle and back-to-back obstruction. Therefore, it is necessary to adjust the handrail judgment criteria, for example, by adjusting the threshold for the proportion of the state without handrail from 0.5 to above 0.6 to reduce false alarms.

[0070] In this embodiment, the frontal facial information refers to the state information that, after analysis by the face detection model, the facial features (such as eyes, nose, and mouth) of the person are clearly visible and facing the camera.

[0071] In this embodiment, the degree of interference refers to the quantitative assessment of the extent to which the accuracy of the judgment of not holding the handrail is affected by the fact that the person's back is facing the camera, causing the hands or key parts of the body to be obscured.

[0072] In this embodiment, the correction parameter refers to a correction value calculated based on the degree of interference and used to dynamically adjust the preset threshold used in the final judgment.

[0073] The beneficial effects of the above technical solution are: by analyzing the direction of a person's face to identify their direction of movement, and by dynamically adjusting the judgment threshold based on the inherent uncertainty of behavior judgment when facing away from the camera, the risk of misjudgment caused by limited viewpoint is effectively reduced, and the accuracy and adaptability of the system's behavior recognition in different scenarios are improved.

[0074] Example 10: Based on Example 1, this example provides a deep learning-based system for detecting unattended handrail behavior, including an alarm reminder module, comprising: The status determination unit is used to obtain the judgment result of the person holding the handrail. When it is determined that the person is not holding the handrail, the personnel attribute information of the current person is retrieved based on the current person's information storage queue. The reminder scheme generation unit is used for: The personnel attribute information is parsed to extract the personalized attribute features of the current personnel, and the personalized attribute features of the current personnel are sorted by voice broadcast based on the preset feature sorting rules. Based on the ranking results of voice broadcasts, the voice conversion program is invoked to convert personalized attribute features into broadcast language, and the voice of the broadcast object is obtained based on the conversion results. At the same time, the preset broadcast voice library is accessed, the target reminder voice is retrieved, and the broadcast target voice is combined with the target reminder voice to obtain the target reminder scheme; Alarm notification unit, used for: The system tracks the location of the person in real time based on the image transmitted from the camera, and when the person enters the reminder area of ​​the voice broadcasting device, the system plays the target reminder plan to the person based on the voice broadcasting device. Based on the playback results, targeted voice reminders are given to people on the stairs who are not holding the handrail.

[0075] In this embodiment, personnel attribute information refers to objective characteristic data about personnel identified from images through a personnel attribute analysis model, such as estimated gender, clothing color, etc.

[0076] In this embodiment, personalized attribute features refer to specific feature items that can be used to form personalized titles, which are parsed from personnel attribute information, such as "little friend", "lady", "gentleman in red shirt", etc.

[0077] In this embodiment, the preset feature sorting rule refers to a set of pre-set rules for prioritizing multiple personalized attribute features, with the aim of selecting the most suitable one or several features to generate the most natural and appropriate title.

[0078] In this embodiment, the voice of the target person refers to a voice segment generated by text-to-speech technology based on the final selected personalized attribute features, used to address the target person in the reminder, such as "a man wearing a white shirt and glasses".

[0079] In this embodiment, the target reminder scheme refers to a complete and executable personalized voice reminder scheme formed by combining the voice of the target being broadcast with general reminder content retrieved from the voice library (such as "please hold the handrail").

[0080] In this embodiment, the reminder area refers to an optimal broadcast trigger area pre-defined for the voice broadcast device in the stairwell environment, ensuring that people can clearly hear the reminder when they arrive at this location and at the appropriate time. For example, it could be the location of the center line of the stairwell.

[0081] The beneficial effects of the above technical solution are: when it detects that a person is not holding the handrail, it can intelligently analyze their personal characteristics and generate a voice reminder with a personalized name, making the warning more targeted and easier to accept. By tracking the person's location in real time and accurately triggering the reminder when they enter the broadcast area, it achieves the timeliness and effectiveness of the reminder. This avoids the poor effect caused by reminders that are too early or too late, and also reduces interference to other people, significantly improving the accuracy and acceptance of safety intervention.

[0082] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A deep learning-based system for detecting unattended handrail behavior, characterized in that, include: The deep learning computing module is used to detect people in images input from the camera based on a pre-trained people detection model, and to perform multi-dimensional feature analysis of people based on the detection results. The handrail status analysis module is used to determine whether a person is holding a handrail based on the results of multi-dimensional feature analysis of people and the preset stair area and handrail area. The alarm and reminder module is used to provide targeted voice reminders to people on the stairs when it is determined from the handrail judgment results that a person is not holding the handrail.

2. The deep learning-based system for detecting unhandrail behavior according to claim 1, characterized in that, The deep learning computing module includes: The image acquisition unit is used to dock with the camera, receive the images transmitted from the camera in real time based on the docking result, and divide the images transmitted from the camera into frames based on the time series to obtain the image sequence; The personnel detection unit is used to sequentially input the images from the camera into a pre-trained personnel detection model based on the image sequence, and to extract features from the images based on the pre-trained personnel detection model to determine the target area where the personnel in the image are located. The personnel location determination unit is used to lock personnel in the target area based on the minimum bounding rectangle, and to determine the coordinate information [x1, y1, x2, y2] of the minimum bounding rectangle based on the locking result and the coordinate system to which the image belongs. Here, x1 is the x-coordinate of the upper left corner of the minimum bounding rectangle, y1 is the y-coordinate of the upper left corner of the minimum bounding rectangle, x2 is the x-coordinate of the lower right corner of the minimum bounding rectangle, and y2 is the y-coordinate of the lower right corner of the minimum bounding rectangle.

3. The deep learning-based system for detecting unhandrail behavior according to claim 1, characterized in that, The deep learning computing module includes: The result acquisition unit is used for: The system obtains the personnel detection results and, when personnel are present, tracks their positions based on a preset algorithm to obtain their position coordinates in different image frames. A unique index number is assigned to each person based on the location tracking results, and an information storage queue is assigned to each person based on the unique index number. Store the location coordinates of personnel in different image frames into the information storage queue under the corresponding unique index number; The multi-dimensional feature analysis unit is used for: Based on the personnel location tracking results, a local image of the personnel's location is captured from the current camera feed. Then, pre-trained personnel hand detection models, pre-trained pose estimation models, pre-trained face detection models, and pre-trained personnel attribute analysis models are retrieved to perform corresponding business feature analysis on the local image of the personnel's location, resulting in multi-dimensional features. These multi-dimensional features include the coordinates of the personnel's hand location, the position information of human skeleton points, the position coordinates of the face, and the personnel attribute information. The information storage unit is used to store the obtained multi-dimensional features into the information storage queue of the corresponding personnel.

4. The deep learning-based system for detecting unhandrail behavior according to claim 1, characterized in that, The handrail status analysis module includes: The object locking unit is used to acquire the image transmitted from the camera and perform object locking on the image transmitted from the camera based on the preset appearance characteristics of the handrail and stairs to obtain the handrail and stairs; Regional delineation units are used for: Based on the object locking result, the four corner points of the handrails on both sides in the image transmitted by the camera are firstly calibrated, and the four corner points of each side are sequentially connected based on the first calibration result to obtain the first quadrilateral, which is the handrail area on both sides. Meanwhile, based on the object locking result, the corner points on both sides of the top step and the corner points on both sides of the bottom step of the stairs are second-calibrated, and the four corner points are sequentially connected based on the second calibration result to obtain a second quadrilateral, which is the staircase area.

5. The deep learning-based system for detecting unhandrail behavior according to claim 4, characterized in that, The regional delineation unit includes: The result acquisition sub-unit is used to acquire the obtained handrail area, and select the center points A and B of the handrail areas on both sides respectively, and connect the center points A and B to obtain the line segment AB; Centerline determination unit, used for: Determine the center point of each side of the four sides of the staircase area, and connect the center points of opposite sides of the quadrilateral to obtain the center lines CD and EF of the staircase area; Calculate the angle values ​​of AB and CD using the following formula: ； Calculate the angle values ​​of AB and EF based on the following formula; ； If the calculation yields Less than If so, CD will be used as the center line of the staircase; otherwise, EF will be used as the center line of the staircase. The direction perpendicular to the center line of the staircase is determined as the direction of pedestrian movement.

6. The deep learning-based system for detecting unhandrail behavior according to claim 1, characterized in that, The handrail status analysis module includes: The hand detection and judgment unit is used for: Obtain the results of multi-dimensional feature analysis of people in the image input from the camera, and when a person's hand is detected in the multi-dimensional feature analysis results, determine the coordinates of the rectangular frame of the person's hand based on the multi-dimensional feature analysis results; The interaction ratio between the hand rectangle and the handrail areas on both sides is calculated based on the coordinates of the hand rectangle. The formula for calculating the interaction ratio between a single hand and a single handrail is as follows: ； When any IOU value is greater than the preset interaction threshold, it is determined that the person in the current frame image has held the handrail, and the determination result and the current frame image are stored in the person's information storage queue. The attitude estimation and judgment unit is used for: When no hand is detected in the multi-dimensional feature analysis of personnel or the interaction ratio is less than or equal to the preset interaction threshold, extract the position information of the personnel skeleton points in the multi-dimensional feature analysis of personnel. Based on the location information of the person's skeletal points, determine whether the representative points of the hand extending from the key points of the person's elbow and wrist are located within the handrail area to determine whether the person is holding the handrail.

7. The deep learning-based system for detecting unhandrail behavior according to claim 6, characterized in that, The attitude estimation and judgment unit includes: The hand coordinate determination sub-unit is used for: The obtained skeletal point position information of the person is used, and the elbow point on the same side as the handrail area is set as G, and the wrist point is set as H. The coordinates of G are determined based on the image coordinates. The coordinates of H and H ]; The coordinates of the extended hand representative point K are determined based on the preset extension coefficient obtained from the management terminal, combined with the elbow point G and the wrist point H. The specific calculation formula is as follows, where the preset extension coefficient is 0.4: ； ； based on and Obtain the coordinates of the extended hand representative point K, and determine the positional relationship between the coordinates of K and the handrail area on the same side; If the coordinates of K are within the same side of the handrail area, it is determined that the person in the current frame image has held the handrail, and the determination result is stored in the current person's information storage queue. Otherwise, it is determined that the current person is not holding the handrail with their current side hand. At the same time, if it is determined that the current person is not holding the handrail with either side hand, it is finally determined that the person in the current frame image is not holding the handrail, and the determination result is stored in the current person's information storage queue.

8. The deep learning-based system for detecting unhandrail behavior according to claim 5, characterized in that, The centerline determination unit includes: The result determines the sub-unit, used for: People are tracked based on images transmitted from cameras, and when a person crosses the center line of the stairwell area, the information storage queue of the current person is accessed. Based on the access results, the number of people who are judged not to be holding the handrail in different frame images is determined. At the same time, based on the access results, the total number of status records is determined. When the proportion of the number of people who are not holding the handrail in the total number of status records exceeds a preset threshold, it is finally determined that the current person is not holding the handrail; otherwise, it is finally determined that the current person is holding the handrail.

9. A deep learning-based system for detecting unhandrail behavior according to claim 8, characterized in that, The results determine the sub-units, including: Information access subunit, used for: Access the information storage queue of the current person and extract the face detection results of the current person based on the access results; If the face detection results contain frontal facial information of the current person, it is determined that the current person is walking in the direction of the camera; otherwise, it is determined that the current person is walking with their back to the direction of the camera. The parameter adjustment subunit is used for: When it is determined that a person is walking with their back to the camera's shooting direction, the degree of interference between walking with their back to the camera and the determination of the behavior of not holding the handrail is determined based on historical observation data. The correction parameters for the preset threshold are determined based on the degree of interference, and the preset threshold is adjusted based on the correction parameters.

10. A deep learning-based system for detecting unhandrail behavior according to claim 1, characterized in that, The alarm notification module includes: The status determination unit is used to obtain the judgment result of the person holding the handrail. When it is determined that the person is not holding the handrail, the personnel attribute information of the current person is retrieved based on the current person's information storage queue. The reminder scheme generation unit is used for: The personnel attribute information is parsed to extract the personalized attribute features of the current personnel, and the personalized attribute features of the current personnel are sorted by voice broadcast based on the preset feature sorting rules. Based on the ranking results of voice broadcasts, the voice conversion program is invoked to convert personalized attribute features into broadcast language, and the voice of the broadcast object is obtained based on the conversion results. At the same time, the preset broadcast voice library is accessed, the target reminder voice is retrieved, and the broadcast target voice is combined with the target reminder voice to obtain the target reminder scheme; Alarm notification unit, used for: The system tracks the location of the person in real time based on the image transmitted from the camera, and when the person enters the reminder area of ​​the voice broadcasting device, the system plays the target reminder plan to the person based on the voice broadcasting device. Based on the playback results, targeted voice reminders are given to people on the stairs who are not holding the handrail.

Images