A pedestrian crossing intention multi-modal prediction method and prediction device

Abstract

The application provides a kind of pedestrian traffic intention multimodal prediction method and prediction device, applied to access control monitoring technical field, based on the image information obtained, extract pedestrian multimodal feature, and whether the current pedestrian is allowed to pass through by decision logic judgment, including the following steps: S1, obtain the image sequence of inbound pedestrian, execute multi-target detection and tracking, identify each pedestrian ID, form boundary box sequence;S2, for each pedestrian, multimodal feature extraction is carried out, the multimodal feature includes card swiping action, gaze direction, running behavior and pedestrian category;S3, after processing multimodal feature, input into decision model to judge, to control gate state.The pedestrian traffic intention multimodal prediction method and prediction device, by fusing the gaze direction of pedestrian, action behavior, hand card swiping action and human body feature information, etc. Comprehensive judgment of the traffic intention of current traffic personnel, improve the accuracy and timeliness of abnormal behavior detection.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of access control monitoring, and particularly relates to a pedestrian passing intention multi-modal prediction method and a prediction device. BACKGROUND

[0002] In modern urban rail transit systems, the management of gate channels is crucial for preventing fare evasion and ensuring passenger safety. Currently, common gates are divided into two types: normally closed gates and normally open gates. Normally closed gates are usually closed when unauthorized, and only open after authorized by card swiping. Normally open gates, on the other hand, remain open under normal circumstances to improve traffic efficiency, and only close and block when unauthorized passage or abnormal situations are detected. Normally open gates improve traffic speed due to their default open state, but also pose higher requirements for abnormal passage detection.

[0003] Current abnormal passage detection mainly relies on infrared beam sensor array technology. By placing infrared beam sensors in the gate channel, it can detect the presence of people passing through and the number of people passing through. However, there are limitations in complex situations: when multiple people pass through closely together ("tailgating"), multiple infrared beams may be blocked at the same time, making it difficult to accurately distinguish the number of people passing through and thus failing to timely identify tailgating. In addition, the installation position of the current infrared sensor in the gate channel is limited, and misjudgment or missed detection often occurs in special situations such as carrying luggage or children. For example, a child with a small stature may pass through from below the sensor without triggering, resulting in fare evasion; conversely, a passenger passing through normally may be misidentified as violating the rules if their movements are slightly abnormal. Moreover, because the sensors can only be deployed on the gate body, their detection area is limited, and the gate reaction time is not sufficient in the event of an emergency, so the current infrared beam sensor method is difficult to fully meet the needs of normally open gates in terms of real-time and accuracy.

[0004] With the development of machine vision technology, some gates have begun to introduce cameras for monitoring. For example, using binocular cameras to obtain 3D depth information to determine the number of people in the channel, or combining face recognition to verify the identity of the person entering and leaving. These vision-based solutions have improved detection capabilities to some extent, such as recognizing faces or detecting multiple people passing through at the same time within a certain range. However, current vision solutions mostly focus on identity authentication (such as face comparison) or people counting, and lack in-depth analysis of the details of pedestrian passing intentions. In addition, analysis of a single visual feature is still not reliable in some scenarios (e.g., determining tailgating by detecting only two people), which may result in missed reports and false positives.

[0005] Overall, existing technologies still have obvious shortcomings in accurately predicting pedestrian passing intentions, and cannot fully meet the needs of normally open gates for early prediction and precise control. SUMMARY

[0006] In view of the above problems in the prior art, the purpose of the present application is to provide a pedestrian traffic intention multi-modal prediction method, which comprehensively judges the traffic intention of the current traffic personnel by fusing the information such as the gaze direction, action behavior, hand card swiping action and human body features of the pedestrians, and improves the accuracy and timeliness of abnormal behavior detection.

[0007] A pedestrian traffic intention multi-modal prediction method, based on image information obtained from a camera device, extracts multi-modal features of pedestrians, and judges whether to allow the current pedestrian to pass through through decision logic, the specific process comprising the following steps:

[0008] S1, obtain the image sequence of the inbound pedestrian, perform multi-target detection and tracking, identify each pedestrian ID, and form a boundary box sequence consistent with the pedestrian ID;

[0009] S2, for each pedestrian, multi-modal feature extraction is performed, the multi-modal features including card swiping action, gaze direction, running behavior and pedestrian category;

[0010] S3, the multi-modal features after processing are input into the decision model for judgment to control the state of the gate;

[0011] Wherein, the judgment process of the decision model is as follows:

[0012] S31, judge whether the pedestrian has swiped the card, if the pedestrian has swiped the card, the decision model outputs a passing probability one; if the pedestrian has not swiped the card, enter the pedestrian intention analysis stage;

[0013] S32, judge whether the pedestrian's gaze is fixed on the card swiping area, if the pedestrian's gaze is fixed on the card swiping area and the probability of running behavior is not greater than a running probability one, the decision model outputs a passing probability two; if the pedestrian's gaze is fixed on the card swiping area and the probability of running behavior is greater than a running probability one, the decision model outputs a passing probability three;

[0014] If the pedestrian's gaze is not fixed on the card swiping area and the probability of running behavior is greater than a running probability two, the decision model outputs a passing probability four; if the pedestrian's gaze is not fixed on the card swiping area and the probability of running behavior is not greater than a running probability two, enter the pedestrian category identification stage;

[0015] S33, based on S32, judge whether the pedestrian is a special passenger, if the pedestrian is a special passenger, the decision model outputs an overall passing probability as a probability five, if the pedestrian is not a special passenger, the decision model outputs a passing probability as a probability six;

[0016] S34, based on S31-S33, the passing probability output by the decision model drives the gate to keep the normally open state or immediately close and alarm; wherein, if the overall passing probability output by the decision model is less than a preset value, the gate is driven to close and alarm,

[0017] If not, the gate machine remains in the open state.

[0018] Preferably, S1 acquires the in-bound pedestrian image sequence, performs multi-target detection and tracking, identifies each pedestrian ID, and forms a pedestrian ID consistent bounding box sequence, the specific process being as follows:

[0019] Pedestrian target detection:

[0020] A lightweight target detection model is used to output pedestrian bounding boxes in each frame of image , wherein B represents the bounding box, t represents the frame number, and i represents the pedestrian ID.

[0021] Pedestrian identity association:

[0022] Pedestrian re-identification is performed through ReID technology to extract pedestrian features and form ReID feature vectors for distinguishing different pedestrians, the pedestrian features including clothing, accessories, and body posture.

[0023] The position / speed of the pedestrian is predicted through the Kalman filter algorithm for predicting the position of the pedestrian in the next frame.

[0024] Multi-target tracking is performed through the ByteTrack algorithm to output the bounding box sequence of pedestrian i in T consecutive frames of image, the expression of the bounding box sequence being: .

[0025] Preferably, the recognition of the card swiping action includes the following steps:

[0026] The corresponding ROI region is set based on the card swiping region of the gate device;

[0027] The two-dimensional coordinates and confidence of the pedestrian skeletal key points are extracted in each frame of image;

[0028] In the consecutive N frames, the two-dimensional coordinates of the wrist key points of the pedestrian continuously fall into the ROI region, and the confidence is not lower than the confidence threshold , and the elbow joint of the pedestrian has a motion trend towards the ROI region, it is determined that the pedestrian has a card swiping behavior.

[0029] Preferably, the two-dimensional coordinates and confidence of the 17 key points of the pedestrian are output in each frame of image through the HRNet-W32 network, the expression being , wherein is the two-dimensional coordinates of the Kth key point in the tth frame of image, is the corresponding confidence, wherein the key point K=9 of the left wrist, the key point k=10 of the right wrist, the key point K=7 of the left elbow, and the key point K=8 of the right elbow.

[0030] Preferably, the recognition of the gaze direction includes the following steps:

[0031] Based on the coordinates of the pedestrian's facial key points and the constructed 3D average face model, the rotation matrix of the pedestrian's head relative to the camera device is calculated using OpenCV's PnP or POSIT algorithm. and translation vector To obtain the pedestrian's head pose, where the rotation matrix... Including yaw angle Pitch angle Roll angle ;

[0032] The system spatially matches the pedestrian's head posture with the card-swiping area determined by the gate device to determine whether the pedestrian's gaze is pointing towards the card-swiping area of ​​the gate device. When the pedestrian's gaze is continuously directed towards the card-swiping area for a preset minimum time, it is determined that the pedestrian has the intention to swipe the card.

[0033] Preferably, spatial matching of pedestrian head posture and card-swiping area is achieved by comparing angle thresholds. The specific process is as follows:

[0034] Set the reference value for the head yaw angle required for pedestrians to normally look at the card swiping area. Head pitch angle reference value ;

[0035] Set the maximum allowable deviation threshold, including yaw angle deviation. Pitch angle deviation ;

[0036] When pedestrian i's head is facing the desired direction and At that time, it is determined that pedestrian i's line of sight falls into the direction of the card swiping area.

[0037] Preferably, the determination of running behavior is based on the coordinates and confidence level of the target pedestrian's skeletal key points, specifically including the following steps:

[0038] Skeletal keypoint extraction: The bounding box sequence is input into the HRNet-W32 keypoint network, which outputs 17 skeletal keypoints, expressed as follows:

[0039]

[0040] in, Let K be the two-dimensional coordinates of the Kth keypoint on the t-frame image. The corresponding confidence level;

[0041] Temporal feature construction: A sliding window of length L is set for each pedestrian i to capture dynamic behavioral features within the sliding window. The expression for the sliding window is: ;

[0042] Based on 17 key points of each frame of image, a dimensional matrix is constructed , forming a spatiotemporal trajectory atlas;

[0043] Setting a normalized timestamp , obtaining a time sequence feature matrix ;

[0044] Calculate the displacement difference of adjacent frame key points:

[0045] ,

[0046] Among them, is the coordinate of the kth key point in the tth frame.

[0047] Based on the above displacement difference features, an enhanced time sequence feature matrix is constructed:

[0048] ;

[0049] Among them: F aux = , indicating the introduced key point displacement difference feature;

[0050] , indicating a time sequence feature set with L frame length, each frame containing original key points and auxiliary features;

[0051] L is the length of the sliding window; i is the pedestrian number.

[0052] Behavior classification: based on the time sequence feature matrix , using a time sequence convolution network to output the running probability ;

[0053] Speed fusion determination: taking the center position difference of the bounding box of pedestrian i separated by 4 frames as the displacement, calculating the trajectory speed , the expression is:

[0054]

[0055] Among them, is the center point coordinate of the ith pedestrian bounding box in the tth frame;

[0056] When , it is determined that pedestrian i has a running risk, wherein, is the running probability threshold, is the speed threshold.

[0057] Preferably, the judgment of the walking person category is based on visual recognition, wherein the human body key points and the height of the pedestrian are compared with the threshold set by the system, when the judgment result is within the set threshold, if the key point distribution and the height are within the normal adult range, it is determined that it is a normal adult, if the human body key points and the height of the pedestrian are lower than the threshold set by the system, it is determined that it is a child, if the walking speed of the pedestrian is less than the walking threshold and the posture stability is less than the posture threshold, the walking person category is determined to be an old person;

[0058] In combination with the pedestrian target detection model, the head direction detection model and the object detection model, the disabled person and the commonly used auxiliary tool thereof, including a wheelchair, a walking stick or a walking aid, are recognized, if the auxiliary tool is recognized, it is determined that the walking person category is a disabled person;

[0059] In combination with the pedestrian target detection model, the head direction detection model and the object detection model, the features of the person pushing the stroller and the stroller, including the frame and the wheels of the stroller, are recognized, when the stroller is recognized, the pusher is associated with the stroller, and it is determined that the walking person category is the person pushing the stroller.

[0060] The beneficial effects of the present application are that the pedestrian passing intention multi-modal prediction method and prediction device,

[0061] Multi-feature fusion, improve accuracy: through the fusion of multiple visual features including card swiping action, gaze direction, running action and walking person category, the pedestrian passing intention is comprehensively judged. Compared with the single sensor or single algorithm scheme, the multi-modal information fusion can provide more abundant discrimination basis, greatly improve the accuracy of abnormal behavior detection, and reduce the false alarm and false alarm.

[0062] Early prediction, timely control: the deep learning model is used for predicting the pedestrian intention, the abnormal intention can be recognized in advance before the pedestrian makes a complete passing action. Once the model determines that there may be abnormal behaviors such as tailing and rushing through the gate, the system can issue instructions to make the normally open gate machine close quickly, and intercept the unauthorized person before entering the passage. Compared with the traditional method which needs to respond only when the gate passage is occupied, this method can respond to potential risks more timely, and ensure the safety of passing.

[0063] Adapt to complex scenes, strong robustness: by introducing human key point recognition and behavior analysis, the present application scheme can adapt to various crowd states. For example, whether the pedestrian is fast running through, walking slowly, carrying luggage or pushing a stroller, the system can recognize and judge through posture and action analysis. At the same time, the features such as gaze direction and hand action enable the system to perceive whether the pedestrian performs card swiping operation or not, so that the system still has good robustness in complex scenes such as multi-person dense passing, light change and partial occlusion, which is significantly better than the traditional scheme which simply depends on a certain signal.

[0064] Intelligent discrimination for special groups: Special consideration is given to the passage of the elderly, children and strollers. Through height determination or facial feature analysis, children or the elderly who move slowly can be identified among the pedestrians, and the gate logic is adjusted to accommodate these situations. For example: Elderly and children together: For the case where an adult is detected carrying a child together and meets the ride regulations, the system can determine normal passage and keep the gate open, avoiding the risk of false gate closure due to the child being mistaken for a follower. Pushing a stroller: When a parent is identified as pushing a stroller, the system automatically extends the open mode and does not immediately close the gate, allowing the parent to smoothly push the stroller into the passage before performing the card operation, improving passage efficiency and eliminating the risk of gate mis-closure due to the length of the stroller. This intelligent discrimination for a variety of special scenarios not only improves the friendliness of the system, but also expands its application range, ensuring safe and barrier-free passage in various complex pedestrian combinations.

[0065] Real-time performance and efficient passage: Each feature extraction and decision-making module is optimized to complete processing within milliseconds, meeting the real-time requirements of peak passenger flow. Since the normally open gate does not need to be physically opened and closed under normal circumstances, the invention ensures that the passage efficiency is not reduced while ensuring safety. Compared to the normally closed gate, which requires each person to stop and swipe the card to open the door, this solution does not require stopping in most normal passage situations, improving passenger experience and passage speed.

[0066] Easy to integrate and promote: The methods and devices of the invention mainly rely on software algorithms and general camera hardware, making them highly versatile. Existing normally open gate systems can be upgraded by adding cameras and computing units without major modifications to the original gate mechanism. At the same time, this technical solution is not only suitable for rail transit gates such as subways and train stations, but can also be promoted to other access control scenarios, such as office building access control, scenic spot ticketing channels, etc. It has a wide range of applications in systems that need to distinguish between normal passage and tailgating, such as office building access control, scenic spot ticketing channels, etc. BRIEF DESCRIPTION OF DRAWINGS

[0067] The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and are used to explain the invention, but do not constitute a limitation of the invention. In the drawings:

[0068] Figure 1 is a flowchart of the decision-making model judgment process of the invention;

[0069] Figure 2 is a principle block diagram of the prediction device of the invention;

[0070] Figure 3 is a connection diagram of the camera equipment and gate equipment of the invention;

[0071] Figure 4is a flowchart of the card swiping action recognition process of the present application;

[0072] Figure 5 is a flowchart of the eye recognition process of the present application;

[0073] Figure 6 is a flowchart of the running action recognition process of the present application;

[0074] Figure 7 is a flowchart of the pedestrian class recognition process of the present application. DETAILED DESCRIPTION

[0075] Embodiment One

[0076] As shown in Figure 1 , a pedestrian passing intention multi-modal prediction method extracts the multi-modal features of pedestrians based on image information obtained from a camera device, and determines whether to allow the current pedestrian to pass through by decision logic, and the specific process includes the following steps:

[0077] S1, obtain the image sequence of the incoming pedestrian, perform multi-target detection and tracking, identify each pedestrian ID, and form a boundary box sequence consistent with the pedestrian ID. The specific process is as follows:

[0078] Pedestrian target detection: a lightweight target detection model is used to output pedestrian bounding boxes in each frame of image obtained by the camera device , where B represents the bounding box, t represents the frame number, and i represents the pedestrian ID. Specifically, the lightweight target detection model can be YOLOv8-Nano, which is a real-time detection model that can obtain the positions of all pedestrians in each frame of image.

[0079] Pedestrian identity association: ReID technology is used for pedestrian re-identification to extract pedestrian features and form ReID feature vectors, which are used to distinguish different pedestrians. Pedestrian features include clothing, accessories, and body posture, etc.

[0080] The position / speed of the pedestrian is predicted by Kalman filtering algorithm, which is used to predict the position of the pedestrian in the next frame to realize the continuity of tracking.

[0081] Multi-target tracking is performed by ByteTrack algorithm, which outputs the bounding box sequence of pedestrian i in consecutive T frames of image, and the expression of the bounding box sequence is: ByteTrack algorithm, as a multi-target tracking algorithm, can maintain the ID consistency of the target pedestrian in complex occlusion scenarios, avoid confusion of different pedestrian features, and is conducive to improving the stability and accuracy of multi-modal feature recognition.

[0082] The cross-frame identity consistency of each pedestrian is ensured through mutual cooperation between the ReID feature vector, Kalman filtering algorithm and ByteTrack algorithm.

[0083] S2, for each pedestrian, multi-modal feature extraction is performed, wherein the multi-modal features include card swiping action, gaze direction, running behavior and pedestrian category.

[0084] The recognition of the card swiping action includes the following steps:

[0085] Based on the card swiping area, a corresponding ROI area is set: since the position of the card swiping device in the gate is relatively fixed, the position range of the card swiping device can be determined by manual measurement or using camera calibration parameters, so as to set the corresponding ROI area. The ROI area can be represented by a rectangular area R, .

[0086] Upper limb joint skeleton key point extraction: in each frame of image, a human pose estimation model is called to extract hand skeleton key points. Specifically, a HRNet-W32 network is used to output 17 key points two-dimensional coordinates and confidence, the expression is , wherein is the two-dimensional coordinate of the Kth key point in the t frame image, is the corresponding confidence. Specifically, the key point K=9 of the left wrist, the key point K=10 of the right wrist, the key point K=7 of the left elbow, and the key point K=8 of the right elbow.

[0087] Card swiping action recognition and determination: when the wrist key points of the pedestrian i fall into the ROI area for N consecutive frames and the confidence is not lower than the confidence threshold , and the corresponding elbow joint shows obvious movement trend towards the ROI area, it is determined that the card swiping action is completed, otherwise it is considered that the card swiping action is not completed.

[0088] The specific implementation process is as follows:

[0089] Firstly, based on the extracted hand key points, it is judged whether the hands of the pedestrian fall into the set ROI area. Specifically, for each pedestrian i in the t frame image, it is checked whether the left and right wrist key point coordinates fall into the rectangular area R, and whether the corresponding confidence value is not lower than the confidence threshold .

[0090] For example, the confidence threshold is set to =0.5, and the indicator function is used to represent the judgment of a single frame:

[0091]

[0092] When , it indicates that the wrist of the pedestrian i is detected to enter the card swiping area; otherwise, no card swiping action is detected.

[0093] Secondly, a time condition of continuous frame judgment verification is set. When the wrist of the pedestrian i continuously locates in the ROI area for N continuous frames, the actual completion of the card swiping action is determined, wherein the frame threshold is set as N = 3.

[0094] Since the entering detection of a single frame can be affected by jitter or instantaneous false detection, a continuous frame determination strategy is adopted to improve robustness, that is, when the wrist of the pedestrian i is detected to enter the card swiping area for N = 3 continuous frames, the card swiping action of the pedestrian i is determined to be completed.

[0095] In addition, the motion direction of the elbow joint is introduced as a redundant check to prevent false judgment caused by accidental overlap of the wrist key point. By observing the position change of the elbow key point corresponding to the wrist, it is ensured that the arm movement trend is consistent with the card swiping action.

[0096] Specifically, the displacement vector of the elbow before and after entering the card swiping area is calculated. If the elbow position moves obviously towards the ROI area direction relative to several frames before, that is, whether the horizontal distance between the elbow and the ROI area is reduced, if reduced, it indicates that the arm indeed has the action of stretching towards the card swiping area; otherwise, if the wrist temporarily enters the ROI area but the elbow does not move, it can be a non-card swiping accidental action, which can be regarded as false triggering and not counted as effective card swiping.

[0097] The card swiping action is determined by comprehensively considering the ROI overlap state, duration and elbow cooperative motion of the wrist. Without introducing a specially trained gesture classification model, the recognition of the card swiping action can be completed.

[0098] As can be seen from the above, the recognition process of the card swiping action can identify whether the card swiping action occurs in real time at the moment when the pedestrian approaches the card swiping area, which provides a key basis for distinguishing normal passing and tailing intrusion. The discrimination method based on skeleton key points has extremely low calculation overhead (only a few key point coordinates are determined, and the time consumption can be ignored), and does not depend on additional training of complex models, and has good interpretability and engineering feasibility. On the actual gate embedded platform, the method can run synchronously with the video frame rate to realize high-speed and reliable detection of the card swiping action, and provide important visual feature support for gate passing abnormal behavior prediction.

[0099] Since the passengers who normally pass the card swiping machine usually direct their gaze to the card swiping device on the gate to align and complete the ticket checking operation; on the contrary, the people who attempt to tailgate often look ahead or around and do not pay attention to the position of the card swiping machine. Based on this behavior feature, by detecting the head direction and line of sight of the passenger, whether the passenger focuses attention on the card swiping area is identified, so as to infer the card swiping intention of the passenger.

[0100] ​Wherein, the recognition process of the gaze direction is as follows:

[0101] 1) Head orientation detection: based on the extracted face key point distribution, combined with the pre-defined three-dimensional face average model coordinates, the rotation matrix of the head relative to the camera device is calculated through the PnP algorithm or the POSIT algorithm of OpenCV and the translation vector , which is used to obtain relevant information data for subsequent steps to determine the walking speed of the pedestrian, the rotation matrix can be converted into head orientation parameters represented by Euler angles, including yaw angle , pitch angle and roll angle , which are used for head tilt correction. Among them, the yaw angle and the pitch angle are the key parameters for judging the gaze direction, which respectively reflect the angle of looking left / right and up / down of the passenger.

[0102] 2) Gaze area matching: the head orientation of the pedestrian is spatially matched with the set ROI area to determine whether the gaze of the pedestrian is directed at the ROI area of the gate, which is used to assist in determining the card swiping intention.

[0103] The gaze area matching method includes the following two kinds, one of which can be selected to realize the gaze recognition and judgment.

[0104] The first kind is to compare and judge by angle threshold

[0105] According to the relative direction of the current position of the pedestrian and the position of the card swiping device, the reference values of the head yaw angle and the pitch angle required for normal gaze ROI area are set .

[0106] For example, if the card swiping device is located in the right front low position of the gate passage, the passenger needs to slightly look right front and lower when swiping the card normally. At this time, the maximum deviation threshold allowed is defined, such as yaw angle deviation , pitch angle deviation . When the head orientation of the pedestrian i satisfies and , it can be determined that the line of sight is roughly in the direction of the ROI area. In addition, in order to improve the reliability, the face detection confidence is required to be not less than the confidence threshold, that is, the confidence , so as to avoid misjudgment caused by inaccurate angle estimation.

[0107] The second kind is to detect and judge by line of sight projection

[0108] 3) Combined with the camera calibration parameters, the spatial relationship between the line of sight and the ROI region is directly determined by geometric method. Specifically, the head pose estimation result is used to extend the pedestrian line of sight direction in three-dimensional space into a ray, and project it to the world coordinate system or the image plane. The pedestrian line of sight direction is determined by the head pose rotation matrix and the relative position of the head in the above steps;

[0109] If the line of sight ray intersects with the predefined ROI region (for example, the ray falls within the ROI rectangular range on the gate machine plane), it means that the passenger's line of sight is indeed directed to the card reader position, and the gaze can be determined to be established.

[0110] Compared with the simple angle threshold, this method uses the absolute spatial position relationship obtained by camera calibration to determine more accurately, but the calculation is relatively complex. In consideration of real-time performance on embedded platform, the angle threshold method is usually preferred for fast screening.

[0111] 4) Card swiping intention determination:

[0112] The gaze region matching is performed frame by frame. When the pedestrian's gaze is continuously directed to the ROI region for a preset minimum time, it is determined that the pedestrian has card swiping intention.

[0113] For example, in this embodiment, the continuous frame number threshold N=3 (about 0.1 second) is defined. When the head of pedestrian i is directed to the ROI region in the continuous N frames and the detection confidence meets the requirement, it is determined that the pedestrian has focused attention on the card swiping device, and the card swiping intention determination condition is met; otherwise, if the head is only briefly directed to the card reader in individual frames (for example, a brief glance) without meeting the continuous time requirement, it is considered that there is no clear card swiping intention.

[0114] By setting the timing constraint, the jitter caused by single frame misjudgment can be greatly reduced, and the accuracy of card swiping intention determination can be improved.

[0115] The running behavior is determined based on the coordinates and confidence of the key points of the target pedestrian, including the following steps:

[0116] Skeletal key point extraction: the sequence of bounding boxes with the same pedestrian ID is input into the HRNet-W32 key point network, and 17 skeletal key point information is output, expressed as:

[0117]

[0118] wherein, is the two-dimensional coordinate of the Kth key point on the tth frame image, is the corresponding confidence;

[0119] Time sequence feature construction:

[0120] A sliding window with length L is set for each pedestrian i, and L is typically 8-12 frames, which is used to capture the dynamic behavior characteristics within 0.25 seconds. The expression of the sliding window is: .

[0121] Based on the 17 key points of each frame of image, a dimensional matrix is constructed , forming a spatiotemporal trajectory atlas.

[0122] Set the normalized timestamp , obtain the time sequence feature matrix , and improve the sensitivity of the behavior speed change rhythm.

[0123] Calculate the displacement difference of the key points of adjacent frames:

[0124] ,

[0125] wherein, is the coordinate of the kth key point in the tth frame.

[0126] Based on the above displacement difference feature, an enhanced time sequence feature matrix is constructed:

[0127] ;

[0128] wherein: Faux= , indicating the introduced key point displacement difference feature;

[0129] , indicating a time sequence feature set with L frame length, each frame containing original key points and auxiliary features;

[0130] L is the length of the sliding window;

[0131] i is the pedestrian number.

[0132] Behavior classification: based on the time sequence feature matrix obtained by the above steps , a time sequence convolution network is used to output the running probability .

[0133] wherein, the time sequence convolution network is trained by combining public skeleton data set with field sampling data, and a weighted cross-entropy loss function is used to improve the recall ability of the time sequence convolution network to the "running" category. In addition, each update of the sliding window, that is, one inference is completed, and the current running confidence of the pedestrian is updated in real time.

[0134] Speed fusion determination: taking the center position difference of the bounding box of pedestrian i separated by 4 frames as the displacement, calculating the trajectory speed , the expression is:

[0135]

[0136] wherein, is the center point coordinate of the i-th pedestrian bounding box in the t-th frame.

[0137] Specifically, a running probability threshold value , a trajectory speed threshold value In the embodiment, the running probability and speed output by the time sequence convolution network are fused and determined in combination to determine that the pedestrian i has a running risk when , wherein, is a running probability threshold value set in advance, is a speed threshold value set in advance. The two threshold values set above can be updated adaptively according to the passenger flow environment of the station.

[0138] The determination of the pedestrian category is based on a multi-modal visual recognition method, which combines human attribute recognition, estimation and auxiliary tool detection, and realizes high-precision determination of the pedestrian category. The pedestrian category includes children, the elderly, the disabled, and people pushing baby strollers.

[0139] wherein, the human skeleton key points, the height of the pedestrian and the threshold value set by the system are compared, when the result of the judgment is within the set threshold value, if the key point distribution and the height are within the normal adult range, it is determined that it is a normal adult; if the human skeleton key points and the height of the pedestrian are lower than the threshold value set by the system (such as about 1.2-1.3 meters), it is determined that it is a child; the method of fusing age and height characteristics can effectively distinguish preschool children from short adult passengers, and improve the recognition accuracy.

[0140] The elderly recognition process includes: if the walking speed of the pedestrian is less than the walking threshold value and the posture stability is less than the posture threshold value, the pedestrian category is determined to be the elderly.

[0141] The disabled person recognition process includes: combining the pedestrian target detection model, the head direction detection model and the object detection model to recognize the disabled person and the auxiliary tools commonly used by the disabled person, such as wheelchairs, crutches or walking aids, etc. If the typical structural features (such as seat frames and wheel outlines) of the wheelchair are detected near or below the passenger, it is determined that the passenger is a disabled person who sits in a wheelchair; similarly, when the passenger is detected using a crutch, a walking frame or other devices, it can also assist in determining that it is a disabled person who cannot move freely. By recognizing these auxiliary devices, the system can timely classify the corresponding passenger as a disabled person.

[0142] The process of identifying the person pushing the stroller includes: combining the pedestrian target detection model, the head direction detection model and the object detection model to push the stroller and the characteristic shape of the stroller, including the frame and wheels of the stroller. Specifically, when the characteristic shape of the stroller is identified in the image, the stroller is associated with the person pushing it immediately behind through the spatial position relationship, and it is determined that the passenger belongs to the parent pushing the stroller. Among them, the association identification of the stroller and the adult needs to consider the distance, relative position and motion direction consistency between the two, so as to avoid misassigning the stroller to the wrong pedestrian.

[0143] S3, inputting the processed multi-modal features into a decision model to determine the state of the gate.

[0144] The determination process of the decision model is as follows:

[0145] S31, determining whether the pedestrian has swiped the card. If the pedestrian has swiped the card, the decision model outputs a passing probability one; if the pedestrian has not swiped the card, the pedestrian intent analysis stage is entered.

[0146] S32, determining whether the pedestrian's gaze is fixed on the card swiping area. If the pedestrian's gaze is fixed on the card swiping area and the probability of running behavior is not greater than a running probability one, the decision model outputs a passing probability two; if the pedestrian's gaze is fixed on the card swiping area and the probability of running behavior is greater than the running probability one, the decision model outputs a passing probability three.

[0147] If the pedestrian's gaze is not fixed on the card swiping area and the probability of running behavior is greater than a running probability two, the decision model outputs a passing probability four; if the pedestrian's gaze is not fixed on the card swiping area and the probability of running behavior is not greater than the running probability two, the pedestrian category identification stage is entered.

[0148] S33, based on S32, determining whether the pedestrian is a special passenger. If the pedestrian is a special passenger, the decision model outputs an overall passing probability as a probability five; if the pedestrian is not a special passenger, the decision model outputs a passing probability as a probability six.

[0149] S34, based on S31-S33, driving the gate to remain in an open state or to be immediately closed and alarmed according to the passing probability output by the decision model; wherein, if the overall passing probability output by the decision model is less than a preset value, the gate is driven to be closed and alarmed,

[0150] In addition, as Figure 2As shown, the camera device is arranged above the existing gate passage to monitor the entrance direction of the gate passage at a top-down angle, and the camera range covers the pedestrian passage area in the entrance direction to collect real-time passage image data. Specifically, in this embodiment, the camera device is a binocular RGB camera, which is fixed to the crossbeam or bracket above the gate passage in a plug-in installation mode, so that the binocular RGB camera can overlook the entire passage area in the entrance direction, ensuring comprehensive monitoring of the behavior of pedestrians entering the gate.

[0151] Embodiment Two

[0152] The values of the passage probability and the running probability in the pedestrian passage intention multi-modal prediction method are user-defined, and the specific parameters can be modified. In this embodiment, a set of values of the passage probability is provided as a reference, as follows:

[0153] S31, it is judged whether the pedestrian has swiped the card. If the pedestrian has swiped the card, the decision model outputs the overall passage probability as 0.98; if the pedestrian has not swiped the card, the pedestrian intention analysis stage is entered.

[0154] S32, if the pedestrian has not swiped the card, it is judged whether the pedestrian's gaze is fixed on the card swiping area. If the pedestrian's gaze is fixed on the card swiping area, and the probability of running behavior is less than a certain pre-set running threshold value 0.5, the decision model outputs the overall passage probability as 0.9; if the pedestrian's gaze is fixed on the card swiping area, and the probability of running behavior is greater than the pre-set running threshold value 0.5, the decision model outputs the overall passage probability as 0.2.

[0155] If the pedestrian's gaze is not fixed on the card swiping area, and the probability of running behavior is greater than the pre-set running threshold value, the decision model outputs the overall passage probability as 0.02; if the pedestrian's gaze is not fixed on the card swiping area, and the probability of running behavior is less than the pre-set running threshold value 0.5, the pedestrian category recognition stage is entered.

[0156] S33, based on S32, it is judged whether the pedestrian is a special passenger. If the pedestrian is a special passenger, the decision model outputs the overall passage probability as 0.9; if the pedestrian is not a special passenger, the decision model outputs the passage probability as 0.2.

[0157] S34, based on S31-S33, the passage probability output by the decision model is used to drive the gate to remain in a normally open state or to be immediately closed and alarmed; if the overall passage probability output by the decision model is less than 0.8, the gate is closed and alarmed.

[0158] If not, the gate remains in a normally open state.

[0159] Embodiment Three

[0160] Since the prediction device is an independent module and is connected with the gate machine through a standardized interface, installation and deployment can be completed without modification of the main structure of the gate machine, and can be adapted to gate machines of different manufacturers or models. Further, the device has good universality and expandability, and is applicable not only to rail transit gate machines but also to other access management scenarios, such as office building access control, scenic spot ticket checking channels, etc., as an independent additional intelligent monitoring unit to improve the pedestrian passage supervision capability of the existing system.

[0161] In addition, as shown in Figure 2 , Figure 3 The prediction device provides three standard connection modes of power supply interface, network interface and serial communication interface, and is connected to the power supply, data communication and control signal interfaces reserved inside the gate machine through the corresponding power line, network cable and serial port line, so as to realize the electrical power supply, data transmission and control signal interaction between the prediction device and the gate machine.

[0162] Through these connections, the prediction device can obtain stable power from the gate machine, and perform bidirectional data communication with the main control system of the gate machine (such as sending pedestrian detection results, receiving gate state information), and can also output control instructions to cooperate with the gate machine to perform opening and closing actions.

[0163] The above only describes the preferred embodiments of the present application and is not used to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments or replace some technical features with equivalent ones. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for multi-modal prediction of pedestrian crossing intention, characterized in that, Based on the image information obtained from the camera device, the multi-modal features of the pedestrian are extracted, and it is judged whether the current pedestrian is allowed to pass through by decision logic, and the specific process includes the following steps: S1, obtain the image sequence of the entering pedestrian, perform multi-target detection and tracking, identify the ID of each pedestrian, and form a boundary box sequence consistent with the pedestrian ID; S2, for each pedestrian, multi-modal feature extraction is performed, and the multi-modal features include card swiping action, gaze direction, running behavior and pedestrian category; S3, the processed multi-modal features are input into the decision model for judgment to control the state of the gate; Wherein, the judgment process of the decision model is as follows: S31, judge whether the pedestrian has swiped the card, if the pedestrian has swiped the card, the decision model outputs the passing probability one; if the pedestrian has not swiped the card, enter the pedestrian intention analysis stage; S32, judge whether the pedestrian's gaze is fixed on the card swiping area, if the pedestrian's gaze is fixed on the card swiping area, and the probability of running behavior is not greater than the running probability one, the decision model outputs the passing probability two; if the pedestrian's gaze is fixed on the card swiping area, and the probability of running behavior is greater than the running probability one, the decision model outputs the passing probability three; If the pedestrian's gaze is not fixed on the card swiping area, and the probability of running behavior is greater than the running probability two, the decision model outputs the passing probability four; if the pedestrian's gaze is not fixed on the card swiping area, and the probability of running behavior is not greater than the running probability two, enter the pedestrian category identification stage; S33, based on S32, judge whether the pedestrian is a special passenger, if the pedestrian is a special passenger, the decision model outputs the overall passing probability as the probability five, if the pedestrian is not a special passenger, the decision model outputs the passing probability as the probability six; S34, based on S31-S33, the passing probability output by the decision model is used to drive the gate to keep the normally open state or close immediately and alarm; wherein, if the overall passing probability output by the decision model is less than a preset value, the gate is driven to close and alarm, If not, keep the gate in the normally open state.

2. The method of claim 1, wherein, S1, obtain the image sequence of the entering pedestrian, perform multi-target detection and tracking, identify the ID of each pedestrian, and form a boundary box sequence consistent with the pedestrian ID, the specific process is as follows: Pedestrian target detection: A lightweight object detection model is used to output a pedestrian bounding box in each frame of image where B denotes a bounding box, t denotes a frame label, and i denotes a pedestrian ID Pedestrian identity association: Pedestrian features are extracted through ReID technology to form ReID feature vectors, which are used to distinguish different pedestrians, and the pedestrian features include clothing, accessories and body posture; The position / speed of the pedestrian is predicted by Kalman filtering algorithm, which is used to predict the position of the pedestrian in the next frame; Through the ByteTrack algorithm, multi-target tracking is performed, and a bounding box sequence of the pedestrian i in the continuous T frame images is output, and the expression of the bounding box sequence is: .

3. The method of claim 1, wherein, The identification of card swiping action includes the following steps: Set the corresponding ROI area based on the card swiping area of the gate device; Extract the two-dimensional coordinates and confidence of the pedestrian skeleton key points in each frame of image; In the continuous N frames, the two-dimensional coordinates of the wrist key points of the pedestrian continuously fall into the ROI region, and the confidence is not lower than the confidence threshold At the same time, the elbow joint of the pedestrian has a motion trend towards the ROI region, and it is determined that the pedestrian has a card swiping behavior.

4. The method of claim 3, wherein, The 17 key points of the pedestrian in each frame of image are output by the HRNet-W32 network, and the two-dimensional coordinates and confidence of the pedestrian are expressed as wherein is the two-dimensional coordinate of the Kth key point on the tth frame of image, is the corresponding confidence, wherein the key point K=9 is the left wrist, the key point K=10 is the right wrist, the key point K=7 is the left elbow, and the key point K=8 is the right elbow.

5. The method of claim 1, wherein, The identification of gaze direction includes the following steps: Based on the extracted face key point distribution, combining a predefined three-dimensional face average model coordinate, a rotation matrix of the head of the pedestrian relative to the camera device is calculated by a PnP algorithm or a POSIT algorithm of OpenCV and a translation vector to obtain the head posture of the pedestrian, wherein the rotation matrix includes a yaw angle , a pitch angle , and a roll angle ; Match the pedestrian head posture with the card swiping area determined by the gate device in space to judge whether the pedestrian's gaze is directed at the card swiping area of the gate device, when the pedestrian's gaze continuously points to the card swiping area for a preset minimum time, it is determined that the pedestrian has the intention to swipe the card.

6. The method of claim 5, wherein, The spatial matching of the pedestrian head posture and the card swiping area is realized by angle threshold comparison judgment, and the specific process is as follows: Reference value of head yaw angle required for setting the card swiping area in normal gaze of the pedestrian Reference value of head pitch angle ; setting a maximum allowed deviation threshold, including yaw angle deviation , pitch angle deviation ; When the head of the pedestrian i is oriented in a direction that satisfies and the line of sight of the pedestrian i is determined to fall in the direction of the card swiping region.

7. The method of claim 1, wherein, The judgment of the running behavior is based on the coordinates and confidence of the target pedestrian's skeleton key points, and specifically includes the following steps: Skeleton key point extraction: input the boundary box sequence into the HRNet-W32 key point network, and output 17 skeleton key point information, expressed as: wherein, is the two-dimensional coordinate of the K key point on the t-th frame image, is the corresponding confidence. Temporal feature construction: Set a sliding window with length L for each pedestrian i to capture the dynamic behavior features within the sliding window, and the expression of the sliding window is: ; Based on 17 key points of each frame of image, a dimensional matrix is constructed , forming a space-time trajectory atlas; Setting a normalized timestamp , resulting in a timing feature matrix ; Calculate the displacement difference of adjacent frame key points: ， wherein, is the coordinate of the Kth key point in the tth frame; Based on the above displacement difference features, an enhanced time sequence feature matrix is constructed: ； wherein: represents the introduced key point displacement difference feature; represents a set of temporal features with L frame length, each frame containing original key points and auxiliary features; L is the length of the sliding window; i is the pedestrian number; Behavior classification: based on enhanced time-series feature matrix , using time-series convolution network to output running probability ; Speed fusion determination: take the center position difference of the pedestrian i bounding box separated by 4 frames as displacement, calculate the trajectory speed , the expression is: wherein, is the coordinate of the center point of the i-th pedestrian bounding box in the t-th frame; When a risk of running for the pedestrian i is determined, wherein is a running probability threshold value, is a speed threshold value.

8. The method of claim 1, wherein, The judgment of the pedestrian category is based on visual recognition, wherein the human skeleton key points and the height of the pedestrian are compared with the threshold set by the system. When the judgment result is within the set threshold, if the key point distribution and the height are within the normal adult range, it is determined that it is a normal adult; If the human skeleton key points and the height of the pedestrian are lower than the threshold set by the system, it is determined that it is a child; If the walking speed of the pedestrian is less than the walking threshold and the posture stability is less than the posture threshold, it is determined that the pedestrian category is an old person; In combination with the pedestrian target detection model, the head direction detection model and the object detection model, the disabled person and the commonly used assistive tools, including wheelchair, crutch or walking aid, are identified. If the assistive tool is identified, it is determined that the pedestrian category is a disabled person; In combination with the pedestrian target detection model, the head direction detection model and the object detection model, the person pushing the stroller and the characteristic shape of the stroller, including the frame and the wheels of the stroller, are identified. When the stroller is identified, the pusher is associated with the stroller, and it is determined that the pedestrian category is a person pushing the stroller.

9. A device for multi-modal prediction of pedestrian crossing intention, characterized in that, The prediction device comprises: A camera device is installed above the gate passage, and the camera range covers the pedestrian passage area in the inbound direction, which is used to collect real-time passage image data; A multi-modal feature extraction module is used to extract the multi-modal features of pedestrians according to the image information obtained by the camera device; the multi-modal feature extraction module includes a card swiping action recognition unit, a gaze direction recognition unit, a running behavior recognition unit, and a pedestrian category recognition unit; A decision model judges the feature information extracted by the multi-modal feature extraction module based on decision logic to determine whether the current pedestrian is allowed to pass; A main control module is used to control the state of the gate device according to the output result of the decision model. The gate is in a normal open state. If the passing probability output by the decision model is less than a preset value, the gate is driven to close and an alarm is sounded. If the passing probability output by the decision model is not less than the preset value, the gate remains in the normal open state.

Images