Pedestrian behavior recognition system in vehicle driving scene

Abstract

The invention provides a pedestrian behavior recognition system in a vehicle driving scene, and relates to the technical field of artificial intelligence and computer vision, and the system firstly fuses multi-modal sensor data to lock a high-risk pedestrian target, and then generates a probabilistic sight line cone with uncertainty measurement by using a deep network. On the basis, a space alignment Gaussian kernel with an environmental risk peak value as the center is introduced, an attention focus enhancement algorithm is constructed, and deep coupling calculation is carried out on pedestrian sight probability distribution and a dynamic environmental risk field. The mechanism can accurately quantify the spatial proximity and direction consistency of the sight focus and the key risk source, effectively suppress divergent sight and background noise, amplify an active gaze signal, and finally realize accurate pre-judgment of the pedestrian crossing intention in combination with whole body attitude features, thereby providing a reliable decision basis for an autonomous vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and computer vision, and more particularly, to a pedestrian behavior recognition system in a vehicle driving scene. BACKGROUND

[0002] With the rapid development of automatic driving technology, the driving safety of vehicles in complex urban road environments has become the focus of research. In the mixed traffic scene of pedestrians and vehicles, pedestrians are the most vulnerable and uncertain traffic participants. Accurate prediction of their movement intentions is crucial for the decision-making and planning of autonomous vehicles. If the system cannot accurately identify the crossing intention of pedestrians in time, it is easy to cause a collision accident due to the lagging response of the vehicle, or frequent false braking due to excessive caution, which seriously affects the comfort and efficiency of travel.

[0003] Existing pedestrian behavior recognition technologies mainly rely on trajectory-based kinematic prediction or action classification based on key points of the skeleton. However, the trajectory-based method assumes that the future movement of pedestrians is an inertial extension of the historical state, making it difficult to capture sudden behaviors such as sudden stops or sudden turns. While simple action classification can identify walking or standing states, it often falls short in complex interactive scenarios. For example, it is often difficult to distinguish between a distracted pedestrian looking down at a mobile phone and an interactive pedestrian observing the road conditions and preparing to cross, based solely on body movements. This false interaction phenomenon often leads to misjudgment by the system. The root cause of this problem lies in the fact that existing technologies often separate pedestrian features and environmental elements, lacking a fine spatial relationship modeling between the gaze focus of pedestrians and environmental risk sources. In particular, when dealing with attention mechanisms, traditional methods often use a global weighted integral approach, which simply multiplies and adds the gaze probability map and the environmental risk map at the pixel level. This rough calculation method has significant technical defects: it ignores the spatial proximity and directional consistency between the gaze focus and the risk source, making it difficult for the system to effectively distinguish between active interaction that accurately focuses on the risk source and passive behavior that inadvertently scans in the direction of the risk source. In addition, this type of method lacks a dynamic suppression mechanism for gaze prediction uncertainty, resulting in similar weight contributions from ambiguous divergent gaze and precise focused gaze. This limits the accuracy and robustness of intention recognition.

[0004] Therefore, an optimized pedestrian behavior recognition system in a vehicle driving scene is desired. SUMMARY

[0005] To solve the above technical problems, the present application is proposed. Embodiments of the present application provide a pedestrian behavior recognition system in a vehicle driving scene, which includes:

[0006] a data acquisition module configured to acquire camera data and lidar data;

[0007] a pedestrian global feature extraction module configured to perform pedestrian global feature extraction on the camera data and the lidar data to obtain a pedestrian target list;

[0008] an intention preliminary screening and high-risk target locking module configured to perform intention preliminary screening and high-risk target locking on the pedestrian target list based on a self-vehicle state to obtain a high-risk pedestrian list;

[0009] a head region refinement and probabilistic line-of-sight cone estimation module configured to perform head region refinement and probabilistic line-of-sight cone estimation on the high-risk pedestrian list based on the camera data to obtain a pedestrian list with line-of-sight information;

[0010] an attention focus analysis module configured to perform attention focus analysis on the pedestrian list with line-of-sight information, the camera data and the self-vehicle state to obtain a pedestrian attention score list;

[0011] an intention fusion decoding and behavior prediction output module configured to perform intention fusion decoding and behavior prediction output on the pedestrian attention score list, whole-body posture features in the pedestrian target list and the self-vehicle state to obtain a behavior category and an intention probability.

[0012] Compared with the prior art, the pedestrian behavior recognition system in a vehicle driving scene provided by the application firstly realizes accurate detection and three-dimensional positioning of pedestrians in the scene through multi-modal data registration of a camera and a lidar, and extracts whole-body posture features of the pedestrians. Subsequently, based on the relative position and orientation information of the self-vehicle motion state and the pedestrians, rapid quantitative evaluation of collision risk is performed, thereby screening out high-risk targets with potential crossing intention and realizing efficient allocation of computing resources. For the high-risk pedestrians, further refined line-of-sight estimation of the head region is performed, and a probabilistic line-of-sight cone model is creatively introduced to represent the line-of-sight direction and its predicted uncertainty. On this basis, an environmental risk saliency map is generated to quantify the threat level of each element in the scene, and the degree of overlap between the line-of-sight focus and the risk source is calculated through a spatially aligned Gaussian kernel mechanism, thereby effectively solving the problem that traditional methods cannot distinguish between active interaction and passive scanning. Finally, the intention decoding is performed by fusing the attention score, the posture feature and the self-vehicle state, and the behavior category and the crossing probability of the pedestrian are output, thereby providing a reliable basis for automatic driving decision. BRIEF DESCRIPTION OF DRAWINGS

[0013] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which:

[0014] Figure 1 A system block diagram of a pedestrian behavior recognition system in a vehicle driving scene according to an embodiment of the present application.

[0015] Figure 2 A data flow diagram of a pedestrian behavior recognition system in a vehicle driving scene according to an embodiment of the present application.

[0016] Figure 3 A block diagram of a pedestrian global feature extraction module in a pedestrian behavior recognition system in a vehicle driving scene according to an embodiment of the present application.

[0017] Figure 4 A block diagram of an intent preliminary screening and high-risk target locking module in a pedestrian behavior recognition system in a vehicle driving scene according to an embodiment of the present application.

[0018] Figure 5 A block diagram of a head region refinement and probabilistic gaze cone estimation module in a pedestrian behavior recognition system in a vehicle driving scene according to an embodiment of the present application.

[0019] Figure 6 A block diagram of an attention focus analysis module in a pedestrian behavior recognition system in a vehicle driving scene according to an embodiment of the present application.

[0020] Figure 7 A block diagram of a pedestrian attention score list generation unit in a pedestrian behavior recognition system in a vehicle driving scene according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] Hereinafter, example embodiments according to this application will be described in detail with reference to accompanying drawings. As apparent from the following description of the embodiments of this application, the embodiments described are only a part of embodiments of this application, and do not limit the present application unless otherwise explicitly provided in the following description of embodiments of this application.

[0022] As used in the specification and claims, the words "a," "an" and / or "the" do not exclude the possibility that more than one of the described components or steps are present. The words "include," "comprise," "comprising" and / or "comprises" do not exclude the presence of other elements or steps than those listed in a claim.

[0023] While this application makes various references to certain modules of the systems according to embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The modules described are merely illustrative, and different aspects of the systems and methods may use different modules.

[0024] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0025] In the fields of autonomous driving and assisted driving, accurately predicting pedestrians' intention to cross the road is crucial. Figure 1 Pedestrian behavior recognition remains a highly challenging problem. Existing behavior recognition technologies are often limited to classifying body movements or simply extrapolating trajectories. This not only makes it difficult to capture subtle early signals before an action occurs, but also lacks in-depth modeling of the spatial relationship between gaze focus and environmental risk sources. Consequently, the system cannot effectively distinguish between distracted pedestrians looking at their phones and interactive pedestrians observing road conditions and preparing to cross, easily leading to safety hazards such as mis-braking or delayed response. Therefore, this application proposes a pedestrian behavior recognition system for vehicle driving scenarios. The system first uses cameras and LiDAR to acquire scene data, quickly identifying high-risk pedestrians through global feature extraction and kinematic analysis. Then, it performs refined head localization and probabilistic gaze cone estimation on these targets to obtain gaze information with uncertainty metrics. Based on this, the system constructs an environmental risk saliency map containing dynamic targets and static elements, and innovatively utilizes a spatially aligned Gaussian kernel mechanism to weighted couple the pedestrian's gaze probability distribution with the environmental risk peak, thereby calculating an attention score that accurately reflects the pedestrian's level of attention to the risk source. Ultimately, the system integrates the attention score, full-body posture features, and vehicle status, and outputs a high-confidence behavior prediction result through the intent decoding module, effectively solving the problem of intent recognition under false interaction interference and improving driving safety.

[0026] Figure 1 This is a system block diagram of a pedestrian behavior recognition system in a vehicle driving scenario according to an embodiment of this application. Figure 2 This is a schematic diagram of data flow in a pedestrian behavior recognition system for a vehicle driving scenario according to an embodiment of this application. Figure 1 and Figure 2As shown, the pedestrian behavior recognition system 100 in a vehicle driving scene according to an embodiment of the present application comprises: a data acquisition module 110, configured to acquire camera data and lidar data; a pedestrian global feature extraction module 120, configured to perform pedestrian global feature extraction on the camera data and the lidar data to obtain a pedestrian target list; an intention preliminary screening and high-risk target locking module 130, configured to perform intention preliminary screening and high-risk target locking on the pedestrian target list based on a self-vehicle state to obtain a high-risk pedestrian list; a head region refinement and probabilistic gaze cone estimation module 140, configured to perform head region refinement and probabilistic gaze cone estimation on the high-risk pedestrian list based on the camera data to obtain a pedestrian list with gaze information; an attention focus analysis module 150, configured to perform attention focus analysis on the pedestrian list with gaze information, the camera data and the self-vehicle state to obtain a pedestrian attention score list; and an intention fusion decoding and behavior prediction output module 160, configured to perform intention fusion decoding and behavior prediction output on the pedestrian attention score list, the full-body posture features in the pedestrian target list and the self-vehicle state to obtain a behavior category and an intention probability.

[0027] In the above-described pedestrian behavior recognition system 100 in a vehicle driving scene, the data acquisition module 110 is configured to acquire camera data and lidar data. It can be understood that, since the pedestrian behavior recognition task requires both precise visual texture to analyze fine actions and gaze direction, and accurate spatial three-dimensional data to quantify collision risk and distance, a single sensor cannot meet the dual requirements of semantic richness and physical accuracy. Therefore, in the technical solution of the present application, camera data and lidar data are acquired to construct a holographic perception base covering the visible light frequency band and the spatial geometric dimension. In this way, the system can ensure that it has the most complete original information required to describe the state of pedestrians at the data source, providing solid data support for subsequent multi-modal feature fusion and deep intention reasoning.

[0028] More specifically, in one specific example of the present application, the system accesses different types of perception device data streams in real time through a vehicle-mounted sensor interface. First, a high-definition camera installed at the front windshield of the vehicle collects high-frame-rate RGB video stream data, which is rich in color and texture information and is used to capture the face orientation and body details of pedestrians. At the same time, a lidar installed on the roof or the front of the vehicle collects 360-degree or forward-view three-dimensional point cloud data, which measures the distance and contour of surrounding objects by emitting laser beams and receiving echoes. Finally, a high-precision time synchronization protocol (such as PTP or PPS signal) is used to timestamp align the two data streams, ensuring that each frame of image can find a frame of point cloud data closest in time, thereby providing synchronized input frame pairs for subsequent data fusion processing.

[0029] In the pedestrian behavior recognition system 100 described above for a vehicle driving scenario, the pedestrian global feature extraction module 120 is used to extract global pedestrian features from camera data and LiDAR data to obtain a list of pedestrian targets. It is understood that while the two-dimensional image data acquired by the camera is rich in texture details for recognizing limb movements, it is difficult to directly provide accurate depth and distance information. While the point cloud data acquired by the LiDAR has high-precision three-dimensional spatial measurement capabilities, it lacks semantic features for fine-grained behavior analysis. Single-modal data perception cannot meet the need for comprehensive analysis of pedestrian states. Therefore, in the technical solution of this application, global pedestrian features are further extracted from camera data and LiDAR data to obtain a list of pedestrian targets. This completes the spatiotemporal registration and correlation fusion of visual semantic features and physical spatial information, constructing a unified pedestrian data representation. This ensures that the system simultaneously grasps the pedestrian's fine posture attributes and their precise physical distance relative to the vehicle, laying a comprehensive and reliable perceptual data foundation for subsequent collision risk quantification assessment and micro-movement intention analysis.

[0030] Figure 3 This is a block diagram of a pedestrian global feature extraction module in a pedestrian behavior recognition system for a vehicle driving scenario according to an embodiment of this application. Figure 3 As shown, the pedestrian global feature extraction module 120 includes: a pedestrian detection and localization unit 121, used to perform vision-based pedestrian detection and localization on camera data to obtain a list of pedestrian body bounding boxes; a multimodal data registration and depth information association unit 122, used to perform multimodal data registration and depth information association on the list of pedestrian body bounding boxes and lidar data based on calibration parameters to obtain a list of pedestrian depths; a pedestrian global pose feature extraction unit 123, used to extract global pose features of pedestrians based on the list of pedestrian body bounding boxes and camera data to obtain a list of full-body pose features; and a multidimensional information aggregation unit 124, used to perform multidimensional information aggregation on the list of full-body pose features, the list of pedestrian depths, and the list of pedestrian body bounding boxes to obtain a list of pedestrian targets.

[0031] In the pedestrian behavior recognition system 100 for a vehicle driving scenario described above, the pedestrian detection and localization unit 121 is used to perform vision-based pedestrian detection and localization on camera data to obtain a list of pedestrian body bounding boxes. It is understood that since the original camera data contains a large amount of complex background noise and non-pedestrian targets, and subsequent depth information association and pose analysis both rely on the localization of specific target regions, a lack of accurate initial target screening mechanisms would lead to wasted computing power and difficulty in targeted feature extraction. Therefore, in the technical solution of this application, vision-based pedestrian detection and localization are further performed on the camera data to obtain a list of pedestrian body bounding boxes, thereby separating the pedestrian targets of interest from the panoramic image and determining their two-dimensional coordinate range on the image plane. This provides an accurate spatial index for subsequent multimodal data registration, ensuring that the system only processes effective pedestrian regions in depth, thus improving the efficiency and targeting of the overall perception process.

[0032] More specifically, in a concrete example of this application, the system first performs data preprocessing on the acquired vehicle-mounted forward-looking camera data, converting the original continuous video frame sequence to a color space and normalizing its size to fit the input specifications of the target detection model. Subsequently, the processed image data is input into a pre-trained deep convolutional neural network model for feature extraction and regression analysis. This model uses a multi-layer convolutional structure to capture pedestrian texture features in the image and predicts candidate regions containing potential pedestrian targets and their corresponding confidence scores on the feature map. Finally, non-maximum suppression and confidence threshold filtering are performed on the original prediction results output by the model to remove redundant detection boxes with excessive overlap and low confidence, retaining the detection results with the highest confidence and most accurate location, thereby generating a list of pedestrian body bounding boxes containing the coordinates of the top-left corner, width, and height information of each pedestrian target.

[0033] In the pedestrian behavior recognition system 100 for a vehicle driving scenario described above, the multimodal data registration and depth information association unit 122 is used to perform multimodal data registration and depth information association on the pedestrian body bounding box list and LiDAR data based on calibration parameters to obtain a pedestrian depth list. It is understood that since the two-dimensional image acquired by the camera only reflects the positional information of the pixel plane, it cannot directly quantify the actual physical distance between the pedestrian and the vehicle. This distance parameter is crucial for collision risk assessment. Furthermore, although the LiDAR point cloud data contains precise three-dimensional coordinates, it lacks a direct correspondence with specific pedestrian targets in the image. Therefore, in the technical solution of this application, multimodal data registration and depth information association are further performed on the pedestrian body bounding box list and LiDAR data based on calibration parameters to obtain a pedestrian depth list. This maps the discrete three-dimensional spatial point cloud onto the two-dimensional image plane and performs geometric matching with the corresponding pedestrian detection boxes. In this way, each visually detected pedestrian object can be given precise depth attributes, thereby providing the necessary physical distance basis for subsequent collision time calculation and the construction of a three-dimensional line-of-sight cone.

[0034] More specifically, in a concrete example of this application, the system first reads the pre-calibrated intrinsic parameter matrix of the camera and the extrinsic parameter matrix of the LiDAR relative to the camera, and constructs a projection transformation model from the LiDAR coordinate system to the pixel coordinate system based on these calibration parameters. Then, using this model, the system batch projects the 3D point cloud data acquired by the LiDAR in the current frame onto the 2D image plane, generating a set of projected points with depth information. Next, it iterates through each detection box in the pedestrian body bounding box list, filtering out all projected points falling within the pixel range of that detection box to form a candidate point cloud set for the pedestrian. Finally, it performs statistical filtering on the depth values ​​in the candidate point cloud set, determining the representative distance value of the pedestrian target relative to the vehicle by calculating the median of the point cloud depth within the set or the average value after removing outliers, and generating a pedestrian depth list corresponding one-to-one with the body bounding boxes.

[0035] In the pedestrian behavior recognition system 100 described above for a vehicle driving scenario, the pedestrian global pose feature extraction unit 123 is used to extract global pose features of the pedestrian based on a list of pedestrian body bounding boxes and camera data to obtain a full-body pose feature list. It is understood that simple two-dimensional detection boxes can only provide the pedestrian's position and scale information, and cannot reflect the pedestrian's body orientation, limb movements, and other fine-grained states. These pose features are crucial for initially determining whether a pedestrian has a tendency to cross the road or whether their attention is focused. Therefore, in the technical solution of this application, global pose features of the pedestrian are further extracted based on a list of pedestrian body bounding boxes and camera data to obtain a full-body pose feature list, thereby analyzing the skeletal key point structure and overall movement posture of the pedestrian from the image texture. This allows the system to obtain preliminary clues about the pedestrian's intentions in the early perception stage, providing key semantic feature support for the subsequent rapid screening of high-risk targets.

[0036] More specifically, in a concrete example of this application, the system first crops local regions of interest (ROIs) image patches for each pedestrian target from the original camera image based on the coordinate information in the pedestrian body bounding box list. Then, these image patches are input into a lightweight pose estimation neural network model. This model extracts keypoint heatmaps from the image using a feature pyramid structure and regresses and predicts the two-dimensional pixel coordinates of multiple human skeletal keypoints, including the shoulders, hips, knees, and head. Next, based on the geometric positional relationships of these keypoints, the system further calculates the pedestrian's principal axis vector and a coarse orientation angle, while combining confidence information to generate a high-dimensional feature vector describing the pedestrian's overall motion state. Finally, all the processed keypoint coordinate data, orientation angles, and feature vectors are integrated into structured data units, and a list of full-body pose features corresponding one-to-one with the detection boxes is output sequentially.

[0037] In the pedestrian behavior recognition system 100 for a vehicle driving scenario described above, the multi-dimensional information aggregation unit 124 is used to aggregate multi-dimensional information from the full-body posture feature list, the pedestrian depth list, and the pedestrian body bounding box list to obtain a pedestrian target list. It is understood that since the aforementioned steps acquire pedestrian feature data from three independent dimensions—visual detection, depth measurement, and posture analysis—if these discrete data lists are not effectively structured and temporally correlated, subsequent decision-making modules will find it difficult to call a unified pedestrian object model and maintain continuous observation of the same target over time. Therefore, in the technical solution of this application, multi-dimensional information aggregation is further performed on the full-body posture feature list, the pedestrian depth list, and the pedestrian body bounding box list to obtain a pedestrian target list. This encapsulates the dispersed multimodal features into a unified pedestrian entity object and assigns it a unique identity in the time series. In this way, a complete pedestrian state description containing location, distance, action, and ID information can be constructed, providing standardized data input for subsequent intent tracking and behavior prediction.

[0038] More specifically, in a concrete example of this application, the system first combines the pedestrian body bounding boxes detected in the current frame, the corresponding pedestrian depth values, and the extracted full-body pose features based on the one-to-one correspondence of list indices to construct a candidate detection object set for the current moment. Then, using a multi-object tracking algorithm, the system calculates the association cost matrix between the candidate detection objects in the current frame and the existing tracking target trajectories in the previous frame. This matrix is ​​weighted based on the bounding box intersection-union ratio, feature similarity, and motion state prediction error. Next, the Hungarian algorithm is used to solve for the optimal matching of the cost matrix. Successfully matched detection objects are updated to their corresponding tracking trajectories, inheriting their historical IDs and motion state information; unmatched detection objects are initialized as new tracking targets and assigned entirely new unique IDs. Finally, all active tracking target information is organized and encapsulated into a structured object containing ID, bounding box, depth, pose features, and velocity estimates, outputting the final list of pedestrian targets.

[0039] In the pedestrian behavior recognition system 100 for a vehicle driving scenario described above, the intent screening and high-risk target locking module 130 is used to perform intent screening and high-risk target locking on the pedestrian target list based on the vehicle's status to obtain a high-risk pedestrian list. It is understandable that in complex urban driving environments, there are usually many low-risk pedestrians in the field of vision who are stationary or walking significantly away from the lane. Performing high-computational-power-consuming head-level fine-grained analysis and eye-tracking on all targets would result in a serious waste of onboard computing resources and would fail to meet real-time requirements. Therefore, in the technical solution of this application, the intent screening and high-risk target locking on the pedestrian target list are further performed based on the vehicle's status to obtain a high-risk pedestrian list. This utilizes low-cost kinematic features to quickly filter out non-threatening targets, retaining only key objects with potential collision risks or a tendency to cross the road. This allows for significant optimization of system computational efficiency through a hierarchical processing strategy, ensuring that valuable computing resources are concentrated on in-depth intent analysis of pedestrians who truly pose a safety threat, thereby improving the overall system's response speed and practical effectiveness.

[0040] Figure 4 This is a block diagram of an intent screening and high-risk target locking module in a pedestrian behavior recognition system for a vehicle driving scenario, according to an embodiment of this application. Figure 4 As shown, the intent screening and high-risk target locking module 130 includes: a pedestrian kinematics and orientation state analysis unit 131, used to perform pedestrian kinematics and orientation state analysis on each pedestrian object in the pedestrian target list based on the vehicle's state to obtain the pedestrian's body orientation angle and pedestrian's lateral relative velocity; a collision risk quantification assessment unit 132, used to perform collision risk quantification assessment based on the longitudinal components of pedestrian depth and relative velocity in the pedestrian object to obtain the collision time; and a high-risk target screening unit 133, used to perform high-risk target screening on the pedestrian target list based on the collision time and pedestrian's body orientation angle to obtain a high-risk pedestrian list.

[0041] In the pedestrian behavior recognition system 100 under the aforementioned vehicle driving scenario, the pedestrian kinematics and orientation state analysis unit 131 is used to perform pedestrian kinematics and orientation state analysis on each pedestrian object in the pedestrian target list based on the vehicle's state to obtain the pedestrian's body orientation angle and lateral relative velocity. It is understood that since a pedestrian's body orientation is often a priori indication of their next movement direction, and the pedestrian's lateral movement velocity relative to the vehicle is a key physical quantity for determining whether they are entering the vehicle's driving path, static position information alone cannot accurately assess dynamic collision risk. Therefore, in the technical solution of this application, based on the vehicle's state, pedestrian kinematics and orientation state analysis is further performed on each pedestrian object in the pedestrian target list to obtain the pedestrian's body orientation angle and lateral relative velocity, thereby quantifying the pedestrian's current posture tendency and the dynamic trend of spatial conflict with the vehicle. This provides specific quantitative indicators for subsequent collision risk assessment, enabling the system to distinguish between behaviors with different risk levels, such as standing on the roadside and immediately crossing.

[0042] More specifically, in a concrete example of this application, the system first traverses the list of pedestrian targets, extracts the coordinates of key points on the shoulders and hips from the full-body posture features of each pedestrian object, calculates the vector connecting the midpoints of the two shoulders and the midpoints of the two hips as the principal axis vector of the body, and combines it with the real-time heading angle data of the vehicle to calculate the geometric angle between this principal axis vector and the vehicle's driving direction, thereby obtaining the body facing angle describing the direction the pedestrian is facing. Simultaneously, using the depth changes and pixel displacements of the pedestrian object in consecutive multi-frame images, combined with the real-time velocity vector obtained by the vehicle via the CAN bus, a relative motion model is constructed to calculate the absolute velocity vector of each pedestrian in the world coordinate system. Next, this absolute velocity vector is projected onto the vehicle coordinate system with the vehicle as the origin, decomposed into a longitudinal component along the lane direction and a lateral component perpendicular to the lane direction, and the lateral component is extracted as the pedestrian's lateral relative velocity characterizing the pedestrian's crossing tendency.

[0043] In the pedestrian behavior recognition system 100 for a vehicle driving scenario described above, the collision risk quantification assessment unit 132 is used to quantify the collision risk based on the longitudinal components of pedestrian depth and relative speed in the pedestrian object to obtain the collision time. It is understood that since spatial distance alone cannot truly reflect the urgency of a potential collision, a pedestrian physically close but moving away from the vehicle has a significantly lower risk level than a pedestrian slightly farther away but approaching head-on at a high relative speed. Relying solely on location data for filtering would lead to inefficient allocation of computational resources and delayed safety warnings. Therefore, in the technical solution of this application, collision risk quantification assessment is further performed based on the longitudinal components of pedestrian depth and relative speed in the pedestrian object to obtain the collision time. This introduces a time dimension into the risk assessment model, predicting the remaining time window before potential contact occurs if the current movement trend remains unchanged. This allows the system to dynamically classify targets according to urgency, ensuring that valuable perception and decision-making resources are prioritized for targets with immediate collision risk, thereby optimizing system computing power while ensuring safety.

[0044] More specifically, in a particular example of this application, the system first extracts the depth attribute value of each pedestrian object from the list of pedestrian targets, and uses it as the basis for determining the distance between the vehicle and the first pedestrian. Longitudinal distance between pedestrians Simultaneously, the component along the vehicle's direction of travel is extracted from the previously calculated relative velocity vector and used as the pedestrian's longitudinal relative velocity. Subsequently, the system employs a constant-velocity model based on physical kinematics to numerically calculate the collision risk. This calculation process strictly follows the following formula: In this formula, Representing the The estimated collision time for each pedestrian The longitudinal distance obtained from pedestrian depth, Let be the longitudinal relative velocity of the pedestrian. The logical judgment part in the formula indicates that: when the condition is met... At this point, the longitudinal distance between the pedestrian and the vehicle is decreasing, meaning they are in a dangerous state of approaching each other. The collision time at this time is equal to the current longitudinal distance divided by the relative approach speed (i.e., the negative of the relative speed). And when the condition is met... When the time of collision is positive infinity, it means that the pedestrian is moving away from the vehicle or is stationary longitudinally relative to the vehicle. This indicates that there is no risk of collision in the current state. Taking a typical urban road scenario as an example, if a vehicle is traveling at 30 kilometers per hour and detects a pedestrian crossing an intersection at walking speed 30 meters ahead, the system will calculate the relative approach speed between the two. If the calculated speed is... If the value is less than the preset safety threshold (e.g., 2.5 seconds), the logic will trigger the subsequent high-risk target locking mechanism, marking the pedestrian as a key object that urgently needs line-of-sight analysis.

[0045] In the pedestrian behavior recognition system 100 under the aforementioned vehicle driving scenario, the high-risk target screening unit 133 is used to screen the pedestrian target list for high-risk targets based on collision time and pedestrian body orientation angle to obtain a high-risk pedestrian list. It is understandable that a single evaluation indicator often cannot fully reflect the true threat level of a pedestrian. For example, a pedestrian facing a vehicle but at a safe distance does not pose an immediate threat, and a pedestrian walking parallel to the sidewalk with their back to the vehicle, although relatively close, does not require emergency intervention. If only collision time or body orientation is relied upon for a single-dimensional judgment, it is easy to miss detections or make false alarms, thus wasting limited computing resources on non-critical targets. Therefore, in the technical solution of this application, the pedestrian target list is further screened for high-risk targets based on collision time and pedestrian body orientation angle to obtain a high-risk pedestrian list, thereby constructing a multi-dimensional filtering mechanism that couples time urgency and spatial posture intent. In this way, it is possible to accurately locate targets that simultaneously possess the potential for physical collision and the tendency to cross over, providing a precise set of inputs for subsequent high-cost head refinement and line-of-sight estimation modules, thereby maximizing the system's operational efficiency while ensuring full coverage of safety perception.

[0046] More specifically, in a concrete example of this application, the system first presets a collision time safety threshold and a body orientation angle warning threshold based on the vehicle's current braking performance and road environment type. Then, the system iterates through each pedestrian object in the pedestrian target list, extracting its calculated collision time and body orientation angle values. Next, the system performs a joint logic judgment on each pedestrian object: specifically, it determines whether the pedestrian's body orientation angle causes their body axis to intersect with the vehicle's travel path, and whether this angle is within the warning threshold range, to confirm whether they have a crossing posture; simultaneously, it checks whether their collision time is less than the set safety threshold to confirm whether they have a time urgency. Only when a pedestrian object simultaneously meets both the conditions of having a crossing posture and a time urgency for collision is it marked as a valid threat target. Finally, the system extracts all pedestrian objects marked as valid threats from the original list, reorganizes them to generate a high-risk pedestrian list, and ignores or marks low-risk pedestrians who walk parallel to the lane or move away from the vehicle, thereby completing the hierarchical control of target risk.

[0047] In the pedestrian behavior recognition system 100 for a vehicle driving scenario described above, the head region refinement and probabilistic gaze cone estimation module 140 is used to refine the head region and estimate the probabilistic gaze cone based on camera data to obtain a pedestrian list with gaze information. It is understandable that since full-body posture and movement trajectory only reflect the macroscopic physical state of a pedestrian, they cannot reveal their internal psychological cognitive processes. Especially in determining whether a pedestrian sees and is aware of the vehicle's presence, relying solely on body movements can easily lead to false interaction misjudgments. For example, a pedestrian may indeed be facing the lane preparing to cross, but their attention is actually drawn to their phone, completely unaware of the approaching vehicle. Therefore, in the technical solution of this application, the head region refinement and probabilistic gaze cone estimation are further performed on the high-risk pedestrian list based on camera data to obtain a pedestrian list with gaze information, thereby introducing microscopic visual-cognitive cues and quantifying the inherent uncertainty in gaze estimation. In this way, the system can deeply analyze the pedestrian's attention allocation mechanism, and even under conditions of long distance or low resolution, it can robustly infer the pedestrian's observation focus through probability models, thereby providing decisive perceptual basis for distinguishing between unconscious dangerous behavior and conscious avoidance behavior.

[0048] Figure 5 This is a block diagram of the head region refinement and probabilistic gaze cone estimation module in a pedestrian behavior recognition system for vehicle driving scenarios according to an embodiment of this application. Figure 5 As shown, the head region refinement and probabilistic gaze cone estimation module 140 includes: a high-risk pedestrian head localization and image enhancement unit 141, used to perform high-risk pedestrian head localization and image enhancement on each high-risk pedestrian object in the high-risk pedestrian list based on camera data to obtain an enhanced head image; a gaze parameter joint estimation unit 142, used to perform gaze parameter joint estimation based on a deep network on the enhanced head image to obtain a gaze parameter tuple, the gaze parameter tuple including a central gaze vector and an uncertainty scalar; a probabilistic gaze cone modeling unit 143, used to perform probabilistic gaze cone modeling on the gaze parameter tuple to obtain a probabilistic gaze cone object; and an information aggregation unit 144, used to aggregate information from the probabilistic gaze cone object and the high-risk pedestrian object to obtain a pedestrian list with gaze information.

[0049] In the pedestrian behavior recognition system 100 described above for a vehicle driving scenario, the high-risk pedestrian head localization and image enhancement unit 141 is used to perform high-risk pedestrian head localization and image enhancement on each high-risk pedestrian in the high-risk pedestrian list based on camera data to obtain an enhanced head image. It is understood that, during vehicle operation, high-risk pedestrians are often located tens of meters away from the vehicle. At this time, the proportion of their head pixels in the entire image is extremely low, and they are often affected by environmental noise such as uneven lighting, motion blur, or partial occlusion. Directly estimating gaze on the low-resolution original full-body image will lead to huge prediction errors, seriously affecting the reliability of the system. Therefore, in the technical solution of this application, high-risk pedestrian head localization and image enhancement are further performed on each high-risk pedestrian in the high-risk pedestrian list based on camera data to obtain an enhanced head image. This allows for accurate separation of the head region containing rich gaze cues from the full-body background, and the reconstruction of missing high-frequency texture details through image signal processing technology. This significantly improves the quality of input data for the gaze estimation network, ensuring that the model can still capture subtle features such as eye contours and facial orientation even at long distances or under poor lighting conditions, thereby greatly improving the accuracy and stability of gaze prediction.

[0050] More specifically, in a concrete example of this application, the system first, for each pedestrian object in the high-risk pedestrian list, uses its known body bounding box coordinates, combined with prior knowledge of human anatomical proportions, to crop an image patch located in the upper region of the body bounding box as a candidate search region for the head. Then, a lightweight head detection model specifically optimized for small targets is run within this candidate region to locate the precise bounding box of the head with sub-pixel accuracy. Next, the cropped original low-resolution head image is input into a super-resolution reconstruction network, which uses high-frequency texture mapping relationships learned by deep convolutional layers to upsample and restore details of the image, generating a high-resolution magnified head image. Finally, adaptive histogram equalization and illumination normalization are performed on the magnified image to eliminate the effects of shadows and overexposure, and the processed image is adjusted to a standard size, outputting an enhanced head image for subsequent gaze analysis.

[0051] In the pedestrian behavior recognition system 100 in a vehicle driving scenario described above, the joint gaze parameter estimation unit 142 is used to perform joint gaze parameter estimation based on a deep network on the enhanced head image to obtain a gaze parameter tuple, which includes a central gaze vector and an uncertainty scalar. It is understood that since the pedestrian's true gaze direction depends not only on head posture but also on fine eye movements, and gaze estimation in monocular images is inherently an ill-posed problem, a single, deterministic gaze vector prediction is often unreliable when imaging noise or viewing angle limitations exist. It cannot reflect the model's confidence in the current prediction result, and blindly trusting highly uncertain predictions will mislead subsequent decisions. Therefore, in the technical solution of this application, the enhanced head image is further subjected to joint gaze parameter estimation based on a deep network to obtain a gaze parameter tuple including a central gaze vector and an uncertainty scalar. This allows for the explicit quantification of cognitive uncertainty caused by image quality or posture blurring while predicting the most likely gaze direction. This allows subsequent modules to dynamically adjust their reliance on line-of-sight information based on the level of uncertainty, assigning high weight to clear targets and remaining cautious towards ambiguous ones, thereby building a more robust and secure perception system.

[0052] More specifically, in a concrete example of this application, the system employs a specially designed dual-branch convolutional neural network as the gaze estimator. First, the enhanced head image is input into the network's backbone feature extraction module to extract a deep semantic feature map containing facial keypoints and texture information. Subsequently, this feature map is simultaneously fed into two parallel fully connected layer branches: the first branch regresses and predicts the gaze unit direction vector in 3D space, representing the central gaze direction that the model considers to have the highest probability under the current image input; the second branch predicts a non-negative scalar value as the concentration / variance parameter of the direction distribution (e.g., κ of the vMF distribution or σ of the normal angle model), directly characterizing the discreteness of the gaze distribution, i.e., the magnitude of uncertainty. During the training phase, the network jointly optimizes these two branches by maximizing the log-likelihood function of the gaze distribution. Finally, during the inference phase, the network outputs both the central gaze vector and the uncertainty scalar in a single forward propagation, encapsulating them into a gaze parameter tuple for subsequent probabilistic modeling.

[0053] In the pedestrian behavior recognition system 100 in the aforementioned vehicle driving scenario, the probabilistic gaze cone modeling unit 143 is used to perform probabilistic gaze cone modeling on the gaze parameter tuple to obtain a probabilistic gaze cone object. It is understood that since the central gaze vector and uncertainty scalar obtained in the previous step are merely abstract numerical parameters, they cannot be directly used for spatial geometric calculations to determine whether a pedestrian's gaze covers a specific environmental object. A mathematical model capable of mapping this probability distribution to three-dimensional physical space is needed. Therefore, in the technical solution of this application, probabilistic gaze cone modeling is further performed on the gaze parameter tuple to obtain a probabilistic gaze cone object, thereby constructing a three-dimensional cone structure with the pedestrian's head as the vertex and diverging towards the gaze direction. The probability density distribution within this cone strictly follows the predicted uncertainty parameters. In this way, the pedestrian's visual attention range can be visualized as a physical field, enabling the system to accurately assess the probability of the pedestrian's attention to different environmental elements by calculating the spatial intersection of this cone with entities such as roads and vehicles.

[0054] More specifically, in a concrete example of this application, the system uses the central gaze vector in the gaze parameter tuple as the central axis of the cone, and uses an uncertainty scalar to define the cone's divergence angle or probability decay rate. Specifically, the system employs a vMF distribution or a similar directional probability distribution function to construct a probability density function defined on a three-dimensional unit sphere. In this function, as the angle between the spatial direction vector and the central gaze vector increases, the probability density value decays exponentially, and the rate of decay is entirely controlled by the uncertainty scalar: the lower the uncertainty, the faster the decay, resulting in a more slender and focused gaze cone; the higher the uncertainty, the slower the decay, resulting in a wider and more divergent gaze cone. Finally, the system encapsulates this probability density function and its geometric parameters into a queryable probability gaze cone object. This object supports inputting any spatial coordinate point and returns the probability value of that point being gazed at by a pedestrian, thus completing the conversion from parameters to a spatial field.

[0055] In the pedestrian behavior recognition system 100 for a vehicle driving scenario described above, the information aggregation unit 144 is used to aggregate information from probabilistic line-of-sight cone objects and high-risk pedestrian objects to obtain a list of pedestrians with line-of-sight information. It is understandable that since the probabilistic line-of-sight cone objects are generated in an independent computational process, while the final behavioral decision is still made using pedestrian entity objects containing comprehensive information such as ID, location, and speed, if the newly generated line-of-sight attributes are not bound to the original pedestrian identity information, the system will not know which specific high-risk pedestrian the line-of-sight cone belongs to, leading to data fragmentation. Therefore, in the technical solution of this application, information aggregation is further performed on the probabilistic line-of-sight cone objects and high-risk pedestrian objects to obtain a list of pedestrians with line-of-sight information, thereby completing the final assembly from single-modal features to multi-dimensional semantic objects, ensuring that each high-risk pedestrian object possesses full-stack perception data from macroscopic movement to microscopic line-of-sight. This provides a structurally complete and clearly indexed input data stream for the subsequent attention focus analysis module, ensuring seamless integration between feature engineering and decision logic.

[0056] More specifically, in a concrete example of this application, the system establishes a temporary list container and then initiates an iterative loop to traverse each high-risk pedestrian index in the processing queue. In each iteration, the system first extracts the original high-risk pedestrian object corresponding to the current index, and then obtains the probability line-of-sight cone object generated in the previous steps that matches the pedestrian ID. Next, the system calls the attribute extension interface of the data object to embed the probability line-of-sight cone object as a new member variable into the data structure of the pedestrian object, thereby giving the pedestrian object the ability to describe its line-of-sight distribution. After the embedding is completed, the updated pedestrian object is added to the temporary list container. When all high-risk pedestrians have been processed, the container is output as the final list of pedestrians with line-of-sight information. Each element in this list now contains not only position and speed, but also deeper cognitive information describing where they are looking and how accurately they are looking.

[0057] In the pedestrian behavior recognition system 100 under the aforementioned vehicle driving scenario, the attention focus analysis module 150 is used to perform attention focus analysis on a list of pedestrians with gaze information, camera data, and vehicle status to obtain a list of pedestrian attention scores. It is understood that since a pedestrian's gaze itself has no meaning, it only has intent recognition value when it interacts with a specific risk source in the environment (such as an oncoming vehicle). Furthermore, the direction of the gaze alone cannot directly reflect the intensity of this interaction. For example, although the angle of a pedestrian looking at an open road and looking at a speeding vehicle may be similar, the safety implications are completely different. Therefore, in the technical solution of this application, attention focus analysis is further performed on the list of pedestrians with gaze information, camera data, and vehicle status to obtain a list of pedestrian attention scores. This projects the pedestrian's subjective gaze onto the objective dynamic traffic scene, and quantifies the pedestrian's level of awareness of potential danger by calculating the spatiotemporal overlap between the two. In this way, the complex cognitive psychological process can be transformed into a numerical attention score, intuitively reflecting whether the pedestrian has paid attention to the vehicle, providing the most direct criterion for distinguishing between unintentional intrusion and intentional avoidance.

[0058] Figure 6 This is a block diagram of the attention focus analysis module in a pedestrian behavior recognition system for a vehicle driving scenario according to an embodiment of this application. Figure 6 As shown, the attention focus analysis module 150 includes: a scene element perception and semantic information extraction unit 151, used to perform scene element perception and semantic information extraction on camera data to obtain a semantic segmentation map and a dynamic target list; an environmental risk quantification unit 152, used to perform environmental risk quantification on the semantic segmentation map, the dynamic target list, and the vehicle status to obtain an environmental risk saliency map; a gaze probability map acquisition unit 153, used to project each pedestrian object with gaze information in the pedestrian list with gaze information based on calibration parameters to obtain a gaze probability map; and a pedestrian attention score list generation unit 154, used to generate a pedestrian attention score list based on the gaze probability map, the environmental risk saliency map, and the pedestrian list with gaze information.

[0059] In the pedestrian behavior recognition system 100 for a vehicle driving scenario described above, the scene element perception and semantic information extraction unit 151 is used to perform scene element perception and semantic information extraction on camera data to obtain a semantic segmentation map and a dynamic target list. It is understood that since a pedestrian's attention focus is relative to specific objects in the environment, if the system does not understand which areas in the current scene are roads, which areas are vehicles, and which areas are traffic lights, it cannot determine where the pedestrian's gaze is actually falling, and therefore cannot assess the safety of their behavior. Therefore, in the technical solution of this application, scene element perception and semantic information extraction are further performed on camera data to obtain a semantic segmentation map and a dynamic target list, thereby constructing a full-element digital environment model, clarifying the semantic attributes of each spatial location in the scene and the existence state of dynamic objects. This provides a base map for subsequent risk saliency map generation, ensuring that the system can identify all potential objects of interest, thereby supporting refined gaze-environment interaction analysis.

[0060] More specifically, in a concrete example of this application, the system employs a parallel processing architecture to perform deep analysis on real-time input RGB images from a camera. On one hand, the image is input into a real-time semantic segmentation network model, which performs pixel-level classification inference and outputs a semantic segmentation map of the same size as the original image, where each pixel value corresponds to a static environment category such as road, sidewalk, vegetation, building, or traffic sign. On the other hand, the same frame of image is input into an object detection and tracking network to identify and lock onto all dynamic traffic participants in the scene, such as other motor vehicles, non-motor vehicles, and pedestrians, and outputs a dynamic object list containing these object categories, bounding boxes, and motion IDs. Through this process, the system transforms the unstructured video stream into a structured scene semantic description, providing the necessary environmental context for understanding what people are seeing.

[0061] In the pedestrian behavior recognition system 100 in the aforementioned vehicle driving scenario, the environmental risk quantification unit 152 is used to quantify the environmental risk of the semantic segmentation map, the dynamic target list, and the vehicle's status to obtain an environmental risk saliency map. It is understandable that not all objects in the environment have the same safety warning significance for pedestrians. For example, the risk level of a car approaching at high speed is obviously much higher than that of a stationary flower bed on the roadside. Treating all environmental elements equally would fail to highlight the key areas that pedestrians should focus on, resulting in a lack of specificity in the calculation of attention scores. Therefore, in the technical solution of this application, the semantic segmentation map, the dynamic target list, and the vehicle's status are further quantified to obtain an environmental risk saliency map. This allows for the assignment of differentiated weight values ​​to each pixel in the scene based on the potential threat level posed by objects to pedestrians. This generates a heatmap-like risk distribution representation, highlighting truly dangerous areas (such as the vehicle's position or oncoming vehicles) in the map, ensuring that the system only considers these high-risk areas to be in a valid risk assessment state when pedestrians are looking at them.

[0062] More specifically, in a concrete example of this application, the system first initializes a floating-point matrix of all zeros as a base risk map. Next, based on the category labels in the semantic segmentation map, basic static risk weights are assigned to traffic-sensitive areas such as lanes and zebra crossings. Then, it iterates through each traffic participant in the dynamic target list, combining the vehicle's state data and radar ranging information to calculate the relative distance and approach speed between the target and the vehicle. Using a preset risk attenuation function, the closer the distance and the higher the relative speed, the greater the risk weight value of the target. The system then superimposes the calculated dynamic risk weights onto the corresponding target pixel areas in the base risk map, especially assigning the highest risk weight to the area in front of the vehicle (or directly in front of it). Finally, the superimposed risk map is Gaussian smoothed to eliminate jagged edges and simulate the spatial diffusion effect of risk, generating the final environmental risk saliency map.

[0063] In the pedestrian behavior recognition system 100 for a vehicle driving scenario described above, the gaze probability map acquisition unit 153 is used to project each pedestrian object with gaze information in the pedestrian list with gaze information based on calibration parameters to obtain a gaze probability map. It is understood that since the probability gaze cone is a geometric model defined in three-dimensional space, while the environmental risk saliency map is a pixel matrix defined on a two-dimensional image plane, their dimensions and coordinate systems are not unified, making direct mathematical operations impossible to assess the degree of overlap. Therefore, in the technical solution of this application, the gaze probability map is further obtained by projecting each pedestrian object with gaze information in the pedestrian list based on calibration parameters, thereby mapping the three-dimensional gaze distribution onto a two-dimensional visual plane, aligning it spatially with the environmental risk map. This transforms the abstract problem of spatial geometric intersection into an intuitive calculation of image region overlap, facilitating subsequent calculation of attention scores using efficient matrix operations.

[0064] More specifically, in a concrete example of this application, the system reads the probability gaze cone data structure carried by each pedestrian object in the list. First, dense ray sampling is performed around the central axis of the gaze cone in three-dimensional space, based on the solid angle range determined by its uncertainty parameters. Then, using the camera intrinsic matrix and extrinsic calibration data, each sampled ray is projected from the three-dimensional world coordinate system to the two-dimensional image pixel coordinate system to determine its landing point position on the image plane. Next, based on the probability density value of the ray in the three-dimensional cone, the pixels around the projection landing point are assigned and accumulated to construct a single-channel probability distribution map of the same size as the original image. Finally, the map is normalized to ensure that the sum of all pixel values ​​is 1, thereby generating a gaze probability map specific to that pedestrian. In this map, the brighter the area, the greater the probability that the pedestrian is looking in that direction.

[0065] In the pedestrian behavior recognition system 100 for a vehicle driving scenario described above, the pedestrian attention score list generation unit 154 is used to generate a pedestrian attention score list based on a gaze probability map, an environmental risk saliency map, and a list of pedestrians with gaze information. It is understandable that if a simple global weighted integral method is used to calculate pedestrian attention scores—that is, directly multiplying and summing the pedestrian's gaze probability map and the environmental risk saliency map pixel by pixel—it fails to effectively distinguish between two fundamentally different behavioral patterns: active interaction and passive scanning. Specifically, existing methods neglect a crucial special relationship: the spatial proximity and directional consistency between the gaze focus and the risk source. Due to the lack of consideration for spatial proximity, a high-risk object far from the gaze center, even if only located at the edge of a cone with extremely low gaze probability, will still generate a non-zero attention contribution. This makes it impossible for the system to distinguish whether a pedestrian is precisely looking at the risk source or unintentionally scanning its direction. Furthermore, due to the lack of assessment of directional consistency, this method cannot quantify the coupling effect between the confidence level of gaze prediction and spatial alignment. A divergent gaze cone caused by pose ambiguity and a highly convergent, precise gaze may produce similar scores in the calculation. This mechanistic defect makes the system less sensitive to strong interactive signals and less effective at suppressing weakly correlated noise, thus affecting the accuracy and robustness of the final intent judgment. For example, at a busy intersection, a pedestrian may only quickly scan their surroundings without actually noticing an oncoming vehicle. If the system misjudges that the pedestrian has noticed the vehicle and abandons braking, the consequences could be disastrous. Alternatively, a pedestrian may be intently watching a vehicle; if the system misjudges this due to the high uncertainty of their gaze and applies emergency braking, it will affect traffic efficiency.

[0066] Therefore, in the technical solution of this application, a pedestrian attention score list is generated based on the gaze probability map, the environmental risk saliency map, and the pedestrian list with gaze information. A spatially aligned Gaussian kernel-based attention focus enhancement calculation method is proposed. This method introduces a virtual focus centered on the environmental risk peak, significantly amplifying the strong interactive signal of precise gaze focusing on the risk source. This allows for a precise numerical distinction between a glance and a fixed gaze, and dynamically adjusts the confidence level of the score based on the reliability of gaze prediction. In this way, when facing complex scenarios, the system activates a high-priority avoidance strategy only when pedestrians exhibit clear and high-confidence risk attention behavior, thereby reducing unnecessary emergency braking while ensuring safety and improving the human-like decision-making level of autonomous driving.

[0067] Figure 7 This is a block diagram of a pedestrian attention score list generation unit in a pedestrian behavior recognition system for a vehicle driving scenario according to an embodiment of this application. Figure 7As shown, the pedestrian attention score list generation unit 154 includes: an environmental risk peak extraction and localization subunit 1541, used to extract and localize environmental risk peaks from the environmental risk saliency map to obtain a risk peak list; a spatially aligned Gaussian kernel map construction subunit 1542, used to construct a spatially aligned Gaussian kernel map based on the risk peak list; and a pedestrian attention score determination subunit 1543, used to determine the pedestrian attention score based on the spatially aligned Gaussian kernel map, the gaze probability map, and the probability gaze cone object.

[0068] In the pedestrian behavior recognition system 100 for a vehicle driving scenario described above, the environmental risk peak extraction and localization subunit 1541 is used to extract and localize environmental risk peaks from the environmental risk saliency map to obtain a list of risk peaks. It is understood that since the environmental risk saliency map is usually a continuous and diffuse heatmap, it may contain a large amount of low-intensity background noise or large areas of uniform high-risk regions (such as an entire lane). If attention calculation is performed directly based on the entire map, there is a lack of clear spatial focus points, making it difficult to highlight the core threat points that pedestrians should pay the most attention to (such as the center of the vehicle's front or the nearest point). Therefore, in the technical solution of this application, the environmental risk saliency map is further processed to extract and localize environmental risk peaks to obtain a list of risk peaks. This transforms the continuous, diffuse risk map into discrete, key risk source localization points, providing anchor points for subsequently constructing accurate attention focus. For example, in congested traffic, although the entire road is a risk zone, the most pressing threat often comes from the front of the vehicle closest to the pedestrian. By extracting risk peaks, the system can accurately pinpoint this most dangerous point, thereby determining whether the pedestrian's gaze is truly focused on it.

[0069] More specifically, in a particular example of this application, the system provides a risk saliency map of the input. A non-maximum suppression algorithm is applied, which identifies the highest points of local risk in the image by comparing pixel values ​​within a local sliding window and suppressing pixels with non-local maxima. Next, an adaptive risk threshold is set to filter out low-intensity peaks (such as distant non-motorized vehicles), retaining only the peaks of high-risk targets. Finally, the pixel coordinates of these points and their corresponding risk values ​​are extracted to form a sparse list of risk peaks. Each element in this list represents a specific, discrete hazard center, such as the center point of the front bumper of an oncoming vehicle or the location of a traffic light about to change, providing a precise geometric reference for subsequent calculations of the spatial alignment between the line of sight and these specific points.

[0070] In the pedestrian behavior recognition system 100 for a vehicle driving scenario described above, the spatially aligned Gaussian kernel map construction subunit 1542 is used to construct a spatially aligned Gaussian kernel map based on a list of risk peaks. It is understood that since risk peak points are merely discrete coordinates, lacking a description of their radiating influence on the surrounding space, and pedestrians' attention is often focused on a small area around the risk source rather than a precise single pixel. For example, when a pedestrian looks at a car, their gaze may fall on different locations such as the front of the car, the headlights, or the windshield; all of these should be considered valid interactions. If attention calculation is performed only for a single peak point, interaction signals are easily lost due to minor deviations in gaze. Therefore, in the technical solution of this application, a spatially aligned Gaussian kernel map is further constructed based on the list of risk peaks to create a weighted map that can quantify spatial proximity, i.e., an attention gravity field centered on the key risk source. This approach creates a gravitational field for each critical risk point (such as the center of an oncoming vehicle) that decays Gaussian with distance, with higher-risk objects exhibiting stronger gravitational fields. This fundamentally alters the assumption of equal pixel weights in the original mechanism, successfully constructing an attention priority map centered on critical risk sources. Through this mechanism, the system mathematically imbues the area surrounding the risk source with a kind of magnetism. When a pedestrian's gaze sweeps across these areas, the closer to the center, the higher the likelihood of being judged as having received effective attention, thus simulating the human visual capture characteristics of salient targets.

[0071] More specifically, in a concrete example of this application, the system creates an all-zero floating-point matrix of the same size as the camera image as an initial map. Then, the system iterates through each peak point in the risk peak list, generates a Gaussian kernel centered at that point's coordinates, and superimposes this kernel onto the matrix. For any pixel in the image, its weight value in the spatially aligned Gaussian kernel map is calculated using the following formula:

[0072] in, To spatially align Gaussian kernel maps in The value of the point represents the weight of the pixel in the aggregated Gaussian kernel map constructed from all risk peaks. List of risk peaks For risk peaks, this is an element in the list of risk peaks, containing coordinates. (Pixel coordinates) and weights (Peak risk weight) The Euclidean norm is used to calculate the straight-line distance between two pixels. This is a spatial scale parameter, a hyperparameter that controls the width of the Gaussian kernel. It can adaptively adjust according to the scale of the target in the scene. For example, for large vehicles that are relatively close, a larger setting can be used. To cover the entire front area of ​​the vehicle; for pedestrians or small obstacles at a distance, set a smaller [size / size]. To avoid weight overflow, a weighted summation is performed on the Gaussian kernels generated from all risk peaks, resulting in the final generated... The diagram not only highlights the core location of the high-risk target, but also preserves its radiating influence on the surrounding space, forming a continuous and smooth surface of attention potential.

[0073] In the pedestrian behavior recognition system 100 for a vehicle driving scenario described above, the pedestrian attention score determination subunit 1543 is used to determine the pedestrian attention score based on a spatially aligned Gaussian kernel map, a gaze probability map, and a probability gaze cone object. It is understandable that since the original attention calculation typically uses simple product summation, this method cannot effectively suppress low-quality interaction signals caused by blurred vision or inaccurate predictions (because even with low probability density, the integral can be significant if the area is large), and it also cannot adequately reward highly focused, intentional gaze behaviors. In safety-critical driving scenarios, there is a greater tendency to trust pedestrians who see accurately and correctly. Therefore, in the technical solution of this application, the pedestrian attention score is further determined based on a spatially aligned Gaussian kernel map, a gaze probability map, and a probability gaze cone object. Specifically, an enhanced attention score calculation is performed to couple the pedestrian's gaze information with a newly constructed attention priority map and incorporate consideration of gaze prediction confidence, thereby obtaining a more accurate attention evaluation result. Thus, through the above technical means, this solution achieves a significant optimization of the pedestrian attention score quantification method. The ultimate technical effect and goal is to more accurately distinguish between a pedestrian's genuine interaction intentions and unintentional environmental glances, thereby providing more reliable input for autonomous driving decision-making systems. For example, when a pedestrian stares intently at a vehicle and prepares to cross the road, their gaze probability map is highly concentrated and largely overlaps with the Gaussian kernel at the center of the vehicle. The system calculates a very high score, and the vehicle will slow down and yield in advance. Conversely, if a pedestrian is merely lingering on the roadside with a wandering gaze, although their body may be facing the road, the calculated score is lower due to the high uncertainty of their gaze and the lack of focus on the risk source. In this case, the vehicle can maintain normal driving and avoid unnecessary sudden braking that could affect the safety of following vehicles.

[0074] More specifically, in a particular example of this application, the system uses a pedestrian gaze probability map. Spatial Alignment Gaussian Kernel Map Perform an inner product and multiply by a confidence penalty factor. The specific calculation formula is as follows:

[0075] in, The enhanced attention score is the improved final score. The uncertainty of line-of-sight is represented by the confidence level of the predicted line-of-sight; the higher the value, the greater the uncertainty. The uncertainty penalty coefficient is a hyperparameter used to adjust the intensity of the penalty. For line-of-sight probability maps in The value of the point represents the pixel where the pedestrian's line of sight falls. probability density on This represents the summation over all pixels in the image domain, essentially accumulating the sum for each pixel in the image. Notably, the summation term calculates the overlap between the pedestrian's gaze focus and the risk gravitational field; this term is only large when the gaze focus highly overlaps with the risk peak region, thus addressing the spatial proximity limitation. The confidence penalty factor, on the other hand, suppresses the score for high-uncertainty gaze predictions, addressing the directional consistency limitation. Thus, the final output is an enhanced attention score that reflects both the spatial alignment between the gaze and the risk source and the confidence level of the gaze prediction. If the calculated... If the threshold is exceeded, it indicates that the pedestrian has a high degree of interaction intent and risk awareness, and the system will trigger an early warning or active avoidance mechanism.

[0076] In the pedestrian behavior recognition system 100 under the aforementioned vehicle driving scenario, the intent fusion decoding and behavior prediction output module 160 is used to perform intent fusion decoding and behavior prediction output on the pedestrian attention score list, the full-body posture features in the pedestrian target list, and the vehicle state to obtain the behavior category and intent probability. It is understandable that since a pedestrian's final behavioral decision is the result of multiple factors, a simple attention score only reflects the cognitive state, and a simple limb movement only reflects the physical state. Environmental constraints (such as vehicle speed) determine the feasibility of the behavior. If these three factors are separated and judged independently, logical paradoxes are likely to occur (for example, a pedestrian may be looking at the car but remain still, or lean forward but not look at the car). Therefore, in the technical solution of this application, intent fusion decoding and behavior prediction output are further performed on the pedestrian attention score list, the full-body posture features in the pedestrian target list, and the vehicle state to obtain the behavior category and intent probability. This is used to construct a deep fusion inference engine with multimodal features, simulating the thought process of a human driver comprehensively judging road conditions. In this way, the system can output a dual prediction result that includes both deterministic categories and probabilistic confidence levels. This allows the autonomous driving system to not only know what a pedestrian might do, but also the reliability of this judgment, thereby supporting hierarchical and smooth path planning and speed control strategies.

[0077] More specifically, in a concrete example of this application, the system employs a multilayer perceptron or long short-term memory network as the intent decoder. First, a comprehensive feature vector is constructed by concatenating the scalar attention score calculated in the preceding steps, the high-dimensional action vector extracted from the full-body posture features (including keypoint coordinates, orientation angle, and speed), and the vehicle's current speed and acceleration state data. This vector is then input into the decoder network, undergoing multilayer nonlinear transformations and feature cross-learning to map it into a predefined behavior space. Finally, the Softmax classification layer outputs probability distributions for different behavior categories such as "crossing," "waiting," and "avoiding," selecting the category with the highest probability as the final predicted behavior category. This probability value is then output as the intent probability, allowing the downstream control module to dynamically adjust the vehicle's driving strategy based on the risk level.

[0078] In summary, a pedestrian behavior recognition system for vehicle driving scenarios according to embodiments of this application is described. It first screens high-risk targets based on full-body posture and kinematic features, then performs refined head analysis on these targets to construct a probabilistic gaze cone with uncertainty metrics. To address the challenge of distinguishing between false interactions and genuine crossing intentions based solely on actions, this scheme constructs a dynamic environmental risk saliency map and utilizes a spatially aligned Gaussian kernel mechanism to deeply couple the gaze probability distribution with environmental risk sources. This mechanism, by introducing uncertainty penalties and spatial proximity weighting, amplifies the strong interaction signals of pedestrians precisely focusing on risk sources and effectively suppresses noise interference caused by gaze divergence or unintentional scanning. Finally, the system fuses and decodes the generated high-confidence attention score with posture and vehicle state, achieving accurate prediction of subtle pedestrian intentions and improving the robustness of autonomous driving decisions.

Claims

1. A pedestrian behavior recognition system in a vehicle driving scene, characterized in that, Comprise: a data acquisition module for acquiring camera data and lidar data; a pedestrian global feature extraction module for performing pedestrian global feature extraction on the camera data and the lidar data to obtain a pedestrian target list; an intent preliminary screening and high-risk target locking module for performing intent preliminary screening and high-risk target locking on the pedestrian target list based on a self-vehicle state to obtain a high-risk pedestrian list; a head region refinement and probabilistic gaze cone estimation module for performing head region refinement and probabilistic gaze cone estimation on the high-risk pedestrian list based on the camera data to obtain a pedestrian list with gaze information; an attention focus analysis module for performing attention focus analysis on the pedestrian list with gaze information, the camera data, and the self-vehicle state to obtain a pedestrian attention score list; an intent fusion decoding and behavior prediction output module for performing intent fusion decoding and behavior prediction output on the pedestrian attention score list, whole-body posture features in the pedestrian target list, and the self-vehicle state to obtain a behavior category and an intent probability.

2. The system, as claimed in claim 1, wherein The pedestrian global feature extraction module comprises: a pedestrian detection and positioning unit for performing vision-based pedestrian detection and positioning on the camera data to obtain a pedestrian body bounding box list; a multi-modal data registration and depth information association unit for performing multi-modal data registration and depth information association on the pedestrian body bounding box list and the lidar data based on calibration parameters to obtain a pedestrian depth list; a pedestrian global posture feature extraction unit for performing pedestrian global posture feature extraction based on the pedestrian body bounding box list and the camera data to obtain a whole-body posture feature list; a multi-dimensional information aggregation unit for performing multi-dimensional information aggregation on the whole-body posture feature list, the pedestrian depth list, and the pedestrian body bounding box list to obtain the pedestrian target list.

3. The system, as claimed in claim 2, wherein The intent preliminary screening and high-risk target locking module comprises: a pedestrian kinematics and orientation state analysis unit for performing pedestrian kinematics and orientation state analysis on each pedestrian object in the pedestrian target list based on the self-vehicle state to obtain a pedestrian body orientation angle and a pedestrian transverse relative speed; a collision risk quantitative assessment unit for performing collision risk quantitative assessment based on the pedestrian depth in the pedestrian object and the longitudinal component of the relative speed to obtain a collision time; a high-risk target screening unit for performing high-risk target screening on the pedestrian target list based on the collision time and the pedestrian body orientation angle to obtain the high-risk pedestrian list.

4. The system, as claimed in claim 3, wherein The collision risk quantitative assessment unit comprises: performing collision risk quantitative assessment with the following formula: ； wherein, is the longitudinal distance acquired from the pedestrian depth, is the longitudinal relative speed of the pedestrian, is the time to collision.

5. The system, as claimed in claim 1, wherein The head region refinement and probabilistic gaze cone estimation module comprises: a high-risk pedestrian head positioning and image enhancement unit for performing high-risk pedestrian head positioning and image enhancement on each high-risk pedestrian object in the high-risk pedestrian list based on the camera data to obtain an enhanced head image; a gaze parameter joint estimation unit for performing depth network-based gaze parameter joint estimation on the enhanced head image to obtain a gaze parameter tuple, the gaze parameter tuple comprising a central gaze vector and an uncertainty scalar; The probability line-of-sight cone modeling unit is configured to perform probability line-of-sight cone modeling on the line-of-sight parameter tuple to obtain a probability line-of-sight cone object; The information aggregation unit is configured to perform information aggregation on the probability line-of-sight cone object and the high-risk pedestrian object to obtain a line-of-sight information bearing pedestrian list.

6. The system, as claimed in claim 1, wherein The attention focus analysis module comprises: The scene element perception and semantic information extraction unit is configured to perform scene element perception and semantic information extraction on the camera data to obtain a semantic segmentation map and a dynamic target list; The environmental risk quantification unit is configured to perform environmental risk quantification on the semantic segmentation map, the dynamic target list, and the ego vehicle state to obtain an environmental risk saliency map; The line-of-sight probability map acquisition unit is configured to project each line-of-sight information bearing pedestrian object in the line-of-sight information bearing pedestrian list based on the calibration parameter to obtain a line-of-sight probability map; The pedestrian attention score list generation unit is configured to generate a pedestrian attention score list based on the line-of-sight probability map, the environmental risk saliency map, and the line-of-sight information bearing pedestrian list.

7. The system, as claimed in claim 6, wherein The pedestrian attention score list generation unit comprises: The environmental risk peak extraction and positioning subunit is configured to perform environmental risk peak extraction and positioning on the environmental risk saliency map to obtain a risk peak list; The spatially aligned Gaussian kernel map construction subunit is configured to construct a spatially aligned Gaussian kernel map based on the risk peak list; The pedestrian attention score determination subunit is configured to determine a pedestrian attention score based on the spatially aligned Gaussian kernel map, the line-of-sight probability map, and the probability line-of-sight cone object.

Images