A pedestrian intent reasoning method that integrates scene interaction features and hierarchical temporal modeling

Abstract

This application provides a pedestrian intention reasoning method that integrates scene interaction features and hierarchical temporal modeling, relating to the field of autonomous driving technology, and solves the technical problems of low accuracy and generalization ability in pedestrian intention prediction in existing technologies. The method includes: acquiring pedestrian-vehicle distance data, vehicle speed data, and scene image data; performing semantic segmentation on the scene image data, identifying and labeling the object attributes of each pixel to obtain a segmented image, bounding box coordinates, and pedestrian pose key points; the bounding box coordinates represent the pedestrian's position in the segmented image; extracting scene temporal features from the segmented image; extracting spatiotemporally correlated pedestrian motion intention features based on the bounding box coordinates, pedestrian pose key points, pedestrian-vehicle distance data, and vehicle speed data; and using a hierarchical temporal strategy to fuse the scene temporal features and spatiotemporally correlated pedestrian motion intention features, and predicting the pedestrian's crossing intention. This application is used in the pedestrian intention reasoning process.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a method and apparatus for pedestrian intention reasoning that integrates scene interaction features and hierarchical temporal modeling. Background Technology

[0002] Pedestrian intention reasoning aims to predict future actions by combining observable pedestrian behavior patterns with surrounding traffic environment information. Accurate prediction of pedestrian crossings, waiting, or other behaviors enables autonomous driving systems to achieve more rational path planning and decision-making, thereby improving driving safety and traffic efficiency. Although deep learning-based pedestrian intention reasoning methods have made some progress, existing research mostly relies on single-modal analysis (such as posture or trajectory), making it difficult to effectively characterize the dynamic interactions between pedestrians, vehicles, and the environment. In complex traffic scenarios, pedestrian behavior is influenced by multiple factors such as environmental changes, individual habits, and psychological states, exhibiting greater randomness and uncertainty. Relying solely on rule-based methods often fails to fully capture the complex relationships between pedestrians and the environment, resulting in low accuracy and generalization ability of the reasoning results, thus affecting the safe operation of autonomous vehicles. Therefore, there is an urgent need to propose a pedestrian intention reasoning method that integrates multi-source data to better address these issues. Summary of the Invention

[0003] This application provides a pedestrian intention reasoning method that integrates scene interaction features and hierarchical temporal modeling, which solves the technical problems of low accuracy and generalization ability of pedestrian intention prediction in the prior art.

[0004] To achieve the above objectives, this application adopts the following technical solution:

[0005] Firstly, this paper provides a pedestrian intention reasoning method that integrates scene interaction features and hierarchical temporal modeling. The method includes: acquiring pedestrian-vehicle distance data, vehicle speed data, and scene image data; using DeepLabV3 to perform semantic segmentation on the scene image data, identifying and labeling the object attributes of each pixel to obtain segmented images, bounding box coordinates, and pedestrian pose key points; object attributes include pedestrians, vehicles, buildings, and environmental objects; bounding box coordinates represent the pedestrian's position in the segmented image; extracting scene temporal features from the segmented image using a VGG network model and average pooling operations; based on bounding box coordinates, pedestrian pose key points, pedestrian-vehicle distance data, and vehicle speed data, extracting spatiotemporally correlated pedestrian motion intention features through multi-layered stacked GRU units with an attention mechanism introduced in each GRU unit; and employing a hierarchical temporal strategy to fuse scene temporal features and spatiotemporally correlated pedestrian motion intention features, and predicting the pedestrian's crossing intention.

[0006] In conjunction with the first aspect mentioned above, in one possible implementation, the extraction of scene temporal features from the segmented image includes: removing pedestrians from the segmented image through a masking operation to obtain a scene image; extracting high-level scene temporal features from the scene image based on a VGG network; and converting the high-level scene temporal features into compact high-level scene temporal features through an average pooling algorithm and mapping them to a unified feature space to obtain scene temporal features.

[0007] In conjunction with the first aspect mentioned above, one possible implementation involves compact high-level scene temporal features. Satisfy the following formula:

[0008]

[0009] Where AP() represents the average pooling function, VGG() represents the VGG network model, and Mask() represents the masking operation. This is the segmented image at time t.

[0010] In conjunction with the first aspect mentioned above, in one possible implementation, based on bounding box coordinates, pedestrian posture key points, pedestrian-vehicle distance data, and vehicle speed data, spatiotemporally correlated pedestrian motion intention features are extracted, including: extracting pedestrian position temporal features based on bounding box coordinates; extracting pedestrian key posture temporal features based on multiple pedestrian posture key points; performing feature encoding on pedestrian-vehicle distance data and vehicle speed data respectively to obtain pedestrian-vehicle distance temporal features and vehicle speed temporal features; aligning and fusing pedestrian position temporal features, pedestrian key posture temporal features, pedestrian-vehicle distance temporal features, and vehicle speed temporal features to obtain spatiotemporally correlated pedestrian motion intention features.

[0011] In conjunction with the first aspect mentioned above, in one possible implementation, pedestrian location temporal features Satisfy the following formula:

[0012]

[0013] in, , used to represent the coordinates of the top-left and bottom-right corners of the bounding box. l Indicates the coordinates of the top left corner. r The coordinates of the lower right corner are represented by i, which is a natural number greater than or equal to zero and is used to represent the index of the pedestrian; tm, t-m+1, ..., t represent the observation sequence from frame tm to frame t.

[0014] In conjunction with the first aspect mentioned above, one possible implementation involves employing a hierarchical temporal strategy to fuse scene temporal features and spatiotemporally correlated pedestrian motion intention features, and then predicting pedestrian crossing intentions. This includes: inputting scene temporal features into the first branch of the hierarchical temporal encoder to extract global scene dynamic information features; inputting spatiotemporally correlated pedestrian motion intention features into the second branch of the hierarchical temporal encoder to extract key pedestrian motion intention features; the second branch of the hierarchical temporal encoder consists of multiple stacked GRUs and attention mechanisms, used to extract key pedestrian motion intention features from the spatiotemporally correlated pedestrian motion intention features; performing cross-modal alignment and weighted fusion of global scene dynamic information features and extracted key pedestrian motion intention features within a unified feature space to obtain comprehensive temporal features; inputting the comprehensive temporal representation into a multi-layer fully connected neural network, performing nonlinear mapping and dimensionality reduction layer by layer, and predicting pedestrian crossing intentions.

[0015] In conjunction with the first aspect mentioned above, in one possible implementation, key pedestrian movement intention features Satisfy the following formula:

[0016]

[0017]

[0018]

[0019] in, It is a weight matrix. It is the length of observation. This is the encoder's last hidden state. It is the sum of all attention-weighted hidden states. It is the previous hidden state of the encoder. It is the attention weight vector.

[0020] In conjunction with the first aspect mentioned above, in one possible implementation, after predicting the pedestrian's intention to cross the street, the method further includes optimizing the binary classification task result using a binary cross-entropy loss function.

[0021] In conjunction with the first aspect mentioned above, in one possible implementation, the loss function Satisfy the following formula:

[0022]

[0023] in, For predicting labels, To correspond to the actual labels, n is the total number of categories.

[0024] Secondly, a pedestrian intention reasoning device integrating scene interaction features and hierarchical temporal modeling is provided, comprising a communication unit and a processing unit. The communication unit acquires pedestrian-vehicle distance data, vehicle speed data, and scene image data. The processing unit uses DeepLabV3 to perform semantic segmentation on the scene image data, identifying and labeling the object attributes of each pixel to obtain a segmented image, bounding box coordinates, and pedestrian pose key points. Object attributes include pedestrians, vehicles, buildings, and environmental objects; bounding box coordinates represent the pedestrian's position in the segmented image. Temporal features of the scene in the segmented image are extracted using a VGG network model and average pooling operations. Based on the bounding box coordinates, pedestrian pose key points, pedestrian-vehicle distance data, and vehicle speed data, spatiotemporally correlated pedestrian motion intention features are extracted using multi-layered stacked GRU units with an attention mechanism introduced in each GRU unit. A hierarchical temporal strategy is used to fuse the scene temporal features and spatiotemporally correlated pedestrian motion intention features, and the pedestrian's crossing intention is predicted.

[0025] Thirdly, this application provides an electronic device, including: a processor and a storage medium; the storage medium includes instructions, and the processor is configured to execute the instructions to implement the methods described in the first aspect and any possible implementation thereof. This electronic device may be an electronic device or a chip within an electronic device.

[0026] Fourthly, this application provides a pedestrian intention reasoning system that integrates scene interaction features and hierarchical temporal modeling, including: sensors and electronic devices. The sensors are used to acquire pedestrian-vehicle distance data, vehicle speed data, and scene image data; the electronic devices are used to perform semantic segmentation on the scene image data, identify and label the object attributes of each pixel, and obtain segmented images, bounding box coordinates, and pedestrian posture key points; object attributes include: pedestrians, vehicles, buildings, and environmental objects; bounding box coordinates represent the pedestrian's position in the segmented image; scene temporal features are extracted from the segmented image; based on the bounding box coordinates, pedestrian posture key points, pedestrian-vehicle distance data, and vehicle speed data, spatiotemporally correlated pedestrian motion intention features are extracted; a hierarchical temporal strategy is used to fuse the scene temporal features and spatiotemporally correlated pedestrian motion intention features, and predict the pedestrian's crossing intention result.

[0027] Fifthly, this application provides a computer-readable storage medium storing instructions that, when executed on a pedestrian intention reasoning device that integrates scene interaction features and hierarchical temporal modeling, cause the pedestrian intention reasoning device to perform the methods described in the first aspect and any possible implementation thereof.

[0028] In a sixth aspect, this application provides a computer program product containing instructions that, when run on an electronic device, cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0029] This application provides a pedestrian intention reasoning method that integrates scene interaction features and hierarchical temporal modeling. By fusing multi-source data such as scene images, pedestrian pose key points, bounding boxes, vehicle speed, and pedestrian-vehicle distance, it achieves comprehensive perception of pedestrian behavior and the surrounding environment. Hierarchical temporal modeling captures environmental features and pedestrian dynamic trends. A cross-modal attention mechanism is combined to weight key actions and environmental factors, highlighting important features and improving the accuracy and robustness of pedestrian crossing intention prediction. Finally, a multi-layer fully connected network outputs a probability distribution, providing quantifiable decision-making basis for autonomous driving systems. This method exhibits good generalization ability and adaptability, maintaining stable prediction performance even in complex traffic scenarios, thereby effectively improving driving safety and traffic efficiency. It solves the technical problems of low accuracy and generalization ability in pedestrian intention prediction in existing technologies.

[0030] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0031] Figure 1 A system architecture diagram of a pedestrian intent reasoning system that integrates scene interaction features and hierarchical temporal modeling, provided in an embodiment of this application;

[0032] Figure 2 A flowchart illustrating the pedestrian intent reasoning method that integrates scene interaction features and hierarchical temporal modeling, as provided in the embodiments of this application;

[0033] Figure 3 A flowchart illustrating another pedestrian intent reasoning method that integrates scene interaction features and hierarchical temporal modeling, provided for an embodiment of this application;

[0034] Figure 4A flowchart illustrating another pedestrian intent reasoning method that integrates scene interaction features and hierarchical temporal modeling, provided for an embodiment of this application;

[0035] Figure 5 A flowchart illustrating another pedestrian intent reasoning method that integrates scene interaction features and hierarchical temporal modeling, provided for an embodiment of this application;

[0036] Figure 6 A schematic diagram of the pedestrian intent reasoning device that integrates scene interaction features and hierarchical temporal modeling, provided in an embodiment of this application;

[0037] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0038] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0039] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0040] The pedestrian intent reasoning method provided in this application embodiment can be applied to, for example... Figure 1 In the pedestrian intention reasoning system shown, such as Figure 1 As shown, the system includes: sensor 101 and electronic device 102.

[0041] The system includes a sensor 101 for acquiring pedestrian-vehicle distance data, vehicle speed data, and scene image data; and an electronic device 102 for using DeepLabV3 to perform semantic segmentation on the scene image data, identifying and labeling the object attributes of each pixel to obtain a segmented image, bounding box coordinates, and pedestrian pose key points. The object attributes include pedestrians, vehicles, buildings, and environmental objects. The bounding box coordinates represent the pedestrian's position in the segmented image. Temporal features of the scene in the segmented image are extracted using a VGG network model and average pooling operations. Based on the bounding box coordinates, pedestrian pose key points, pedestrian-vehicle distance data, and vehicle speed data, spatiotemporally correlated pedestrian motion intention features are extracted using multi-layered stacked GRU units with an attention mechanism introduced in each GRU unit. A hierarchical temporal strategy is used to fuse the scene temporal features and spatiotemporally correlated pedestrian motion intention features, and the pedestrian's intention to cross the street is predicted.

[0042] To address the technical problems of low accuracy and generalization ability in pedestrian intention prediction in existing technologies, this application provides a pedestrian intention reasoning method that integrates scene interaction features and hierarchical temporal modeling. The method includes: acquiring pedestrian-vehicle distance data, vehicle speed data, and scene image data; performing semantic segmentation on the scene image data, identifying and labeling the object attributes of each pixel to obtain a segmented image, bounding box coordinates, and pedestrian posture key points; object attributes include: pedestrians, vehicles, buildings, and environmental objects; bounding box coordinates represent the pedestrian's position in the image; extracting scene temporal features from the segmented image; extracting spatiotemporally correlated pedestrian motion intention features based on the bounding box coordinates, pedestrian posture key points, pedestrian-vehicle distance data, and vehicle speed data; and using a hierarchical temporal strategy to fuse the scene temporal features and spatiotemporally correlated pedestrian motion intention features to predict the pedestrian's crossing intention.

[0043] Figure 2 A flowchart illustrating the pedestrian intent reasoning method provided in this application embodiment is shown below. Figure 2 As shown, the method includes:

[0044] S201. Obtain distance data between people and vehicles, vehicle speed data, and scene image data.

[0045] Among them, the pedestrian-vehicle distance data refers to the relative spatial distance between the autonomous vehicle and the surrounding pedestrians, the vehicle speed data refers to the real-time speed of the vehicle, and the scene image data refers to the road and surrounding environment images captured by the vehicle's camera.

[0046] In one possible implementation, a pedestrian intent reasoning device that integrates scene interaction features and hierarchical temporal modeling acquires the aforementioned data through a vehicle-mounted multi-sensor fusion system. The distance between the pedestrian and vehicle is estimated using monocular depth sensing, the vehicle speed is collected by the vehicle control unit, and scene images are continuously acquired by a front-facing camera to form an image sequence.

[0047] It should be noted that the acquired data was collected continuously at multiple time steps so that subsequent time series modeling could capture dynamic change features, and the acquired image sequences were aligned with the corresponding sensor data in terms of timestamps.

[0048] As an example, in this embodiment, the vehicle speed is obtained directly from the vehicle's LiDAR sensor. In the PIE dataset, vehicle speed is expressed in kilometers per hour. (Vehicle and pedestrian...) Distance between This application embodiment uses monocular depth estimation to calculate a distance metric reflecting relative spatial position. The input data includes two parts: a 2D bounding box representing pedestrian location information. (k = 0 to 3 represents the coordinates from the top left corner to the bottom right corner). Another is the monocular depth estimation map, which provides depth information for each pixel relative to the camera, represented as... (i and j are pixel coordinates). First, extract the depth values ​​of all pixels within the 2D bounding box region from the depth estimation map, then calculate the average depth for the bounding box. Its upper left corner is ( , The bottom right corner is represented as ( , ), pixel coordinates within this region Must meet ≤ ≤ and ≤ ≤ After extraction, the depth values ​​of all pixels within the bounding box are averaged to obtain the average depth of the region.

[0049] As an example, in an embodiment of this application, based on the aforementioned average depth, pedestrians and vehicles in time relative distance Satisfy the following formula:

[0050] in, Indicates time Shi Xingren The relative distance between the vehicle and the autonomous vehicle.

[0051] S202. Perform semantic segmentation on the scene image data, identify and label the object attributes of each pixel, and obtain the segmented image, bounding box coordinates, and pedestrian pose key points.

[0052] The object attributes include pedestrians, vehicles, buildings, and environmental objects; the bounding box coordinates represent the pedestrian's position in the image, and the pose keypoints represent the positions of the pedestrian's major joints.

[0053] In one possible implementation, a pedestrian intent reasoning device that integrates scene interaction features and hierarchical temporal modeling performs pixel-by-pixel classification on the input scene image through a pre-trained semantic segmentation network to generate segmented images. At the same time, it uses an object detection algorithm to obtain the bounding box of the pedestrian and extracts the coordinates of the pedestrian key points through a pose estimation network.

[0054] As an example, in this embodiment of the application, the encoder-decoder structure and dilated convolution technique of a pre-trained DeepLabV3 model are used to identify and distinguish different objects in an image, processing the input image. Perform semantic segmentation to generate segmented images. In this process, each pixel is assigned a category label to identify the object category it belongs to (such as pedestrians, vehicles, buildings, etc.).

[0055] As an example, in an embodiment of this application, image segmentation... Satisfy the following formula:

[0056]

[0057] S203. Extract scene temporal features from the segmented image.

[0058] In one possible implementation, pedestrians are removed from the segmented image using a masking operation to obtain a scene image. High-level scene temporal features are extracted from the scene image using a VGG network. Then, average pooling is used to transform the high-level scene temporal features into compact high-level scene temporal features, which are then mapped to a unified feature space to obtain the scene temporal features.

[0059] It should be noted that the extracted features need to be normalized to ensure the consistency and stability of features at different time steps.

[0060] S204. Based on bounding box coordinates, pedestrian posture key points, pedestrian-vehicle distance data, and vehicle speed data, extract spatiotemporally correlated pedestrian motion intention features.

[0061] In one possible implementation, pedestrian position temporal features are extracted based on bounding box coordinates; pedestrian key posture temporal features are extracted based on multiple pedestrian posture key points; pedestrian-vehicle distance data and vehicle speed data are respectively feature-encoded to obtain pedestrian-vehicle distance temporal features and vehicle speed temporal features; pedestrian position temporal features, pedestrian key posture temporal features, pedestrian-vehicle distance temporal features and vehicle speed temporal features are aligned and fused to obtain spatiotemporally correlated pedestrian motion intention features.

[0062] It should be noted that data from different modalities need to be time-aligned before modeling to ensure that GRU can effectively capture temporal dependencies. This step can comprehensively characterize pedestrian behavior dynamics and interactions with vehicles, providing a detailed and reliable feature foundation for intent prediction.

[0063] S205. A hierarchical temporal strategy is adopted to fuse scene temporal features and spatiotemporally related pedestrian movement intention features, and predict pedestrian crossing intention results.

[0064] In one possible implementation, the scene temporal features are input into the first branch of the hierarchical temporal encoder to extract global scene dynamic information features; the spatiotemporally correlated pedestrian motion intention features are input into the second branch of the hierarchical temporal encoder to extract key pedestrian motion intention features; the second branch of the hierarchical temporal encoder consists of multiple stacked GRUs and attention mechanisms, used to extract key pedestrian motion intention features of spatiotemporally correlated pedestrian motion intention features; cross-modal alignment and weighted fusion of global scene dynamic information features and extracted key pedestrian motion intention features are performed in a unified feature space to obtain comprehensive temporal features; the comprehensive temporal representation is input into a multi-layer fully connected neural network, and nonlinear mapping and dimensionality reduction are performed layer by layer to predict pedestrian crossing intention results.

[0065] In one possible implementation, after S205, the pedestrian intent reasoning method provided in this application embodiment further includes: optimizing the binary classification task results using a binary cross-entropy loss function.

[0066] As an example, in an embodiment of this application, the loss function Satisfy the following formula:

[0067]

[0068] in, For predicting labels, To correspond to the actual labels, n is the total number of categories.

[0069] In this embodiment, by acquiring pedestrian-vehicle distance data, vehicle speed data, and scene image data, the dynamic relationship between pedestrians and autonomous vehicles, as well as the surrounding environment, can be comprehensively reflected, providing multi-dimensional basic data support for subsequent temporal modeling. By performing semantic segmentation and object detection on the scene images, not only is accurate identification and posture information extraction of pedestrians, vehicles, and background objects achieved, but high-quality input is also provided for motion feature modeling. Furthermore, by extracting scene temporal features from the segmented images, the dynamic evolution information of the environment over time is captured, thereby improving the understanding of the global traffic scene. Combining multi-source data such as bounding box coordinates, pedestrian posture key points, pedestrian-vehicle distance, and vehicle speed, spatiotemporally correlated pedestrian motion intention features are extracted, enabling the model to characterize pedestrian movement patterns and interactions with vehicles at the spatiotemporal level, ensuring the comprehensiveness and accuracy of feature modeling. Finally, a hierarchical temporal strategy is adopted to perform cross-modal alignment and weighted fusion of scene temporal features and motion intention features, so as to achieve deep integration of complex multi-source information, thereby effectively improving the accuracy, robustness and generalization ability of pedestrian crossing intention prediction, and solving the technical problem of low accuracy and generalization ability of pedestrian intention prediction in existing technologies.

[0070] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, the above S203 can be implemented through the following S301, S302 and S303, which are explained in detail below:

[0071] S301. Remove pedestrians from the segmented image using a mask operation to obtain the scene image.

[0072] Masking refers to masking or zeroing out specific categories of pixel regions in a segmented image to remove pedestrian targets and retain only the background and other environmental information.

[0073] In one possible implementation, a pedestrian intent reasoning device that integrates scene interaction features and hierarchical temporal modeling first receives the segmentation result output by a semantic segmentation network, which assigns a category label to each pixel. The device generates a binary mask based on the category labels, where pixels labeled "pedestrian" have a value of 1, and all other pixels have a value of 0. Next, this mask is multiplied pixel-by-pixel with the original segmented image, setting all pedestrian pixels to zero, thereby removing pedestrian-related information and obtaining a scene image that does not contain pedestrians.

[0074] It should be noted that this operation can avoid interference caused by pedestrian features in scene modeling, highlight the dynamic changes of environmental and traffic elements, and thus ensure that the scene features extracted later are purer and more reliable.

[0075] S302. Extract high-level scene temporal features from scene images based on VGG network.

[0076] Among them, the VGG network is a classic deep convolutional neural network structure, which consists of multiple layers of convolutional and pooling layers stacked together, and can extract multi-scale semantic features of images layer by layer.

[0077] In one possible implementation, a pedestrian intent reasoning device that integrates scene interaction features with hierarchical temporal modeling takes a scene image (with pedestrians removed) as input and passes it sequentially through the convolutional and pooling layers of a VGG network. Low-level features such as edges and corners are extracted in the early layers, while semantic features such as road morphology, building layout, and vehicle distribution are captured at deeper levels. Finally, a high-level semantic representation of the scene is obtained in the fully connected layer. To maintain temporal consistency, scene images from different time steps are input into the same VGG model, forming a high-level scene feature sequence arranged chronologically.

[0078] It should be noted that this method can effectively preserve the global contextual semantic information of the scene, enabling the model to perceive dynamic changes in the background environment.

[0079] S303. Using the average pooling algorithm, high-level scene temporal features are transformed into compact high-level scene temporal features and mapped to a unified feature space to obtain scene temporal features.

[0080] Among them, the average pooling algorithm is a dimensionality reduction method that achieves information compression and feature compaction by taking the average value of the feature matrix in the spatial dimension.

[0081] In one possible implementation, a pedestrian intent reasoning device that integrates scene interaction features with hierarchical temporal modeling inputs high-dimensional features from the VGG network into an average pooling layer. A mean operation is performed on the spatial dimension of each feature channel, transforming it into a fixed-length feature vector. Subsequently, the device projects these compact feature vectors onto a unified feature space through a linear mapping network, ensuring their dimensionality remains consistent with other modalities. Ultimately, this yields scene temporal features capable of cross-modal fusion.

[0082] It should be noted that using average pooling not only reduces feature dimensionality and redundancy and improves computational efficiency, but also enhances the stability and generalization ability of the model in the process of time series modeling.

[0083] As an example, in an embodiment of this application, compact high-level scene temporal features Satisfy the following formula:

[0084]

[0085] Where AP() represents the average pooling function, VGG() represents the VGG network model, and Mask() represents the masking operation. The image is a segmented image at time t.

[0086] As an example, in the embodiments of this application, scene temporal features Satisfy the following formula:

[0087]

[0088] in, For the weight vector, For paranoia, For compact, high-level scene temporal features.

[0089] This application employs a method of removing pedestrian subjects from segmented images to ensure the purity of scene features and highlight the dynamic changes in the background environment. Multi-level semantic information is extracted using a VGG network, enabling a comprehensive characterization of road structure, vehicle distribution, and environmental semantics in traffic scenes, thereby enhancing the global perception capability of complex traffic backgrounds. Furthermore, the average pooling algorithm compacts high-dimensional features and maps them to a unified feature space, reducing redundant features and computational overhead, improving alignment and stability during cross-modal feature fusion, and enhancing the efficiency and effectiveness of temporal scene modeling. This provides a more robust environmental semantic foundation for subsequent fusion of pedestrian motion intention features and prediction of pedestrian crossing intentions.

[0090] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 4 As shown, the above S204 can be implemented by the following S401 to S404, which are explained in detail below:

[0091] S401. Extract pedestrian location temporal features based on bounding box coordinates.

[0092] Here, the bounding box coordinates refer to the coordinates of the upper left and lower right corners of the rectangle obtained by the object detection algorithm for locating pedestrians in the image, which are used to represent the spatial position of pedestrians in the image.

[0093] In one possible implementation, a pedestrian intent reasoning device that integrates scene interaction features and hierarchical temporal modeling extracts the pedestrian's bounding box coordinate sequence at each time step, using it as input features to construct the pedestrian's positional change trajectory over time. By modeling the changes in the four coordinate points of the bounding box, the pedestrian's movement trend and spatial distribution pattern over time can be obtained.

[0094] It should be noted that location temporal features can provide basic geometric information for pedestrian behavior modeling, which helps to capture the displacement characteristics and potential intentions of pedestrians in the road environment.

[0095] As an example, in the embodiments of this application, pedestrian location temporal features Satisfy the following formula:

[0096]

[0097] in, , used to represent the coordinates of the top-left and bottom-right corners of the bounding box. l Indicates the coordinates of the top left corner. r The coordinates of the lower right corner are represented by i, which is a natural number greater than or equal to zero and is used to represent the index of the pedestrian; tm, t-m+1, ..., t represent the observation sequence from frame tm to frame t.

[0098] S402. Extract the temporal features of key pedestrian postures based on multiple pedestrian posture key points.

[0099] Among them, pedestrian pose key points refer to the main joints detected by the human pose estimation algorithm (such as the main joint positions of the head, shoulders, elbows, wrists, hips, knees and ankles), which are used to characterize the pedestrian's pose information.

[0100] In one possible implementation, a pedestrian intent reasoning device that integrates scene interaction features and hierarchical temporal modeling extracts multiple key point coordinates of pedestrians in each frame of an image through a pre-trained human pose estimation network, forming a key pose sequence in the temporal dimension. This temporal feature reflects pedestrian body movement patterns, such as lifting a foot, turning, or pausing, helping to determine whether a pedestrian intends to cross the street. Pose features and location features complement each other, enabling a more refined characterization of pedestrian dynamic behavior.

[0101] As an example, in this embodiment of the application, key point features of pedestrian pose are extracted using HRNet, and each key point is represented by its two-dimensional coordinates. =( , The keypoint features of the pedestrian posture are represented as follows: After obtaining the keypoint features, these features are normalized by applying min-max normalization to map the keypoint coordinates to the range [0,1], resulting in normalized keypoint coordinates. By integrating keypoint data from multiple time steps, the dynamic information of the pedestrian is further captured. For each pedestrian sample at time step t, a new posture feature vector is constructed. This vector integrates the current time step and m Keypoint data from one preceding time step. Keypoint data for each time step is represented as a 36-dimensional vector. Each dimension corresponds to the x and y coordinates of 18 key points.

[0102] As an example, in an embodiment of this application, the normalized keypoint coordinates ( Satisfy the following formula:

[0103]

[0104]

[0105] in, These represent the minimum and maximum values ​​of the keypoint coordinates in all pedestrian samples, respectively.

[0106] As an example, in the embodiments of this application, key points Satisfy the following formula:

[0107]

[0108] in, Let be the normalized coordinates of the i-th key point at time t.

[0109] S403. Perform feature encoding on the distance data between people and vehicles and the speed data of the vehicle to obtain the time-series features of the distance between people and vehicles and the time-series features of the speed of the vehicle.

[0110] In one possible implementation, a pedestrian intent reasoning device that integrates scene interaction features and hierarchical temporal modeling processes continuously collected sequences of pedestrian-vehicle distances and vehicle speeds through standardization and time-series encoding, converting them into numerically stable and dimensionally uniform temporal feature representations. This encoding process can utilize fully connected layers or embedding layers to map the input sequences, adapting them to subsequent temporal modeling modules.

[0111] It should be noted that the dynamic changes of these external traffic elements can affect pedestrians' crossing decisions. Therefore, introducing such features into pedestrian intention reasoning can improve the accuracy and robustness of predictions.

[0112] As an example, in an embodiment of this application, vehicle speed Satisfy the following formula:

[0113]

[0114] Where t, t-m+1, and tm represent the observation sequence from frame tm to frame t, i.e., different time points.

[0115] It should be noted that the acquisition of the vehicle speed timing characteristics has been described in detail in S201, and will not be repeated here.

[0116] S404. Align and fuse the temporal features of pedestrian position, the temporal features of pedestrian key posture, the temporal features of pedestrian-vehicle distance, and the temporal features of vehicle speed to obtain spatiotemporally correlated pedestrian motion intention features.

[0117] Feature alignment refers to aligning feature sequences of different modalities and time steps through interpolation, padding, or timestamps to ensure their consistency in the time dimension, so as to facilitate unified modeling.

[0118] In one possible implementation, the pedestrian intention reasoning device that integrates scene interaction features and hierarchical temporal modeling first aligns the pedestrian position temporal features, pedestrian key posture temporal features, pedestrian-vehicle distance temporal features, and vehicle speed temporal features in time, and then splices them together to extract a deep representation of the pedestrian's movement intention in the spatiotemporal dimension.

[0119] It should be noted that the spatiotemporal correlation motion intention features obtained through this step not only include the pedestrian's own motion behavior information, but also integrate the dynamic relationship with vehicles and traffic environment, providing a more refined and reliable feature foundation for subsequent hierarchical temporal modeling and pedestrian intention prediction.

[0120] In this embodiment, by extracting temporal features of pedestrian positions, the motion trajectory and position changes of pedestrians in image sequences are captured, providing spatial location information for subsequent behavior prediction; by extracting temporal features of key pedestrian postures, the action states of pedestrians at different time steps, such as acceleration, deceleration, or turning, are characterized, thereby enhancing the understanding of motion intentions; by performing feature encoding on pedestrian-vehicle distance data and vehicle speed data, dynamic constraints of environmental interaction and vehicle motion are introduced, so that pedestrian intention reasoning is not only based on visual information, but also considers external factors of traffic scenarios; by aligning and fusing the above multi-source temporal features, a deep combination of spatial and temporal features can be achieved, improving the completeness and accuracy of pedestrian motion intention features, and more comprehensively modeling the spatiotemporal relationship between pedestrians, vehicles, and the environment, thereby improving the accuracy of pedestrian crossing intention prediction.

[0121] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 5 As shown, the above S205 can be implemented by the following S501 to S504, which are explained in detail below:

[0122] S501. Input the scene temporal features into the first branch of the hierarchical temporal encoder to extract global scene dynamic information features.

[0123] In one possible implementation, the pedestrian intention reasoning device that integrates scene interaction features and hierarchical temporal modeling takes the scene temporal features obtained in S203 as input and inputs them into the first branch of the hierarchical temporal encoder. It uses multi-layer recursive units to encode the sequence and combines an attention mechanism to focus on key time slices, thereby extracting global scene dynamic information features that reflect the dynamic changes of the global traffic scene.

[0124] It should be noted that the first branch of the hierarchical temporal encoder is mainly aimed at temporal modeling of large-scale traffic scenarios, which can avoid the problem of incomplete features caused by single temporal scale modeling and ensure a comprehensive understanding of the global dynamics of the scene.

[0125] S502. Input the spatiotemporally correlated pedestrian motion intention features into the second branch of the hierarchical temporal encoder to extract key pedestrian motion intention features.

[0126] The second branch of the hierarchical temporal encoder consists of multiple stacked GRUs and attention mechanisms, used to extract key pedestrian motion intention features that are spatiotemporally correlated.

[0127] In one possible implementation, the pedestrian intention reasoning device that integrates scene interaction features and hierarchical temporal modeling takes the spatiotemporally correlated pedestrian motion intention features obtained in S404 as input and inputs them into the second branch of the hierarchical temporal encoder. This branch consists of multiple stacked GRU units, and an attention mechanism is introduced in each layer, so that it can automatically allocate weights for different key moments and extract the key pedestrian motion intention features that contribute the most to the pedestrian motion intention.

[0128] It should be noted that the second branch of the hierarchical temporal encoder can avoid interference from redundant features, highlight motion information that is strongly correlated with pedestrian crossing behavior, and make the prediction results more accurate and interpretable.

[0129] As an example, in the embodiments of this application, key pedestrian movement intention features Satisfy the following formula:

[0130]

[0131]

[0132]

[0133] in, It is a weight matrix. It is the length of observation. This is the encoder's last hidden state. It is the sum of all attention-weighted hidden states. It is the previous hidden state of the encoder. It is the attention weight vector.

[0134] S503. In a unified feature space, cross-modal alignment and weighted fusion are performed on the dynamic information features of the global scene and the key pedestrian motion intention features to obtain comprehensive temporal features.

[0135] In one possible implementation, the pedestrian intention reasoning device that integrates scene interaction features and hierarchical temporal modeling first maps the global scene dynamic information features and the extracted key pedestrian movement intention features into a unified feature space of the same dimension. Then, it adopts an attention-based weighted fusion strategy to dynamically allocate the importance of the two types of features, thereby generating comprehensive temporal features.

[0136] It should be noted that this step can make full use of the complementarity between global scene information and local pedestrian motion intention information, and avoid the influence of the bias of single modality features on the prediction results.

[0137] S504. Input the comprehensive temporal representation into a multilayer fully connected neural network, perform nonlinear mapping and dimensionality reduction layer by layer, and predict the pedestrian's intention to cross the street.

[0138] In one possible implementation, a pedestrian intention reasoning device that integrates scene interaction features and hierarchical temporal modeling takes comprehensive temporal features as input and passes them sequentially through a multi-layer fully connected neural network. Each layer uses a non-linear activation function to perform non-linear mapping on the features, while reducing the feature dimension layer by layer to reduce redundant information and enhance classification and discrimination capabilities. Finally, it outputs the predicted pedestrian crossing intention results.

[0139] It should be noted that this step can reduce computational complexity while ensuring feature representation capabilities, and improve the accuracy and stability of predictions through deep mapping.

[0140] This embodiment of the application extracts global scene dynamic information features by inputting the scene temporal features into the first branch of the hierarchical temporal encoder, which can comprehensively capture the dynamic changes of the road environment and provide global context support for pedestrian intention prediction. The spatiotemporally correlated pedestrian motion intention features are input into the second branch of the hierarchical temporal encoder, and key pedestrian motion intention features are extracted by combining stacked GRU and attention mechanisms. This helps to highlight key pedestrian actions and behavioral patterns, improving the accuracy of pedestrian crossing intention recognition. Cross-modal alignment and weighted fusion of global scene dynamic information features and key pedestrian motion intention features are performed within a unified feature space, fully utilizing the complementarity of scene and pedestrian actions, reducing the impact of single-modal information bias, and improving the expressive power of the fused features. The comprehensive temporal representation is input into a multi-layer fully connected neural network for nonlinear mapping and dimensionality reduction, which not only compresses feature dimensions and reduces redundant information but also enhances the stability and accuracy of the final pedestrian crossing intention prediction. Based on the above steps, this embodiment of the application can achieve deep spatiotemporal fusion of scene and pedestrian motion intention, effectively improving the accuracy, reliability, and generalization ability of pedestrian crossing intention prediction, thereby providing safer and more intelligent decision support for autonomous driving systems.

[0141] The above primarily describes the solutions of the embodiments of this application from the perspective of device implementation. It is understood that each device, such as a pedestrian intent reasoning device that integrates scene interaction features and hierarchical temporal modeling, includes at least one of the hardware structures and software modules corresponding to each function in order to achieve the above-mentioned functions. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0142] This application embodiment can divide the pedestrian intent reasoning device that integrates scene interaction features and hierarchical temporal modeling into functional units based on the above method example. For example, each function can be divided into separate functional units, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0143] When using integrated units, Figure 6 A possible structural schematic diagram of the pedestrian intention reasoning device that integrates scene interaction features and hierarchical temporal modeling (referred to as pedestrian intention reasoning device 60 that integrates scene interaction features and hierarchical temporal modeling) involved in the above embodiments is shown. The pedestrian intention reasoning device 60 that integrates scene interaction features and hierarchical temporal modeling includes a processing unit 601 and a communication unit 602, and may also include a storage unit 603. Figure 6 The structural diagram shown can be used to illustrate the structure of the pedestrian intention reasoning device that integrates scene interaction features and hierarchical temporal modeling involved in the above embodiments.

[0144] when Figure 6 The schematic diagram shown illustrates the structure of the pedestrian intention reasoning device that integrates scene interaction features and hierarchical temporal modeling involved in the above embodiments. The processing unit 601 is used to control and manage the actions of the pedestrian intention reasoning device that integrates scene interaction features and hierarchical temporal modeling. The communication unit 602 is used for the pedestrian intention reasoning device that integrates scene interaction features and hierarchical temporal modeling to communicate with other devices. The storage unit 603 is used to store the program code and data of the pedestrian intention reasoning device that integrates scene interaction features and hierarchical temporal modeling.

[0145] For example, the communication unit 602 is used to acquire distance data between people and vehicles, vehicle speed data, and scene image data.

[0146] The processing unit 601 is used to perform semantic segmentation on scene image data, identify and label the object attributes of each pixel, and obtain segmented images, bounding box coordinates, and pedestrian pose key points. The object attributes include pedestrians, vehicles, buildings, and environmental objects. The bounding box coordinates are the positions of pedestrians in the segmented images. The processing unit extracts scene temporal features from the segmented images. Based on the bounding box coordinates, pedestrian pose key points, pedestrian-vehicle distance data, and vehicle speed data, the processing unit extracts spatiotemporally related pedestrian motion intention features. The processing unit employs a hierarchical temporal strategy to fuse the scene temporal features and spatiotemporally related pedestrian motion intention features, and predicts the pedestrian's intention to cross the street.

[0147] In one possible implementation, the processing unit 601 is further configured to extract scene temporal features from the segmented image, including: removing pedestrians from the segmented image through a masking operation to obtain a scene image; extracting high-level scene temporal features from the scene image based on a VGG network; and converting the high-level scene temporal features into compact high-level scene temporal features through an average pooling algorithm and mapping them to a unified feature space to obtain scene temporal features.

[0148] In one possible implementation, compact high-level scene temporal features Satisfy the following formula:

[0149]

[0150] Where AP() represents the average pooling function, VGG() represents the VGG network model, and Mask() represents the masking operation. The image is a segmented image at time t.

[0151] In one possible implementation, the processing unit 601 is further configured to extract spatiotemporally correlated pedestrian motion intention features based on bounding box coordinates, pedestrian posture key points, pedestrian-vehicle distance data, and vehicle speed data. This includes: extracting pedestrian position temporal features based on bounding box coordinates; extracting pedestrian key posture temporal features based on multiple pedestrian posture key points; performing feature encoding on the pedestrian-vehicle distance data and vehicle speed data respectively to obtain pedestrian-vehicle distance temporal features and vehicle speed temporal features; and aligning and fusing the pedestrian position temporal features, pedestrian key posture temporal features, pedestrian-vehicle distance temporal features, and vehicle speed temporal features to obtain spatiotemporally correlated pedestrian motion intention features.

[0152] In one possible implementation, pedestrian location temporal features Satisfy the following formula:

[0153]

[0154] in, , used to represent the coordinates of the top-left and bottom-right corners of the bounding box. l Indicates the coordinates of the top left corner. r The coordinates of the lower right corner are represented by i, which is a natural number greater than or equal to zero and is used to represent the index of the pedestrian; tm, t-m+1, ..., t represent the observation sequence from frame tm to frame t.

[0155] In one possible implementation, the processing unit 601 is further configured to perform feature fusion on scene temporal features and spatiotemporally related pedestrian motion intention features using a hierarchical temporal strategy, and predict pedestrian crossing intention results, including: inputting scene temporal features into the first branch of the hierarchical temporal encoder to extract global scene dynamic information features; inputting spatiotemporally related pedestrian motion intention features into the second branch of the hierarchical temporal encoder to extract key pedestrian motion intention features; the second branch of the hierarchical temporal encoder consists of multiple stacked GRUs and attention mechanisms, used to extract key pedestrian motion intention features of spatiotemporally related pedestrian motion intention features; performing cross-modal alignment and weighted fusion on global scene dynamic information features and extracted key pedestrian motion intention features in a unified feature space to obtain comprehensive temporal features; inputting the comprehensive temporal representation into a multi-layer fully connected neural network, performing nonlinear mapping and dimensionality reduction layer by layer, and predicting pedestrian crossing intention results.

[0156] In one possible implementation, key pedestrian movement intention features Satisfy the following formula:

[0157]

[0158]

[0159]

[0160] in, It is a weight matrix. It is the length of observation. This is the encoder's last hidden state. It is the sum of all attention-weighted hidden states. It is the previous hidden state of the encoder. It is the attention weight vector.

[0161] In one possible implementation, the processing unit 601 is further configured to, after predicting the pedestrian's intention to cross the street, optimize the binary classification task result using a binary cross-entropy loss function.

[0162] In one possible implementation, the loss function Satisfy the following formula:

[0163]

[0164] in, For predicting labels, To correspond to the actual labels, n is the total number of categories.

[0165] The processing unit 601 can be a processor or a controller, and the communication unit 602 can be a communication interface, transceiver, transceiver circuit, transceiver device, etc. The term "communication interface" is a general term and may include one or more interfaces. The storage unit 603 can be a memory. When the pedestrian intent reasoning device 60, which integrates scene interaction features and hierarchical temporal modeling, is a chip, the processing unit 601 can be a processor or a controller, and the communication unit 602 can be an input interface and / or an output interface, pins, or circuits, etc. The storage unit 603 can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip (e.g., read-only memory (ROM), random access memory (RAM, etc.).

[0166] The communication unit can also be called a transceiver unit. The antenna and control circuit with transceiver functions in the pedestrian intention reasoning device 60 that integrates scene interaction features and hierarchical temporal modeling can be considered as the communication unit 602 of the pedestrian intention reasoning device 60, and the processor with processing functions can be considered as the processing unit 601 of the pedestrian intention reasoning device 60. Optionally, the device in the communication unit 602 that implements the receiving function can be considered as a communication unit, which is used to execute the receiving steps in the embodiments of this application. The communication unit can be a receiver, a receiver circuit, etc. The device in the communication unit 602 that implements the transmitting function can be considered as a transmitting unit, which is used to execute the transmitting steps in the embodiments of this application. The transmitting unit can be a transmitter, a transmitter, a transmitting circuit, etc.

[0167] Figure 6If the integrated units in the process are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. Storage media for storing computer software products include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0168] Figure 6 The units in the process can also be called modules; for example, a processing unit can be called a processing module.

[0169] This application also provides a hardware structure diagram of an electronic device (denoted as electronic device 70), see [link to diagram]. Figure 7 The electronic device 70 includes a processor 701, and optionally, a memory 702 connected to the processor 701.

[0170] In the first possible implementation, see Figure 7 The electronic device 70 also includes a transceiver 703. The processor 701, memory 702, and transceiver 703 are connected via a bus. The transceiver 703 is used to communicate with other devices or communication networks. Optionally, the transceiver 703 may include a transmitter and a receiver. The device in the transceiver 703 that implements the receiving function can be considered as a receiver, which is used to perform the receiving steps in the embodiments of this application. The device in the transceiver 703 that implements the transmitting function can be considered as a transmitter, which is used to perform the transmitting steps in the embodiments of this application.

[0171] Based on the first possible implementation method Figure 7 The structural diagram shown can be used to illustrate the structure of the electronic device involved in the above embodiments.

[0172] in, Figure 7 This can also be illustrated by a system chip in an electronic device. In this case, the actions performed by the aforementioned electronic device can be implemented by this system chip; the specific actions performed can be found above and will not be repeated here.

[0173] In implementation, each step of the method provided in this embodiment can be completed by integrated logic circuits in the processor or by instructions in software form. The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0174] The processor in this application may include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., which are various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor may be a separate semiconductor chip or integrated with other circuits into a single semiconductor chip. For example, it may be integrated with other circuits (such as encoding / decoding circuits, hardware acceleration circuits, or various bus and interface circuits) to form a SoC (System-on-a-Chip), or it may be integrated as a built-in processor within an ASIC. The ASIC with the integrated processor may be packaged separately or together with other circuits. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), PLDs (programmable logic devices), or logic circuits that implement dedicated logic operations.

[0175] The memory in the embodiments of this application may include at least one of the following types: read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; or electrically erasable programmable-only memory (EEPROM). In some scenarios, the memory may also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto.

[0176] This application also provides a computer-readable storage medium including instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0177] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0178] This application also provides a chip including a processor and an interface circuit. The interface circuit is coupled to the processor. The processor is used to run computer programs or instructions to implement the above-described method. The interface circuit is used to communicate with other modules outside the chip.

[0179] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0180] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0181] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A method for pedestrian intent reasoning that integrates scene interaction features and hierarchical temporal modeling, characterized in that, include: Acquire data on the distance between people and vehicles, vehicle speed data, and scene image data; DeepLabV3 was used to perform semantic segmentation on the scene image data, identify and label the object attributes of each pixel, and obtain the segmented image, bounding box coordinates, and pedestrian pose key points. The object attributes include: pedestrians, vehicles, buildings, and environmental objects; the bounding box coordinates represent the position of the pedestrian in the segmented image. Scene temporal features in the segmented image are extracted using a VGG network model and average pooling operation. Based on the bounding box coordinates, the pedestrian posture key points, the pedestrian-vehicle distance data, and the vehicle speed data, spatiotemporally correlated pedestrian motion intention features are extracted through multi-layer stacked GRU units and an attention mechanism is introduced in each GRU unit. A hierarchical temporal strategy is adopted to fuse the temporal features of the scene and the spatiotemporally related pedestrian movement intention features, and to predict the pedestrian crossing intention results. The step of employing a hierarchical temporal strategy to fuse the temporal features of the scene and the spatiotemporally correlated pedestrian movement intention features, and predicting the pedestrian crossing intention, includes: The scene temporal features are input into the first branch of the hierarchical temporal encoder to extract global scene dynamic information features; The spatiotemporally correlated pedestrian motion intention features are input into the second branch of the hierarchical temporal encoder to extract key pedestrian motion intention features; the second branch of the hierarchical temporal encoder consists of multiple stacked GRUs and attention mechanisms, which are used to extract key pedestrian motion intention features of spatiotemporally correlated pedestrian motion intention features. Within a unified feature space, the global scene dynamic information features and the extracted key pedestrian motion intent features are cross-modal aligned and weighted and fused to obtain comprehensive temporal features; The comprehensive temporal features are input into a multi-layer fully connected neural network, and nonlinear mapping and dimensionality reduction are performed layer by layer to predict pedestrian crossing intentions.

2. The method according to claim 1, characterized in that, The extraction of scene temporal features from the segmented image includes: Pedestrians are removed from the segmented image using a masking operation to obtain the scene image; High-level scene temporal features are extracted from the scene images based on the VGG network; The high-level scene temporal features are transformed into compact high-level scene temporal features by using the average pooling algorithm, and then mapped to a unified feature space to obtain scene temporal features.

3. The method according to claim 2, characterized in that, The temporal features of the compact high-level scene Satisfy the following formula: Where AP() represents the average pooling function, VGG() represents the VGG network model, and Mask() represents the masking operation. This is the segmented image at time t.

4. The method according to claim 1, characterized in that, The step of extracting spatiotemporally correlated pedestrian motion intent features based on the bounding box coordinates, pedestrian posture key points, pedestrian-vehicle distance data, and vehicle speed data includes: Based on bounding box coordinates, extract temporal features of pedestrian location; Based on multiple pedestrian posture key points, extract the temporal features of key pedestrian postures; The distance data between people and vehicles and the speed data of vehicles are respectively encoded to obtain the temporal features of the distance between people and vehicles and the temporal features of the speed of vehicles. By aligning and fusing the pedestrian location time-series features, the pedestrian key posture time-series features, the pedestrian-vehicle distance time-series features, and the vehicle speed time-series features, a spatiotemporally correlated pedestrian motion intention feature is obtained.

5. The method according to claim 4, characterized in that, The pedestrian location time sequence features Satisfy the following formula: in, , used to represent the coordinates of the top-left and bottom-right corners of the bounding box. l Indicates the coordinates of the top left corner. r Indicates the coordinates of the bottom right corner. i is a natural number greater than or equal to zero, used to represent the index of the pedestrian; tm, t-m+1, ..., t represent the observation sequence from frame tm to frame t.

6. The method according to claim 1, characterized in that, The key pedestrian movement intention features Satisfy the following formula: in, It is a weight matrix. It is the length of observation. This is the encoder's last hidden state. It is the sum of all attention-weighted hidden states. It is the previous hidden state of the encoder. It is the attention weight vector.

7. The method according to claim 1, characterized in that, After predicting the pedestrian's intention to cross the street, the method further includes optimizing the binary classification task results using a binary cross-entropy loss function.

8. The method according to claim 7, characterized in that, The loss function Satisfy the following formula: in, For predicting labels, To correspond to the actual labels, n is the total number of categories.

9. The method according to claim 1, characterized in that, The scene temporal features Satisfy the following formula: in, For the weight vector, For paranoia, For compact, high-level scene temporal features.

Images