A method for predicting a pedestrian crossing intention

Abstract

The application provides a pedestrian crossing intention prediction method. The method first acquires a road scene video stream and extracts a target pedestrian detection box, and then performs asymmetric expansion processing with a lower side bias. On the basis of retaining the main body information of the pedestrian, environmental features such as curbs, zebra crossings and carriageway boundaries are introduced to construct a time sequence image segment. At the same time, in order to improve the prediction performance by using cross-modal prior knowledge, the application constructs a prediction network based on a visual language pre-training model, uses the visual encoder of the visual language pre-training model to extract the visual semantic features of the image segment, and uses the text encoder to extract the text semantic features of the preset intention description. Finally, the two types of features are mapped to a shared semantic space and the semantic similarity is calculated to output the crossing intention probability of the target pedestrian. The method combines spatial environmental context and large model prior knowledge to improve the accuracy of intention prediction.

Description

Technical Field

[0001] This application relates to the field of visual processing, and in particular to a method for predicting pedestrians' intentions to cross the street. Background Technology

[0002] In the fields of intelligent transportation and autonomous driving perception, pedestrian crossing intention prediction is generally regarded as a spatiotemporal understanding task focused on short-term future behavior. Its goal is to determine, based on a continuous sequence of observations prior to the current moment, whether a target pedestrian is likely to enter the roadway in the short term. Unlike ordinary pedestrian detection tasks, this task not only needs to identify the presence of pedestrians but also requires a comprehensive judgment based on pedestrian movement trends, body posture, spatial location, and the surrounding road environment.

[0003] Existing technical solutions similar to this application typically construct an overall processing chain around "localization, continuous observation, feature extraction, temporal modeling, and intent classification." The general process is as follows: First, a target detector is used to locate pedestrians in video frames, and a sequence of bounding boxes for the same pedestrian is formed using a multi-target tracking algorithm. Then, based on this sequence, a local image of the pedestrian is cropped and preprocessed, including scaling and normalization. On this basis, key points, optical flow, semantic segmentation maps, depth information, or trajectory features are further extracted. Finally, convolutional neural networks, visual Transformers, graph neural networks, GRUs, LSTMs, temporal attention networks, or Transformer encoders are used to jointly model the spatial features of a single frame and the dynamic relationships across frames, outputting the probability of whether the pedestrian has the intent to cross the street. However, this approach suffers from several drawbacks: cropping closely to the human body results in the loss of road context; traditional binary classification heads lack semantic correspondence; and full-parameter fine-tuning is costly and easily perturbs pre-training capabilities. Summary of the Invention

[0004] Therefore, it is necessary to provide a method for predicting pedestrian crossing intentions that can accurately predict the aforementioned technical problems.

[0005] One method for predicting pedestrian crossing intentions includes: S1: Acquire the video stream and perform target tracking on the video stream to obtain a bounding box sequence; S2: Asymmetric expansion and cropping of the bounding box sequence yields a time-series image segment; S3: Construct a prediction network based on a visual language pre-trained model, which includes a text encoder and a visual encoder; S4: Input the temporal image fragment into the visual encoder to wait for visual representation, input text prompts into the text encoder to obtain positive / negative text semantic prototypes; calculate the similarity between the visual representation and the positive / negative text semantic prototypes and normalize it to obtain the street crossing probability.

[0006] Furthermore, in step S1, specifically: The video stream is:

[0007] Using object detection and multi-object tracking algorithms, target pedestrians are obtained. In length of The sequence of bounding boxes within the historical observation window:

[0008] The single-frame bounding box is represented by the center point coordinates as follows:

[0009] In the formula, and They represent the first The x and y coordinates of the center point of the bounding box of the target pedestrian in the frame. and These represent the width and height of the bounding box, respectively.

[0010] To address the issue of missing frames in real-world road scenes caused by occlusion, decreased detection confidence, or short-term tracking interruptions, a fault tolerance threshold is set. When the number of consecutive missing frames does not exceed When linear interpolation is used to complete the bounding box trajectory, when the number of consecutive missing frames exceeds [a certain threshold], [the remaining frame count is determined by the missing frame count]. When the current trajectory segment construction is terminated, the trajectory deviation that may be introduced by long-term interpolation is reduced.

[0011] Further, in step S2, specifically: In the horizontal direction, information about the pedestrian's left and right surroundings is preserved. In the vertical direction, a downward-biased outward expansion strategy is adopted to preserve environmental cues related to crossing judgment, such as the area underfoot, curb, zebra crossing, and carriageway boundary. Let the horizontal expansion coefficient be... The vertical outward expansion coefficient is The vertical downward expansion coefficient is And satisfy:

[0012] Based on the original bounding box The expanded target area boundary is denoted as:

[0013] Its geometric parameter update rule is as follows:

[0014]

[0015]

[0016]

[0017] After horizontal expansion, the x-coordinate of the target center remains unchanged, i.e. After expanding vertically, because the expansion ratio on the lower side is greater than the expansion ratio on the upper side, the center point of the target area changes from... Move down to Based on the extended bounding box In the original image frame Crop a portion of the image; when the cropped area exceeds the original image boundary, perform zero-padding or boundary padding on the out-of-bounds portion; after cropping, uniformly scale the image to the target resolution. The images are then normalized to their pixels. Finally, the frames are stacked in chronological order to obtain a continuous image fragment containing the target pedestrian and the surrounding road environment.

[0019] Furthermore, in step S3, the process of constructing text semantic features by the visual encoder includes: Construct a set of natural language prompts representing two types of intent states; the set of positive prompts is denoted as... , used to describe behavioral states with a tendency to cross the street; the set of negative cues is denoted as This is used to describe a behavioral state in which the person does not show a tendency to cross the street. The sets of positive and negative cues are first combined with learnable contextual cues; Subsequently, the text encoder in the visual language pre-trained model was invoked. Extract text features. The text encoder parameters remain frozen, and only the learnable context cue parameters are updated. For any cue... Its normalized text features are:

[0020] The text features in the positive feedback set are averaged and normalized to obtain the positive category semantic prototype:

[0021] Similarly, by averaging and normalizing the text features in the negative cue set, we obtain the semantic prototype of the negative category:

[0023] Furthermore, learnable contextual cues specifically include: The learnable context hints are of length [length missing]. A continuous vector sequence:

[0024] in, Represents the text word embedding dimension; for any human-generated text prompt First, it is segmented and embedded, and then the learnable context vector is concatenated before the text embedding sequence to form an adaptive cue sequence.

[0025] in, Indicates a text prompt The word segmentation results This represents a text word embedding map.

[0026] Furthermore, in step S3, the visual encoder performs the visual feature extraction process as follows: The visual encoder in the visual language pre-trained model is selected as the visual backbone, denoted as . For any single frame image in the time-series image segment generated in step S2. The visual encoder outputs frame-level visual features:

[0027] Insert a LoRA low-rank adaptation branch in the projection layer of the Transformer self-attention module of the visual encoder.

[0028] Furthermore, in step S3, specifically: Input feature matrix Simultaneously, it enters two branches, one of which is the frozen pre-trained projection layer with weights of 1. The other branch is the LoRA low-rank adaptation branch, which is determined by the matrix. and Composition, in which:

[0029] The merged projection output is:

[0030] in, Indicates the output of the projection layer. For frozen pre-trained weights, and For trainable low-rank fitting parameters, This is the scaling factor. The rank of the low-rank decomposition; When the low-rank adaptation branches of LoRA are applied to the self-attention modules , and When projecting a layer, it can be represented as:

[0031]

[0032]

[0033] After completing frame-level visual feature extraction, the visual features within the historical window are further aggregated temporally; The frame features are concatenated in chronological order as follows:

[0034] Overlay time position encoding Then, input the temporal aggregation network:

[0035] in, The implementation employs recurrent neural networks, temporal convolutional networks, or Transformer encoders based on self-attention mechanisms; temporal aggregation is used to capture continuous changes in pedestrian movements; and temporal features are obtained. .

[0036] Furthermore, in step S3, specifically: During LoRA low-rank adaptation training initialization, the matrix Random initialization is used for the matrix. Initialize as a zero matrix, such that the initial phase satisfies This ensures that the initial output of the model is consistent with the pre-trained model; subsequently, the model gradually learns adaptation parameters related to the target traffic scenario during training; the LoRA low-rank adaptation branch can be loaded into... , , One or more branches in the projection layer, the specific configuration of which can be selected according to the computing power and data scale of the deployment platform.

[0037] Further, in step S4, specifically: temporal features are obtained through a visual encoder based on continuous image segments containing the target pedestrian and its surrounding road environment. Then through the projection matrix The pooled temporal features are mapped to a shared semantic space consistent with the text semantic prototype using a projection matrix:

[0038] in, Indicates the target pedestrian The normalized temporal visual representation; this representation simultaneously includes visual appearance, road context, and historical action change information; Further, in step S4, specifically: Calculate similarity in the shared semantic space and output the probability of pedestrians' intention to cross the street accordingly: The actual behavior results within the future observation interval are used as surrogate supervision signals. Let the future observation interval be:

[0039] If the target pedestrian If any action occurs within this interval, such as crossing the curb, entering the roadway, or beginning to cross the road, then the current time will be... The corresponding sample is labeled as positive; if the above behavior does not occur, it is labeled as negative.

[0040] In the prediction phase, normalized visual representations are calculated. Similarity with positive and negative text semantic prototypes:

[0041]

[0042] in, This represents the similarity between the visual representation of the target pedestrian and the semantic prototype of the pedestrian preparing to cross the street. This indicates the similarity between it and the semantic prototype of "not ready to cross the street".

[0043] Introducing temperature parameters Then, the target pedestrian is obtained through normalized mapping. Probability of predicting pedestrian crossing intentions:

[0044] when Not less than the preset threshold At that time, determine the target pedestrian They must have the intention to cross the street at the current moment; otherwise, they are deemed not to have shown a clear intention to cross the street. During the training phase, a parameter optimization strategy of freezing the backbone and updating locally is adopted; most of the pre-trained parameters of the visual encoder and text encoder are kept frozen, and only the set of trainable parameters is updated. Including LoRA low-rank adaptation parameters Time series aggregation module parameters Cross-modal projection layer parameters and learnable contextual hint parameters ; For batch size The training data is used with binary cross-entropy loss with class compensation weights, combined with parameter regularization constraints. The total loss function is expressed as:

[0045] in, and These are the weights for the positive and negative classes, respectively. , and The regularization coefficient is . This indicates the level at which the LoRA branch is loaded.

[0046] This application constructs a bottom-biased, asymmetrically expanded region based on the pedestrian detection bounding box, ensuring that the input image simultaneously includes environmental cues such as the pedestrian subject, the area under their feet, the curb, the zebra crossing, and the roadway boundary. Furthermore, this application calculates the similarity between visual and textual data in a shared semantic space. This approach results in a clearer correspondence between the prediction results and the semantic description of the text. Attached Figure Description

[0047] Figure 1 Here is a flowchart of a method for predicting pedestrian crossing intentions in one embodiment; Figure 2 This is an architecture diagram of a pedestrian crossing intention prediction method in one embodiment; Figure 3 This is a schematic diagram illustrating the asymmetric expansion of the target area and the method for retaining road environment information in one embodiment. Figure 4 This is a schematic diagram of the frozen backbone and low-rank adapter structure at the visual encoding end in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] Example 1: This embodiment discloses a method for predicting pedestrians' intention to cross the street, including: S1: Acquire the video stream and perform target tracking on the video stream to obtain a bounding box sequence; S2: Asymmetric expansion and cropping of the bounding box sequence yields a time-series image segment; S3: Construct a prediction network based on a visual language pre-trained model, which includes a text encoder and a visual encoder; S4: Input the temporal image fragment into the visual encoder to wait for visual representation, input text prompts into the text encoder to obtain positive / negative text semantic prototypes; calculate the similarity between the visual representation and the positive / negative text semantic prototypes and normalize it to obtain the street crossing probability.

[0050] This embodiment constructs an asymmetric, outward-expanding region with a lower bias based on the pedestrian detection bounding box, ensuring that the input image simultaneously includes environmental cues such as the pedestrian subject, the area under their feet, the curb, the zebra crossing, and the roadway boundary. Furthermore, this application transforms the prediction of crossing intent from a traditional closed classification problem into a matching problem between visual features and textual semantic prototypes. The similarity between visual and textual features is calculated in a shared semantic space. This approach results in a clearer correspondence between the prediction results and the textual semantic description.

[0051] Example 2: This embodiment further discloses information based on Embodiment 1: Further, in step S1, the video stream is:

[0052] Using object detection and multi-object tracking algorithms, target pedestrians are obtained. In length of The sequence of bounding boxes within the historical observation window:

[0053] The single-frame bounding box is represented by the center point coordinates as follows:

[0054] In the formula, and They represent the first The x and y coordinates of the center point of the bounding box of the target pedestrian in the frame. and These represent the width and height of the bounding box, respectively.

[0055] To address the issue of missing frames in real-world road scenes caused by occlusion, decreased detection confidence, or short-term tracking interruptions, a fault tolerance threshold is set. When the number of consecutive missing frames does not exceed When linear interpolation is used to complete the bounding box trajectory, when the number of consecutive missing frames exceeds [a certain threshold], [the remaining frame count is determined by the missing frame count]. When the current trajectory segment construction is terminated, the trajectory deviation that may be introduced by long-term interpolation is reduced.

[0056] Further, in step S2, specifically: In the horizontal direction, information about the pedestrian's left and right surroundings is preserved. In the vertical direction, a downward-biased outward expansion strategy is adopted to preserve environmental cues related to crossing judgment, such as the area underfoot, curb, zebra crossing, and carriageway boundary. Let the horizontal expansion coefficient be... The vertical outward expansion coefficient is The vertical downward expansion coefficient is And satisfy:

[0057] Based on the original bounding box The expanded target area boundary is denoted as:

[0058] Its geometric parameter update rule is as follows:

[0059]

[0060]

[0061]

[0062] After horizontal expansion, the x-coordinate of the target center remains unchanged, i.e. After expanding vertically, because the expansion ratio on the lower side is greater than the expansion ratio on the upper side, the center point of the target area changes from... Move down to Based on the extended bounding box In the original image frame Crop a portion of the image; when the cropped area exceeds the original image boundary, perform zero-padding or boundary padding on the out-of-bounds portion; after cropping, uniformly scale the image to the target resolution. The images are then normalized to their pixels. Finally, the frames are stacked in chronological order to obtain a continuous image fragment containing the target pedestrian and the surrounding road environment.

[0064] Further, in step S3, the text encoder specifically includes: Construct a set of natural language prompts representing two types of intent states; the set of positive prompts is denoted as... This is used to describe behavioral states with a tendency to cross the street: the set of negative cues is denoted as... Used to describe a behavioral state that does not show a tendency to cross the street. The sets of positive and negative cues are first combined with learnable contextual cues; Subsequently, the text encoder in the visual language pre-trained model was invoked. Extract text features. The text encoder parameters remain frozen, and only the learnable context cue parameters are updated. For any cue... Its normalized text features are:

[0065] The text features in the positive feedback set are averaged and normalized to obtain the positive category semantic prototype:

[0066] Similarly, by averaging and normalizing the text features in the negative cue set, we obtain the semantic prototype of the negative category:

[0068] Furthermore, learnable contextual cues specifically include: The learnable context hints are of length [length missing]. A continuous vector sequence:

[0069] in, Represents the text word embedding dimension; for any human-generated text prompt First, it is segmented and embedded, and then the learnable context vector is concatenated before the text embedding sequence to form an adaptive cue sequence.

[0070] in, Indicates a text prompt The word segmentation results This represents a text word embedding map.

[0071] Further, in step S3, the visual encoder specifically comprises: The visual encoder in the visual language pre-trained model is selected as the visual backbone, denoted as . For the single-frame image generated in step 1 The visual encoder outputs frame-level visual features:

[0072] Insert a LoRA low-rank adaptation branch in the projection layer of the Transformer self-attention module of the visual encoder.

[0073] Furthermore, in step S3, specifically: Input feature matrix Simultaneously, it enters two branches, one of which is the frozen pre-trained projection layer with weights of 1. The other branch is the LoRA low-rank adaptation branch, which is determined by the matrix. and Composition, in which:

[0074] The merged projection output is:

[0075] in, Indicates the output of the projection layer. For frozen pre-trained weights, and For trainable low-rank fitting parameters, This is the scaling factor. The rank of the low-rank decomposition; When the low-rank adaptation branches of LoRA are applied to the self-attention modules , and When projecting a layer, it can be represented as:

[0076]

[0077]

[0078] After completing frame-level visual feature extraction, the visual features within the historical window are further aggregated temporally; The frame features are concatenated in chronological order as follows:

[0079] Overlay time position encoding Then, input the temporal aggregation network:

[0080] in, The implementation employs recurrent neural networks, temporal convolutional networks, or Transformer encoders based on self-attention mechanisms; temporal aggregation is used to capture continuous changes in pedestrian movements; and temporal features are obtained. .

[0081] Furthermore, in step S3, specifically: During LoRA low-rank adaptation training initialization, the matrix Random initialization is used for the matrix. Initialize as a zero matrix, such that the initial phase satisfies This ensures that the initial output of the model is consistent with the pre-trained model; subsequently, the model gradually learns adaptation parameters related to the target traffic scenario during training; the LoRA low-rank adaptation branch can be loaded into... , , One or more branches in the projection layer, the specific configuration of which can be selected according to the computing power and data scale of the deployment platform.

[0082] Further, in step S4, specifically: temporal features are obtained through a visual encoder based on continuous image segments containing the target pedestrian and its surrounding road environment. Then through the projection matrix Time series features To a shared semantic space consistent with the text encoder output.

[0083] in, Indicates the target pedestrian The normalized temporal visual representation; this representation simultaneously includes visual appearance, road context, and historical action change information; Further, in step S4, specifically:

[0084] Calculate similarity in the shared semantic space and output the probability of pedestrians' intention to cross the street accordingly: The actual behavior results within the future observation interval are used as surrogate supervision signals. Let the future observation interval be:

[0085] If the target pedestrian If any action occurs within this interval, such as crossing the curb, entering the roadway, or beginning to cross the road, then the current time will be... The corresponding sample is labeled as positive; if the above behavior does not occur, it is labeled as negative.

[0086] In the prediction phase, normalized visual representations are calculated. Similarity with positive and negative text semantic prototypes:

[0087]

[0088] in, This represents the similarity between the visual representation of the target pedestrian and the semantic prototype of the pedestrian preparing to cross the street. This indicates the similarity between it and the semantic prototype of "not ready to cross the street".

[0089] Introducing temperature parameters Then, the target pedestrian is obtained through normalized mapping. Probability of predicting pedestrian crossing intentions:

[0090] when Not less than the preset threshold At that time, determine the target pedestrian They must have the intention to cross the street at the current moment; otherwise, they are deemed not to have shown a clear intention to cross the street. During the training phase, a parameter optimization strategy of freezing the backbone and updating locally is adopted; most of the pre-trained parameters of the visual encoder and text encoder are kept frozen, and only the set of trainable parameters is updated. Including LoRA low-rank adaptation parameters Time series aggregation module parameters Cross-modal projection layer parameters and learnable contextual hint parameters ; For batch size The training data is used with binary cross-entropy loss with class compensation weights, combined with parameter regularization constraints. The total loss function is expressed as:

[0091] in, and These are the weights for the positive and negative classes, respectively. , and The regularization coefficient is . This indicates the level at which the LoRA branch is loaded.

[0092] Example 3: This embodiment provides a method for predicting pedestrians' intention to cross the street, such as... Figure 2 As shown, the method consists of four main modules: an environment-aware visual sequence construction module, an intent semantic text construction and encoding module, a visual feature efficient fine-tuning and temporal aggregation module, and a cross-modal semantic alignment and intent prediction module. Figure 2The overall processing chain of this application is illustrated as follows: The upper left part corresponds to the environment-aware visual sequence construction module, which starts from the local view of the pedestrian, preserves road environment information through asymmetric expansion, and generates a continuous temporal tensor; the upper right part corresponds to the intent semantic text construction and encoding module, which constructs positive and negative prompt sets respectively, and forms text semantic prototypes through learnable context prompts; the lower left part corresponds to the visual feature efficient fine-tuning and temporal aggregation module, which inserts LoRA low-rank adaptation parameters on the basis of freezing the original parameters of the visual backbone, and performs temporal aggregation on historical frame features; the lower right part corresponds to the cross-modal semantic alignment and intent prediction module, which outputs the pedestrian crossing intent probability by calculating the similarity between the visual representation and the positive and negative text semantic prototypes.

[0093] On the visual side, continuous image segments containing the target pedestrian and its surrounding road environment are input into the visual encoding branch of the visual language model, and LoRA is used for efficient parameter adaptation. On the text side, behavioral states such as "preparing to cross the street" and "not prepared to cross the street" are converted into natural language prompts, and semantic prototypes are obtained through a text encoder. Finally, similarity matching between visual features and text prototypes is performed in a shared semantic space to obtain the pedestrian's intention to cross the street prediction results.

[0094] Detailed technical solution: Step 1: Construction of Environment-Aware Visual Sequences This step corresponds to the attached document. Figure 2 The upper left part, and by Figure 3 This section further explains the specific method of asymmetric expansion of the target area. Let the input video stream be:

[0095] The system first uses object detection and multi-object tracking algorithms to acquire the target pedestrian. In length of The sequence of bounding boxes within the historical observation window:

[0096] The single-frame bounding box is represented by the center point coordinates as follows:

[0097] In the formula, and They represent the first The x and y coordinates of the center point of the bounding box of the target pedestrian in the frame. and These represent the width and height of the bounding box, respectively.

[0098] To address the issue of missing frames in real-world road scenarios caused by occlusion, decreased detection confidence, or brief tracking interruptions, the system sets a fault tolerance threshold. When the number of consecutive missing frames does not exceed When linear interpolation is used to complete the bounding box trajectory, when the number of consecutive missing frames exceeds [a certain threshold], [the remaining frame count is determined by the missing frame count]. When the current trajectory segment construction is terminated, the trajectory deviation that may be introduced by long-term interpolation is reduced. After obtaining a continuous bounding box sequence, the system performs asymmetric spatial expansion. Figure 3 The process is illustrated: the left side shows the original pedestrian detection box, the middle side shows the horizontal expansion, and the right side shows the vertical expansion. Unlike traditional methods that closely follow the human body, this application retains information about the pedestrian's left and right surroundings in the horizontal direction, and adopts a downward-biased expansion strategy in the vertical direction to preserve environmental cues related to crossing the street, such as the area underfoot, curbs, zebra crossings, and roadway boundaries.

[0099] Let the horizontal expansion coefficient be... The vertical outward expansion coefficient is The vertical downward expansion coefficient is And satisfy:

[0100] Based on the original bounding box The expanded target area boundary is denoted as:

[0101] Its geometric parameter update rule is as follows:

[0102]

[0103]

[0104]

[0105] Depend on Figure 3 It can be seen that the x-coordinate of the target center remains unchanged after horizontal expansion, that is... After expanding vertically, because the expansion ratio on the lower side is greater than the expansion ratio on the upper side, the center point of the target area changes from... Move down to This design allows the clipping region to include not only the pedestrian but also the area under the pedestrian's feet and the road in front, which helps the model determine whether the pedestrian is approaching the roadway boundary or zebra crossing area. Subsequently, the system is based on the expanded bounding box. In the original image frame Crop a portion of the image. When the cropped area exceeds the boundaries of the original image, zero-padding or boundary padding is applied to the out-of-bounds portion. After cropping, the image is uniformly scaled to the target resolution. The images are then processed and pixel normalized. Finally, the frames are stacked in chronological order to obtain the target pedestrian image. Temporal visual tensor:

[0106] This temporal visual tensor preserves the relative relationship between the target pedestrian and the road environment in space, and preserves the continuous changes in the target pedestrian's actions in time, providing input for subsequent visual encoding and temporal modeling.

[0107] Step 2: Construction and Encoding of Intent Semantic Text This step corresponds to Figure 2 The upper right section. Traditional supervised learning methods typically model pedestrian crossing intention prediction as a closed binary classification problem, making it difficult to explicitly utilize behavioral semantics such as "preparing to cross the street," "waiting," and "walking along the road." This approach transforms intention prediction into a matching problem between visual content and textual semantic descriptions, leveraging the semantic associations formed during the pre-training phase of the visual language model.

[0108] First, the system constructs a set of natural language prompts representing two types of intent states. The set of positive prompts is denoted as... This is used to describe a behavioral state of having a tendency to cross the street, for example: (1) "A pedestrian preparing to cross the street." (2) "A pedestrian standing near the curb and facing the road." (3) "A person moving toward the crosswalk and likely to enter the roadway." The set of negative hints is denoted as This is used to describe a behavioral state where there is no tendency to cross the street, for example: (1) "A pedestrian waiting on the sidewalk and not crossing." (2) "A pedestrian walking parallel to the road without entering it." (3) "A person standing away from the curb and remaining in a safearea." Figure 2The upper right section shows that the positive and negative cue sets are not directly input into the text encoder, but are first combined with learnable contextual cues. Let the learnable contextual cues be of length... A continuous vector sequence:

[0109] in, This represents the text word embedding dimension. For any artificial text prompt... The system first performs word segmentation and embedding mapping, then concatenates the learnable context vectors before the text embedding sequence to form an adaptive cue sequence.

[0110] in, Indicates a text prompt The word segmentation results This represents a text word embedding mapping. The learnable context vectors do not correspond to specific natural language terms, but rather participate in the update as continuous cue parameters during training, thereby enabling the text semantic prototype to better adapt to the target road scenario.

[0111] Subsequently, the system invokes the text encoder in the visual language pre-trained model. Extract text features. The text encoder parameters remain frozen, and only the learnable context cue parameters are updated. For any cue... Its normalized text features are:

[0112] The text features in the positive feedback set are averaged and normalized to obtain the positive category semantic prototype:

[0113] Similarly, by averaging and normalizing the text features in the negative cue set, we obtain the semantic prototype of the negative category:

[0114] During the reasoning phase, if the set of textual cues and learnable contextual cues are already fixed, then and It can be pre-computed and cached to reduce online inference overhead.

[0115] Step 3: Efficient fine-tuning and temporal aggregation of visual features This step corresponds to Figure 2 The lower left part, and by Figure 4 This further explains the frozen backbone and low-rank adaptation structure at the visual encoding end.

[0116] The system selects the visual encoder in the visual language pre-trained model as the visual backbone, denoted as... For the single-frame image generated in step 1 The visual encoder outputs frame-level visual features:

[0117] In order to adapt to the target road scene while maintaining the general representation capability of the base model, this application does not perform full parameter fine-tuning of the visual encoder, but instead inserts a LoRA low-rank adaptation branch in the projection layer of the Transformer self-attention module. Figure 4 This structure is shown: Input feature matrix Simultaneously, it enters two paths: one is the frozen pre-trained projection layer with weights of... The other branch is a trainable low-rank adaptation branch, consisting of a matrix. and Composition, in which:

[0118] The merged projection output is:

[0119] in, Indicates the output of the projection layer. For frozen pre-trained weights, and For trainable low-rank fitting parameters, This is the scaling factor. It is the rank of the low-rank decomposition.

[0120] When LoRA branches act on the self-attention modules , and When projecting a layer, it can be represented as:

[0121]

[0122]

[0123] During training initialization, the matrix Random initialization is used for the matrix. Initialize as a zero matrix, such that the initial phase satisfies This ensures that the initial output of the model is consistent with the pre-trained model. Subsequently, the model gradually learns adaptation parameters relevant to the target traffic scenario during training. The LoRA branch can be loaded into... , , One or more branches in the projection layer, the specific configuration of which can be selected according to the computing power and data scale of the deployment platform.

[0124] After completing frame-level visual feature extraction, the system further performs temporal aggregation of visual features within the historical window. The frame features are concatenated in chronological order as follows:

[0125] Overlay time position encoding Then, input the temporal aggregation network:

[0126] in, This can be implemented using recurrent neural networks, temporal convolutional networks, or Transformer encoders based on self-attention mechanisms. Temporal aggregation is used to capture continuous changes in pedestrian movements, such as dynamic patterns like pausing, starting, approaching the curb, turning onto the road, or walking parallel to the road.

[0127] Subsequently, the temporal characteristics were analyzed. Pooling is performed, and the projection matrix is ​​used. Mapped to a shared semantic space consistent with the text semantic prototype:

[0128] in, Indicates the target pedestrian The normalized temporal visual representation is obtained. This representation simultaneously includes visual appearance, road context, and historical action change information, providing a foundation for subsequent cross-modal semantic matching.

[0129] Step 4: Cross-modal semantic alignment and intent prediction This step corresponds to the attached document. Figure 2 Bottom right section. The system calculates the similarity between visual representations and textual semantic prototypes in a shared semantic space, and outputs the probability of pedestrians' intention to cross the street accordingly.

[0130] Since the true psychological intentions of pedestrians are difficult to observe directly, this application uses the actual behavioral results within a future observation interval as a proxy monitoring signal. Let the future observation interval be:

[0131] If the target pedestrian If any action occurs within this interval, such as crossing the curb, entering the roadway, or beginning to cross the road, then the current time will be... The corresponding sample is labeled as positive; if the above behavior does not occur, it is labeled as negative.

[0132] During the prediction phase, the system does not directly output categories using traditional fully connected classification heads, but instead calculates normalized visual representations. Similarity with positive and negative text semantic prototypes:

[0133]

[0134] in, This indicates the similarity between the visual representation of the target pedestrian and the semantic prototype of "preparing to cross the street". This indicates the similarity between it and the semantic prototype of "not prepared to cross the street". Figure 2 The lower right section shows the similarity calculation and probability output process.

[0135] Introducing temperature parameters Then, the system obtains the target pedestrian through normalized mapping. Probability of predicting pedestrian crossing intentions:

[0136] when Not less than the preset threshold At that time, the system identifies the target pedestrian. They must have the intention to cross the street at the current moment; otherwise, they are deemed not to have shown a clear intention to cross the street.

[0137] During the training phase, a parameter optimization strategy of "frozen backbone, updated locally" is adopted. Most of the pre-trained parameters of the visual encoder and text encoder are kept frozen, and only the set of trainable parameters is updated. Including LoRA low-rank adaptation parameters Time series aggregation module parameters Cross-modal projection layer parameters and learnable contextual hint parameters .

[0138] For batch size The training data is used with binary cross-entropy loss with class compensation weights, combined with parameter regularization constraints. The total loss function is expressed as:

[0139] in, and These are the weights for the positive and negative classes, respectively. , and The regularization coefficient is . This indicates the level at which the LoRA branch is loaded.

[0140] Through the above design, this embodiment preserves the road environment context at the visual input level, introduces text prompt prototypes at the semantic modeling level, adopts LoRA low-rank parameter updates at the model adaptation level, and completes the pedestrian crossing intention judgment through cross-modal similarity at the prediction level, thus forming a relatively complete pedestrian crossing intention prediction scheme.

[0141] This embodiment does not employ a cropping method that only closely follows the human body. Instead, it constructs an asymmetrically expanded region with a heavier lower side based on the pedestrian detection bounding box, so that the input image simultaneously includes environmental cues such as the pedestrian subject, the area under their feet, the curb, the zebra crossing, and the roadway boundary. This design is consistent with... Figure 2 Correspondingly, this better meets the needs of pedestrians in judging the spatial relationship of the road when crossing the street.

[0142] Parameter description: : No. Frame video image.

[0143] Target pedestrian ID.

[0144] : The current moment.

[0145] : Length of historical observation window.

[0146] : Length of the future observation interval.

[0147] Target pedestrian In the The original bounding box of the frame.

[0148] Target pedestrian In the The outer bounding box of the frame.

[0149] , : Coordinates of the center point of the original bounding box.

[0150] , : Coordinates of the center point of the outer bounding box.

[0151] , : Original bounding box width and height.

[0152] , : Width and height of the outer bounding box.

[0153] : Horizontal expansion coefficient.

[0154] : Vertical outward expansion coefficient.

[0155] : Vertical downward expansion coefficient.

[0156] Target pedestrian Temporal visual tensor.

[0157] : Positive hint set.

[0158] : Negative hint set.

[0159] : Positive category text semantic prototype.

[0160] : Negative category text semantic prototype.

[0161] Visual encoder.

[0162] Text encoder.

[0163] : Input feature matrix at the visual encoding end.

[0164] : Frozen pre-trained projected weights.

[0165] , LoRA low-rank adaptation parameter matrix.

[0166] LoRA is the low-rank decomposition rank.

[0167] LoRA is the scaling factor.

[0168] : Projection result after fusing the frozen main branch output and the LoRA branch output.

[0169] Target pedestrian Normalized temporal visual representation.

[0170] Similarity score between visual representation and positive text semantic prototype.

[0171] Similarity score between visual representation and negative text semantic prototype.

[0172] Temperature parameter.

[0173] Target pedestrian The probability of predicting the intention to cross the street.

[0174] Threshold for determining the intention to cross the street.

[0175] This embodiment transforms the prediction of crossing intent from a traditional closed classification problem into a matching problem between visual features and textual semantic prototypes. This is achieved by constructing a set of positive prompts. and negative cue set Furthermore, it introduces learnable contextual cues, enabling the model to leverage behavioral semantic knowledge from visual language pre-trained models.

[0176] This embodiment, based on freezing the backbone parameters of the visual language model, inserts a LoRA low-rank adaptation branch into the visual Transformer projection layer, updating only a small number of task-related parameters. This design is consistent with... Figure 4 Correspondingly, this can reduce the cost of cross-scenario fine-tuning and minimize the disturbance to the general semantic capabilities of the basic model.

[0177] This embodiment does not directly use the traditional binary classification fully connected head as the final discriminator, but instead computes the temporal visual representation in a shared semantic space. With textual semantic prototypes , The similarity between them is calculated and output after temperature normalization. This method enables a clearer correspondence between the prediction results and the semantic description of the text.

[0178] This embodiment uses the LoRA low-rank adaptation method, which performs local updates based on the frozen backbone parameters of the visual language model. This reduces the scale of parameters that need to be trained when deploying in new scenarios, thereby reducing the cost of model fine-tuning and maintenance.

[0179] This embodiment mainly relies on video image sequences, target detection and tracking results, and text semantic prompts to complete intent prediction. It does not require reliance on multiple auxiliary modules such as human keypoints, optical flow, depth estimation, or semantic segmentation, which helps to simplify the overall system chain.

[0180] This embodiment preserves the spatial relationship between pedestrians and the curb, zebra crossing and roadway boundaries during the input stage, so that the model not only focuses on pedestrian posture, but can also make judgments based on road location clues.

[0181] This embodiment utilizes the general semantic representation capabilities of visual language models, combined with learnable text prompts and LoRA fine-tuning, to enable the model to have a better adaptability under different intersections, shooting angles, lighting conditions, and weather conditions.

[0182] This embodiment is based on historical observation windows. By performing temporal aggregation of continuous visual sequences within a single frame, dynamic change information such as pedestrian pauses, starts, approaches the curb, and turns onto the road can be utilized to reduce misjudgments that may result from static judgments of a single frame.

[0183] This embodiment predicts the probability of crossing the street by using the similarity between visual representation and textual semantic prototypes such as "preparing to cross the street" and "not prepared to cross the street". This makes it easy to have a clear correspondence between the output probability and the specific behavioral semantics, which facilitates subsequent system analysis and manual review.

[0184] Example 4: This embodiment provides a feasible implementation method and does not limit the scope of protection of this application.

[0185] The method of this application is deployed on an intelligent vehicle equipped with a forward-looking camera and an onboard computing unit. The system continuously acquires road video frames from the forward-looking camera and predicts the crossing intentions of pedestrians within a certain range in front of the vehicle.

[0186] (1) Data access and pedestrian trajectory formation The system receives the video stream from the forward-facing camera at a fixed frame rate and records the current time as... For video frames Perform pedestrian detection and use a multi-object tracking algorithm to associate the same pedestrian target in adjacent frames to obtain the target pedestrian. Trajectory sequence within the history window:

[0187] In one implementation, the historical observation window length can be set to... When a target pedestrian is missed due to brief occlusion, if the number of consecutive missing frames does not exceed [a certain threshold], [the detection will be considered successful]. The system uses linear interpolation to complete the bounding boxes; if the number of consecutive missing frames exceeds If so, then the intention prediction for that trajectory segment is paused.

[0188] (2) Asymmetric expansion of target region and visual sequence generation For each frame's original bounding box The system follows the appendix Figure 3 The method shown performs asymmetric expansion. In one implementation, the following can be configured:

[0189] This yields the expanded boundary frame. Compared to the original pedestrian detection bounding box, this expanded area retains more information about the area underfoot and road boundaries. Subsequently, the system performs cropping, boundary padding, scaling, and normalization on each frame, uniformly adjusting the cropped image to... Resolution, stacked in chronological order as follows:

[0190] (3) Text prompt set and semantic prototype construction System builds a set of positive suggestions This includes descriptions such as "a pedestrian is preparing to cross the street," "a pedestrian is approaching the curb and facing the road," and "a pedestrian is about to enter the roadway"; a set of negative cues is constructed. Descriptions include "pedestrians waiting on the sidewalk and not crossing the street", "pedestrians walking parallel to the road and not entering the roadway", and "pedestrians remaining in a safe area".

[0191] The system settings allow for a learning context hint length of [length]. The learnable contextual cues and human text cues are concatenated and input into a text encoder to obtain a positive category semantic prototype. and negative category semantic prototype During the online inference phase, the aforementioned textual semantic prototypes can be pre-cached.

[0192] (4) Visual encoding, LoRA fine-tuning and temporal aggregation The system employs a visual encoder from a pre-trained visual language model to extract frame-level visual features and inserts LoRA branches into some projection layers of the visual Transformer. In one implementation, the LoRA rank can be set to... The scaling factor can be set to During training, most parameters of the visual encoder and text encoder are frozen, and only the LoRA parameters, temporal aggregation module parameters, projection layer parameters, and learnable contextual cue parameters are updated.

[0193] right After temporal aggregation of frame visual features, the system obtains the target pedestrian. Temporal visual representation And map it to a shared semantic space consistent with the text semantic prototype.

[0194] (5) Proxy tag generation and model training The labels for training samples are generated from actual behavior within the future observation interval. In one implementation, the future observation length can be set to... Frame. If the target pedestrian In the interval If a vehicle enters the roadway or begins to cross the road, a [setting name] will be set. Otherwise set .

[0195] During model training, a binary cross-entropy loss with class compensation weights is used, and regularization constraints are applied to learnable contextual cues, LoRA parameters, and projection layer parameters to reduce the risk of overfitting.

[0196] (6) Online reasoning and system output During online inference, the system performs steps (1) to (4) for each tracked pedestrian to obtain... Then calculate:

[0197]

[0198] Output the probability of the intention to cross the street. In one implementation, a decision threshold can be set. When the following conditions are met:

[0199] The system outputs a "target pedestrian may be crossing the street" alert to the vehicle perception and decision-making module, and sends the target pedestrian's ID, current bounding box, predicted probability, and timestamp to the downstream planning and control module. The downstream module can then combine vehicle speed, distance, road speed limits, and the status of other traffic participants to further determine whether to slow down, alert the driver, or adjust the driving strategy.

Claims

1. A method for predicting pedestrians' intention to cross the street, characterized in that, include: S1: Acquire the video stream and perform target tracking on the video stream to obtain a bounding box sequence; S2: Asymmetric expansion and cropping of the bounding box sequence yields a time-series image segment; S3: Construct a prediction network based on a visual language pre-trained model, which includes a text encoder and a visual encoder; S4: Input the temporal image fragment into the visual encoder to wait for visual representation, input text prompts into the text encoder to obtain positive / negative text semantic prototypes; calculate the similarity between the visual representation and the positive / negative text semantic prototypes and normalize it to obtain the street crossing probability.

2. The method for predicting pedestrian crossing intentions according to claim 1, characterized in that, Specifically, in step S1: The video stream is: Using object detection and multi-object tracking algorithms, target pedestrians are obtained. In length of The sequence of bounding boxes within the historical observation window: The single-frame bounding box is represented by the center point coordinates as follows: In the formula, and They represent the first The x and y coordinates of the center point of the bounding box of the target pedestrian in the frame. and These represent the width and height of the bounding box, respectively. To address the issue of missing frames in real-world road scenes caused by occlusion, decreased detection confidence, or short-term tracking interruptions, a fault tolerance threshold is set. When the number of consecutive missing frames does not exceed When linear interpolation is used to complete the bounding box trajectory, when the number of consecutive missing frames exceeds [a certain threshold], [the remaining frame count is determined by the missing frame count]. When the current trajectory segment construction is terminated, the trajectory deviation that may be introduced by long-term interpolation is reduced.

3. The method for predicting pedestrian crossing intentions according to claim 1, characterized in that, In step S2, specifically: In the horizontal direction, information about the pedestrian's left and right surroundings is preserved. In the vertical direction, a downward-biased outward expansion strategy is adopted to preserve environmental cues related to crossing judgment, such as the area underfoot, curb, zebra crossing, and carriageway boundary. Let the horizontal expansion coefficient be... The vertical outward expansion coefficient is The vertical downward expansion coefficient is And satisfy: Based on the original bounding box The expanded target area boundary is denoted as: Its geometric parameter update rule is as follows: After horizontal expansion, the x-coordinate of the target center remains unchanged, i.e. ; After expanding vertically outwards, because the expansion ratio on the lower side is greater than the expansion ratio on the upper side, the center point of the target area changes from... Move down to Based on the extended bounding box In the original image frame Crop a portion of the image; when the cropped area exceeds the original image boundary, perform zero-padding or boundary padding on the out-of-bounds portion; after cropping, uniformly scale the image to the target resolution. The images are then normalized to their pixels. Finally, the frames are stacked in chronological order to obtain a continuous image fragment containing the target pedestrian and the surrounding road environment. 。 4. The method for predicting pedestrian crossing intentions according to claim 1, characterized in that, In step S3, the process of constructing text semantic features by the visual encoder includes: Construct a set of natural language prompts representing two types of intent states; the set of positive prompts is denoted as... , used to describe behavioral states with a tendency to cross the street; the set of negative cues is denoted as This is used to describe a behavioral state in which the person does not show a tendency to cross the street. The sets of positive and negative cues are first combined with learnable contextual cues; Subsequently, the text encoder in the visual language pre-trained model was invoked. Extract text features; keep text encoder parameters frozen, only update learnable context cue parameters; for any cue Its normalized text features are: The text features in the positive feedback set are averaged and normalized to obtain the positive category semantic prototype: Similarly, by averaging and normalizing the text features in the negative cue set, we obtain the semantic prototype of the negative category: 。 5. The method for predicting pedestrian crossing intentions according to claim 4, characterized in that, Learnable contextual cues specifically include: The learnable context hints are of length [length missing]. A continuous vector sequence: in, Represents the text word embedding dimension; for any human-generated text prompt First, it is segmented and embedded, and then the learnable context vector is concatenated before the text embedding sequence to form an adaptive cue sequence. in, Indicates a text prompt The word segmentation results This represents a text word embedding map.

6. The method for predicting pedestrian crossing intentions according to claim 1, characterized in that, In step S3, the visual encoder performs visual feature extraction as follows: The visual encoder in the visual language pre-trained model is selected as the visual backbone, denoted as . For any single frame image in the time-series image segment generated in step S2 The visual encoder outputs frame-level visual features: Insert a LoRA low-rank adaptation branch in the projection layer of the Transformer self-attention module of the visual encoder.

7. The method for predicting pedestrian crossing intentions according to claim 6, characterized in that, In step S3, specifically: Input feature matrix Simultaneously, it enters two branches, one of which is the frozen pre-trained projection layer with weights of 1. The other branch is the LoRA low-rank adaptation branch, which is determined by the matrix. and Composition, in which: The merged projection output is: in, Indicates the output of the projection layer. For frozen pre-trained weights, and For trainable low-rank fitting parameters, This is the scaling factor. The rank of the low-rank decomposition; When the low-rank adaptation branches of LoRA are applied to the self-attention modules , and When projecting a layer, it can be represented as: After completing frame-level visual feature extraction, the visual features within the historical window are further aggregated temporally; The frame features are concatenated in chronological order as follows: Overlay time position encoding Then, input the temporal aggregation network: in, The implementation employs recurrent neural networks, temporal convolutional networks, or Transformer encoders based on self-attention mechanisms; temporal aggregation is used to capture continuous changes in pedestrian movements; and temporal features are obtained. .

8. The method for predicting pedestrian crossing intentions according to claim 7, characterized in that, In step S3, specifically: During LoRA low-rank adaptation training initialization, the matrix Random initialization is used for the matrix. Initialize as a zero matrix, such that the initial phase satisfies This ensures that the initial output of the model remains consistent with the pre-trained model; Subsequently, the model gradually learns adaptation parameters related to the target traffic scenario during training; the LoRA low-rank adaptation branch can be loaded into... , , One or more branches in the projection layer, the specific configuration of which can be selected according to the computing power and data scale of the deployment platform.

9. The method for predicting pedestrian crossing intentions according to claim 1, characterized in that, In step S4, specifically: temporal features are obtained by a visual encoder based on continuous image segments containing the target pedestrian and its surrounding road environment. Then through the projection matrix The pooled temporal features are mapped to a shared semantic space consistent with the text semantic prototype using a projection matrix: in, Indicates the target pedestrian The normalized temporal visual representation; this representation simultaneously includes visual appearance, road context, and historical action change information.

10. A method for predicting pedestrian crossing intentions according to claim 9, characterized in that, In step S4, specifically: Calculate similarity in the shared semantic space and output the probability of pedestrians' intention to cross the street accordingly: The actual behavior results within the future observation interval are used as surrogate supervision signals. Let the future observation interval be: If the target pedestrian If any action occurs within this interval, such as crossing the curb, entering the roadway, or beginning to cross the road, then the current time will be... The corresponding sample is labeled as positive; if the above behavior does not occur, it is labeled as negative. In the prediction phase, normalized visual representations are calculated. Similarity with positive and negative text semantic prototypes: in, This represents the similarity between the visual representation of the target pedestrian and the semantic prototype of the pedestrian preparing to cross the street. This indicates the similarity between it and the semantic prototype of "not prepared to cross the street"; Introducing temperature parameters Then, the target pedestrian is obtained through normalized mapping. Probability of predicting pedestrian crossing intentions: when Not less than the preset threshold At that time, determine the target pedestrian They must have the intention to cross the street at the current moment; otherwise, they are deemed not to have shown a clear intention to cross the street. During the training phase, a parameter optimization strategy of freezing the backbone and updating locally is adopted; most of the pre-trained parameters of the visual encoder and text encoder are kept frozen, and only the set of trainable parameters is updated. Including LoRA low-rank adaptation parameters Time series aggregation module parameters Cross-modal projection layer parameters and learnable contextual hint parameters ; For batch size The training data is used with binary cross-entropy loss with class compensation weights, combined with parameter regularization constraints. The total loss function is expressed as: in, and These are the weights for the positive and negative classes, respectively. , and The regularization coefficient is . This indicates the level at which the LoRA branch is loaded.

Images