Pedestrian trajectory prediction method and device applied to vehicle, equipment and medium

Abstract

The invention discloses a pedestrian trajectory prediction method and device applied to a vehicle, equipment and a medium. The method comprises the following steps: in a vehicle driving process, acquiring multiple frames of to-be-processed images of a surrounding environment of a vehicle, detecting a detection frame of at least one pedestrian in the multiple frames of to-be-processed images, and associating the detection frames of the same pedestrian to obtain a pedestrian track frame; determining a pedestrian optical flow graph based on the adjacent frames of the to-be-processed image and the pedestrian trajectory frame; obtaining vehicle state data of the vehicle, and determining target motion features of the pedestrian based on the vehicle state data, the pedestrian trajectory frame and the pedestrian optical flow graph; and determining a predicted position of the pedestrian based on the target motion feature. According to the technical scheme provided by the invention, the accuracy of pedestrian trajectory prediction is improved, so that the effect of optimizing the passing efficiency while ensuring the pedestrian safety is achieved.

Description

Technical Field

[0001] This invention relates to the field of computer processing technology, and in particular to a method, apparatus, device, and medium for predicting pedestrian trajectories in vehicles. Background Technology

[0002] In complex and dynamic urban traffic environments, pedestrians, as key traffic participants, exhibit highly random and uncertain movement behaviors, easily influenced by various factors such as their own intentions, interactions with surrounding vehicles, road structure, and traffic signals. To ensure the driving safety and traffic efficiency of intelligent vehicles, it is typically necessary to predict pedestrian trajectories.

[0003] Currently, existing pedestrian trajectory prediction methods mainly rely on the pedestrian's historical movement data to extrapolate their future trajectory. However, when the vehicle is in motion, prediction methods based solely on historical trajectory coordinates cannot distinguish between the pedestrian's actual movement and the observational bias caused by relative motion, easily leading to inaccurate predictions and resulting in problems such as low driving safety and low traffic efficiency. Summary of the Invention

[0004] This invention provides a method, device, equipment, and medium for predicting pedestrian trajectories in vehicles, in order to improve the accuracy of pedestrian trajectory prediction and thereby optimize traffic efficiency while ensuring pedestrian safety.

[0005] According to one aspect of the present invention, a method for predicting pedestrian trajectories applied to vehicles is provided, the method comprising:

[0006] During vehicle operation, acquire multiple frames of images of the surrounding environment of the vehicle to be processed;

[0007] In the multiple frames of the image to be processed, at least one detection box of a pedestrian is detected, and the detection boxes of the same pedestrian are associated to obtain the pedestrian trajectory box;

[0008] Based on the image to be processed in adjacent frames and the pedestrian trajectory bounding box, determine the pedestrian optical flow map;

[0009] The vehicle status data of the vehicle is acquired, and the target motion features of the pedestrian are determined based on the vehicle status data, the pedestrian trajectory box, and the pedestrian optical flow map.

[0010] Based on the target's motion characteristics, the predicted location of the pedestrian is determined.

[0011] According to another aspect of the present invention, a pedestrian trajectory prediction device for vehicles is provided, the device comprising:

[0012] The image acquisition module is used to acquire multiple frames of images of the surrounding environment of the vehicle during vehicle operation.

[0013] a pedestrian trajectory box determination module, configured to detect a detection box of at least one pedestrian in a plurality of frames of the to-be-processed images, and associate the detection boxes of the same pedestrian to obtain a pedestrian trajectory box;

[0014] a pedestrian optical flow map determination module, configured to determine a pedestrian optical flow map based on the to-be-processed images of adjacent frames and the pedestrian trajectory box;

[0015] a target motion feature determination module, configured to acquire vehicle state data of the vehicle, and determine a target motion feature of the pedestrian based on the vehicle state data, the pedestrian trajectory box and the pedestrian optical flow map;

[0016] a predicted position determination module, configured to determine a predicted position of the pedestrian based on the target motion feature.

[0017] According to another aspect of the present application, an electronic device is provided, which comprises:

[0018] at least one processor; and a memory connected to the at least one processor in communication; wherein,

[0019] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the pedestrian trajectory prediction method applied to a vehicle according to any one of the embodiments of the present application.

[0020] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the pedestrian trajectory prediction method applied to a vehicle according to any one of the embodiments of the present application when executed by the processor.

[0021] According to another aspect of the present application, a computer program product is provided, which comprises a computer program for implementing the pedestrian trajectory prediction method applied to a vehicle according to any one of the embodiments of the present application when executed by a processor.

[0022] The technical scheme of the embodiment of the present application comprises the following steps: acquiring multiple frames of to-be-processed images of the surrounding environment of the vehicle during vehicle driving, detecting at least one pedestrian bounding box in the multiple frames of to-be-processed images, and associating the bounding boxes of the same pedestrian to obtain a pedestrian trajectory box; determining a pedestrian optical flow map based on adjacent frames of to-be-processed images and the pedestrian trajectory box; acquiring vehicle state data of the vehicle, and determining a target motion feature of the pedestrian based on the vehicle state data, the pedestrian trajectory box and the pedestrian optical flow map; and determining a predicted position of the pedestrian based on the target motion feature. The problems of low driving safety and low traffic efficiency caused by the fact that the prediction method based on historical trajectory coordinates in the prior art cannot distinguish between the real motion of the pedestrian and the observation deviation caused by relative motion and is prone to prediction inaccuracy are solved. The multiple frames of to-be-processed images of the surrounding environment of the vehicle during vehicle driving are acquired, the bounding boxes of the same pedestrian are detected and associated to construct a pedestrian trajectory box, then the pedestrian optical flow map focused on the pedestrian region is generated in combination with adjacent frames of to-be-processed images and the pedestrian trajectory box, and the vehicle state data, the pedestrian trajectory box and the pedestrian optical flow map are further fused to comprehensively model the macro motion trend, the micro dynamic details and the influence of the relative motion of the host vehicle of the pedestrian, so as to accurately extract the target motion feature of the pedestrian and predict its future position, so that the real motion of the pedestrian and the observation deviation caused by the motion of the host vehicle can be clearly distinguished, and the accuracy and robustness of trajectory prediction are improved. Thus, the behavior of the pedestrian can be more reliably predicted in a complex urban traffic scene, and the technical effect of optimizing traffic efficiency while ensuring pedestrian safety by avoiding overly conservative or aggressive decisions is achieved.

[0023] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1 is a flowchart of a pedestrian trajectory prediction method applied to a vehicle according to an embodiment of the present application;

[0026] Figure 2 is a flowchart of a pedestrian trajectory prediction method applied to a vehicle according to an embodiment of the present application;

[0027] Figure 3 is a flowchart of a pedestrian trajectory prediction method applied to a vehicle according to an embodiment of the present application;

[0028] Figure 4 is a structural schematic diagram of a pedestrian trajectory prediction device applied to a vehicle according to an embodiment of the present application;

[0029] Figure 5 is a structural schematic diagram of an electronic device implementing a pedestrian trajectory prediction method applied to a vehicle according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] It should be noted that in the technical solutions of the present disclosure, the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solutions of the present disclosure comply with relevant laws and regulations, are used for legal purposes, and do not violate public order and good customs. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain user personal information security and network security. It should also be noted that in the technical solutions of the present disclosure, the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solutions of the present disclosure are carried out with the knowledge and permission of the user, in compliance with relevant privacy protection regulations.

[0033] Figure 1Fig. 1 is a flowchart of a pedestrian trajectory prediction method applied to a vehicle according to an embodiment of the present application. The embodiment can be applied to predict the trajectory of a pedestrian during vehicle driving to avoid obstacles in advance and ensure the safety of pedestrians and vehicles. The method can be performed by a pedestrian trajectory prediction device applied to a vehicle, which can be implemented in the form of hardware and / or software and can be configured in a computing device. Figure 1 The method comprises the following steps.

[0034] S110. During vehicle driving, a plurality of frames of to-be-processed images of the surrounding environment of the vehicle are acquired.

[0035] The vehicle can be any vehicle that needs to avoid obstacles. The surrounding environment of the vehicle can refer to a dynamic and static traffic scene covering a certain range in front, side and / or rear of the vehicle. For example, the surrounding environment of the vehicle can be the road environment around the vehicle. The to-be-processed images are real-time surrounding environment images collected during vehicle driving, which can be collected by at least one camera deployed on the vehicle. The plurality of frames of to-be-processed images are two-dimensional or three-dimensional image frames collected continuously or approximately continuously in time, which are used to reflect the change process of the environment over time. For example, at each time, the to-be-processed images can include a plurality of frames of images collected in real time by a four-way surround-view camera.

[0036] During vehicle driving, the external environment of the vehicle can be sensed in real time by a vehicle-mounted camera and other sensors, and a plurality of frames of to-be-processed images can be collected. The number and type of cameras deployed can be set according to actual collection requirements, and the embodiment is not limited in detail. Each camera set can capture each to-be-processed image at the same time, which can reflect the surrounding environment of the vehicle.

[0037] S120. At least one detection box of a pedestrian is detected in the plurality of frames of to-be-processed images, and the detection boxes of the same pedestrian are associated to obtain a pedestrian trajectory box.

[0038] The detection box is a region in the to-be-processed image that marks the position of the pedestrian with a rectangular bounding box or other forms. The detection box can include coordinate information (such as a center point, a width and height, or a corner point) of the detection target (such as a pedestrian or other obstacles) and attributes such as detection confidence. The detection boxes of the same pedestrian refer to a plurality of bounding boxes belonging to the same pedestrian individual in a plurality of frames of to-be-processed images. Each detection box represents the spatial position and scale information of the pedestrian in the corresponding frame. The pedestrian trajectory box refers to a sequence of continuous detection boxes of the same pedestrian obtained by cross-frame association, which is used to represent the spatial motion path of the pedestrian in a period of time.

[0039] In this embodiment, pedestrian detection can be performed on each frame of the image to be processed to obtain at least one detection box for a pedestrian. Specifically, the vehicle-mounted computing unit can sequentially receive multiple frames of images to be processed, input each frame of the image to be processed into a pre-trained target detection model, and the model independently performs pedestrian detection on each frame of the image to be processed, perceives the spatial distribution of pedestrians in the environment, and outputs detection boxes for all pedestrians identified in each frame of the image to be processed. Optionally, the target detection model can be a YOLO model, Faster R-CNN (Faster Region-based Convolutional Neural Network), SSD (Single Shot MultiBox Detector), or Mask R-CNN (Mask Region-based Convolutional Neural Network); it can also be an end-to-end detection model built based on a deep learning architecture (such as Transformer or RNN (Recurrent Neural Network)).

[0040] For example, see Figure 2 The system can acquire a sequence of images to be processed (i.e., multiple frames of images); the Mask-RCNN network can be used to compute pedestrian detection boxes in the images. The advantage of using the Mask-RCNN network is that it ensures detection accuracy and classification performance while meeting the real-time requirements for pedestrian trajectory prediction at the required frame rate. The sequence of images to be processed can be input into the Mask-RCNN network, and the network outputs pedestrian detection boxes for each frame. The sequence of images to be processed can be represented as: ; express The RGB image at that time (i.e., the image to be processed). and These represent the height and width of the image to be processed, respectively. This indicates the number of channels in the image to be processed. For the observation time domain, that is, how many seconds of historical images to be processed are used for trajectory prediction.

[0041] For a single frame of an image to be processed, the Mask-RCNN network can represent the detection bounding box of at least one pedestrian in the image through a function mapping. The function of the Mask-RCNN network can be expressed as:

[0042] ;in, ; express Time of the first A pedestrian detection bounding box in the image to be processed the center position (including the horizontal coordinate and the vertical coordinate ) and the size (including the width and the height ) of the bounding box of the detected pedestrian in the to-be-processed image ; the number of pedestrians detected in the to-be-processed image , and the mapping symbol.

[0043] In the training of the target detection model, how to fuse dynamic visual clues (such as motion blur and relative displacement trend) can be learned, so as to have the ability to suppress the interference of the host vehicle motion in the inference stage, and ensure the stability and generalization of the model in the complex urban scene.

[0044] Further, a multi-target tracking algorithm (such as DeepSORT or ByteTrack) can be used to associate the pedestrian detection boxes in the to-be-processed images of consecutive frames across frames, to generate stable and continuous pedestrian trajectory boxes.

[0045] For example, in order to obtain the time sequence association information of pedestrians in the pedestrian trajectory prediction task, the pedestrian detection boxes in each frame of to-be-processed image can be associated to form a continuous vector corresponding to the pedestrian, that is, a pedestrian trajectory box. For example, through a pedestrian tracking algorithm, the pedestrian trajectory box of each pedestrian can be associated.

[0046] The pedestrian trajectory box can be represented as: ; .

[0047] wherein, represents the pedestrian trajectory box formed by splicing the pedestrian detection boxes within the observation time domain of the th pedestrian in the scene at the th moment.

[0048] S130, determining a pedestrian optical flow map based on adjacent frame to-be-processed images and pedestrian trajectory boxes.

[0049] wherein, the adjacent frame to-be-processed image refers to two or more images captured by the vehicle-mounted camera in succession or with a very short interval, for capturing the motion changes of dynamic objects (such as pedestrians) in the environment. The pedestrian optical flow map refers to a pixel-level motion vector field calculated for the region defined by the pedestrian trajectory box between adjacent frame to-be-processed images, wherein each pixel point in the pedestrian optical flow map contains the displacement information of the same pixel in the horizontal and vertical directions between adjacent frames, reflecting the local motion pattern of the pedestrian on the image plane.

[0050] ​In the embodiment, the pedestrian region can be cropped from the adjacent frame to-be-processed image by using the pedestrian trajectory box, and then a pixel-level motion vector can be calculated in the pedestrian region by using an optical flow estimation algorithm (such as Lucas-Kanade (LK optical flow algorithm) or Farnebäck method (optical flow algorithm based on polynomial expansion)), to generate a pedestrian optical flow map. A pre-trained deep optical flow estimation network (such as RAFT (Recurrent All-Pairs Field Transforms, or recurrent all-pairs field transform model) or FlowNet (optical flow network)) can also be used to generate a high-precision full-image optical flow field from the adjacent frame to-be-processed image, and then the full-image optical flow field is regionally cropped according to the pedestrian trajectory box to obtain the pedestrian optical flow map.

[0051] For example, when predicting the pedestrian trajectory, the adjacent two frames of to-be-processed images can be input into the RAFT model, and a dense optical flow field is output as the pedestrian optical flow map. The objective function of the RAFT model can be represented as: ; The horizontal displacement and the vertical displacement of a plurality of pixels are included in the objective function. ; The horizontal displacement of a pixel at time t is represented as The horizontal displacement of a pixel at time t is represented as The vertical displacement of a pixel at time t is represented as The vertical displacement of a pixel at time t is represented as The to-be-processed image at time t is represented as The to-be-processed image at time t is represented as The to-be-processed image at time t is represented as The to-be-processed image at time t is represented as The pedestrian optical flow map between t and t is represented as The pedestrian optical flow map between t and t is represented as The pedestrian optical flow map between t and t is represented as The pedestrian optical flow map between t and t is represented as The pedestrian optical flow map between t and t is represented as The pedestrian optical flow map between t and t is represented as

[0052] It should be noted that the size of the optical flow map obtained by comparing the pixel positions between the adjacent frames of to-be-processed images is consistent with the size of the original to-be-processed image, but only the region describing the pedestrian motion in the optical flow map has a promoting effect on trajectory prediction, and the remaining regions are redundant information. Therefore, the optical flow map can be aligned with the pedestrian detection box, and the corresponding pedestrian region can be cropped according to the pedestrian detection box, so as to obtain the pedestrian optical flow map of the pedestrian.

[0053] Specifically, based on the adjacent frame to-be-processed images and the pedestrian trajectory box, the pedestrian optical flow map is determined, including: based on the adjacent frame to-be-processed images, a pixel optical flow map is determined; and based on the pedestrian trajectory box, the pixel optical flow map is cropped to obtain the pedestrian optical flow map.

[0054] ​​The pixel optical flow map refers to a pixel-level motion vector field covering the entire image to be processed, which is calculated between adjacent frames, wherein each pixel position contains a two-dimensional vector representing the displacement (including horizontal and vertical components) of the point from the current frame to the next frame on the image plane, used to describe the visual motion pattern of the entire image scene. The pedestrian optical flow map refers to a local optical flow sub-map containing only the pedestrian region, which is cropped from the pixel optical flow map, used to focus on the fine-grained motion details of the pedestrian on the image plane.

[0055] In this embodiment, an optical flow estimation algorithm (such as Lucas-Kanade sparse optical flow or Farnebäck dense optical flow) can be used to process the image to be processed in the adjacent frame, to generate a pixel optical flow map covering the entire image. Using the corresponding pedestrian trajectory frame in the current image to be processed as a spatial mask, a geometric cropping operation is performed on the pixel optical flow map to obtain a rectangular region aligned with the pedestrian detection frame coordinates, and a pedestrian optical flow map is obtained.

[0056] Alternatively, a depth optical flow estimation model (such as RAFT, PWC-Net or GMFlow) can be used to process the image to be processed in the adjacent frame, to generate a high-precision, noise-resistant pixel optical flow map. According to the coordinate information of the pedestrian trajectory frame, the pixel optical flow map is accurately spatially cropped; to further improve robustness, an extended boundary (such as an external expansion of 10%) technique can also be introduced to accommodate the case where the optical flow map exceeds the detection frame in the movement of the pedestrian's limbs, ensuring that the key pedestrian movement area is not truncated, and obtaining the pedestrian optical flow map.

[0057] On the basis of the above, a time alignment or consistency checking mechanism (such as removing abnormal frames or interpolating to complete) can also be introduced to ensure that the obtained pedestrian optical flow map is smooth and coherent in the time dimension.

[0058] For example, referring to Figure 2 , the pixel optical flow map is aligned with the pedestrian detection frame, and the first pedestrian region in the pixel optical flow map is cropped to obtain the pedestrian optical flow map. The pedestrian optical flow map can be represented as: ; wherein, represents the pedestrian optical flow map corresponding to the first pedestrian detection frame at time t.

[0059] The technical solution provided by the embodiment realizes accurate extraction from a global visual motion field to individual dynamic details by first generating a pixel optical flow map based on adjacent frames of to-be-processed images, and then cutting the pedestrian optical flow map according to the pedestrian trajectory frame. This operation effectively strips the background interference (such as moving vehicles, swaying vegetation, etc.), concentrates the computing resources on the motion area closely related to the safety of the pedestrian, so that the obtained pedestrian optical flow map can more truly and delicately reflect the local motion state of the pedestrian (such as foot movement, turning tendency or sudden acceleration), enhance the understanding ability of the pedestrian's intention, and ensure the accuracy of the judgment on the pedestrian's behavior.

[0060] In S140, vehicle state data of the vehicle is acquired, and a target motion feature of the pedestrian is determined based on the vehicle state data, the pedestrian trajectory frame and the pedestrian optical flow map.

[0061] The vehicle state data refers to multidimensional sensing information reflecting the motion state of the host vehicle, such as, but not limited to, vehicle speed, acceleration, heading angle, yaw rate, steering angle, etc., for describing the dynamic behavior and motion trend of the vehicle at the current time. In the embodiment, the pedestrian trajectory frame can represent the macroscopic spatiotemporal motion path of the pedestrian as a whole. The pedestrian optical flow map can represent the local dynamic details of the pedestrian (such as limb swing, gait change, etc.). The target motion feature refers to a comprehensive feature representation generated by fusing the vehicle state data, the pedestrian trajectory frame and the pedestrian optical flow map, for describing the relative motion state and future behavior trend of the pedestrian in the coordinate system of the host vehicle.

[0062] In actual application, the pedestrian trajectory frame can be encoded by a fully connected layer or a recurrent network to generate a trajectory embedding feature; the pedestrian optical flow map can be convolutionally encoded or statistical feature extraction to obtain an optical flow embedding feature; and the vehicle state data can be normalized and embedded to obtain a vehicle state feature. Further, the trajectory embedding feature, the optical flow embedding feature and the vehicle state feature can be aligned in dimension and spliced to input a fusion network (such as a multilayer perceptron) for joint modeling, and finally output the target motion feature.

[0063] Alternatively, an end-to-end feature extraction model (such as a neural network) can be constructed in advance, with the pedestrian trajectory frame, the pedestrian optical flow map and the vehicle state data as input data of the feature extraction model. The pedestrian trajectory frame can be processed by a time series encoder (such as a Transformer or an LSTM) in the feature extraction model, the pedestrian optical flow map can be extracted by a convolutional backbone network in the feature extraction model, and the vehicle state data can be input to an intermediate layer of the feature extraction model for feature processing. The three features are fused by a cross-attention or feature interaction module in the feature extraction model to generate the target motion feature.

[0064] Or, the pedestrian trajectory box and the pedestrian optical flow map can be relatively motion compensated using the vehicle state data (e.g., converting the pedestrian motion to a world coordinate system or a stationary reference system), and then the compensated trajectory features and the optical flow features are extracted respectively. The compensated trajectory features, the optical flow features, and the vehicle state data are input into a fusion module to generate target motion features that conform to physical laws and retain original observation details.

[0065] In S150, a predicted position of the pedestrian is determined based on the target motion features.

[0066] The pedestrian refers to a traffic participant other than the vehicle in a traffic scene in which the vehicle is located. The predicted position refers to a spatial coordinate position in which the pedestrian can appear at a future preset time point (e.g., 0.5 seconds, 1 second later), which can be expressed in the form of an image coordinate or a world coordinate.

[0067] In this embodiment, the target motion features can be input into a regression prediction head composed of one or several fully connected layers, and the predicted position coordinates of the pedestrian at a future time point (e.g., x and y coordinates on a two-dimensional image plane or X and Y coordinates in a three-dimensional world coordinate) are output. The prediction head is trained end-to-end, that is, the model can learn a non-linear mapping relationship from complex motion features to future positions by using real trajectory labels for training in the training stage.

[0068] Alternatively, the target motion features can be input into a time sequence generation module (e.g., an LSTM, a GRU, or a Transformer decoder), which can output predicted positions at multiple future time steps step by step to form a predicted trajectory. Finally, a position at a key time point (e.g., 1 second later) can be selected as the predicted position of the pedestrian. This method can make the prediction result more consistent with physical laws; meanwhile, trajectory smoothing loss or dynamics prior can be introduced to further improve the rationality and stability of the prediction.

[0069] Alternatively, the target motion features can be parameterized into a probability distribution (e.g., a Gaussian distribution or a multi-modal mixed distribution), from which a predicted position can be obtained by sampling or taking an expected value. Specifically, the mean and variance of the probability distribution can be output, and the mean is the final predicted position while the variance reflects the prediction confidence. The advantage of this setting is that in a complex scene in which the behavior of the pedestrian is highly random or has multiple intentions (e.g., lingering at an intersection or suddenly changing direction), the driving safety can be improved by dynamically adjusting the avoidance strategy based on the predicted position of the pedestrian.

[0070] On the basis of the above scheme, after the predicted position of the pedestrian is determined, the high-precision and high-confidence predicted position of the pedestrian can be used for path planning, collision warning, and active avoidance of the vehicle, to ensure the safety and efficiency of driving.

[0071] The technical scheme of the embodiment of the present application comprises the following steps: acquiring multiple frames of to-be-processed images of the surrounding environment of a vehicle during vehicle driving; detecting a detection box of at least one pedestrian in the multiple frames of to-be-processed images, and associating the detection boxes of the same pedestrian to obtain a pedestrian trajectory box; determining a pedestrian optical flow map based on adjacent frames of to-be-processed images and the pedestrian trajectory box; acquiring vehicle state data of the vehicle, and determining a target motion feature of the pedestrian based on the vehicle state data, the pedestrian trajectory box and the pedestrian optical flow map; and determining a predicted position of the pedestrian based on the target motion feature. The embodiment of the present application solves the problem that the prediction mode based on historical trajectory coordinates in the prior art cannot distinguish between the real motion of a pedestrian and the observation deviation caused by relative motion, is prone to prediction inaccuracy, and thus leads to low driving safety and low traffic efficiency, and achieves the following technical effects: multiple frames of to-be-processed images of the surrounding environment of a vehicle are acquired during vehicle driving, the detection boxes of the same pedestrian are detected and associated to construct a pedestrian trajectory box, then, the pedestrian optical flow map focused on the pedestrian region is generated in combination with adjacent frames of to-be-processed images and the pedestrian trajectory box, and the vehicle state data, the pedestrian trajectory box and the pedestrian optical flow map are further fused to comprehensively model the macro motion trend, the micro dynamic details and the influence of the relative motion of the host vehicle of the pedestrian, so as to accurately extract the target motion feature of the pedestrian and predict the future position thereof, so that the real motion of the pedestrian and the observation deviation caused by the motion of the host vehicle can be clearly distinguished, and the accuracy and robustness of trajectory prediction are improved. Thus, the behavior of the pedestrian can be more reliably predicted in a complex urban traffic scene, and the technical effect of optimizing traffic efficiency while ensuring pedestrian safety by avoiding overly conservative or aggressive decisions is achieved.

[0072] Figure 3 A flowchart of a pedestrian trajectory prediction method applied to a vehicle is provided according to the embodiment of the present application. The technical scheme of the present embodiment is further refined on the basis of the above-mentioned embodiment, and the specific implementation can be referred to the detailed description of the embodiment of the present application. The technical features same as or similar to the foregoing embodiments will not be described herein.

[0073] As shown in Figure 3 , the method of the present embodiment can specifically comprise the following steps:

[0074] S210, acquiring multiple frames of to-be-processed images of the surrounding environment of a vehicle during vehicle driving.

[0075] S220, detecting a detection box of at least one pedestrian in the multiple frames of to-be-processed images, and associating the detection boxes of the same pedestrian to obtain a pedestrian trajectory box.

[0076] S230, determining a pedestrian optical flow map based on adjacent frames of to-be-processed images and the pedestrian trajectory box.

[0077] S240, processing the pedestrian trajectory box to obtain a detection box embedding feature.

[0078] wherein the detection box embedding feature refers to a low-dimensional dense vector mapped from the pedestrian trajectory box, used to express the motion pattern, spatio-temporal dynamics and context information of the pedestrian, etc.

[0079] In an implementation manner, for each pedestrian trajectory box, its geometric property (such as center coordinate change, speed, acceleration) feature and appearance statistical feature (such as average size, trajectory length) are extracted, and these extracted features are encoded into a fixed-dimensional vector through splicing or weighted fusion, as the detection box embedding feature of the trajectory.

[0080] In another implementation manner, the pedestrian trajectory box can be regarded as time series data, and a recurrent neural network (such as LSTM, RNN or GRU) can be used to model the pedestrian trajectory box to obtain the detection box embedding feature. Specifically, each frame detection box in the pedestrian trajectory box is input into the recurrent neural network in time sequence, and the motion dynamics of the pedestrian is automatically learned from the hidden state of the recurrent neural network. Finally, the hidden state (or its aggregated form) output by the recurrent neural network is taken as the detection box embedding feature of the whole trajectory. This manner can effectively capture the nonlinear motion trend and long-term time series dependence of the pedestrian.

[0081] In yet another implementation manner, the pedestrian trajectory box can be modeled as a graph structure or a sequence structure, and a graph neural network (GNN) or a Transformer architecture can be used to aggregate the global features of the pedestrian trajectory box to obtain the detection box embedding feature. For example, under the Transformer framework, each detection box in the pedestrian trajectory box is encoded into an embedding vector containing position and timestamp information. The inter-frame dependence is modeled through a self-attention mechanism, and the detection box embedding feature of the whole trajectory is generated through a pooling operation or by introducing a specific query vector (such as a learnable trajectory query). The advantage of this setting is that it can simultaneously fuse local details and global context, thereby improving the feature expression capability in complex scenarios such as occlusion recovery, trajectory completion and multi-pedestrian interaction modeling.

[0082] In addition, the detection boxes belonging to the same pedestrian can be matched and connected between different frames according to the appearance features, motion consistency or spatio-temporal distance of the detection boxes, to construct the pedestrian trajectory box. The robust detection box embedding feature is generated by performing feature extraction, encoding or transformation on the pedestrian trajectory box, thereby improving the stability and accuracy of the pedestrian trajectory representation.

[0083] Through any of the above manners, the original and discrete single-frame detection result is converted into a structured pedestrian trajectory, and further refined into a semantic-rich embedding feature. This not only effectively suppresses the noise and uncertainty in single-frame detection, but also explicitly encodes the spatio-temporal behavior pattern of the pedestrian. This process can ensure the accuracy of subsequent trajectory prediction, intention recognition and risk assessment.

[0084] In the embodiment, the pedestrian trajectory bounding box is processed to obtain a bounding box embedding feature, including: processing the pedestrian trajectory bounding box based on a full connection layer to obtain the bounding box embedding feature.

[0085] The full connection layer is a full connection (FC) layer for feature transformation, which maps an input fixed-dimension vector to another dimension hidden representation space. The bounding box embedding feature is a low-dimensional dense vector generated after processing the pedestrian trajectory bounding box by the full connection layer, which is used to compactly express the motion pattern and geometric dynamics of the pedestrian.

[0086] In an implementation manner, each frame bounding box parameter (such as normalized center point x, y coordinates and width w, height h) in the pedestrian trajectory bounding box can be spliced into an input vector of a fixed length in time sequence; if the trajectory length is variable, the dimension of the pedestrian trajectory bounding box can be unified by padding or truncation. Further, the input vector is input into the full connection layer, and the bounding box embedding feature is output through linear mapping of the weight matrix and the bias term learned by the full connection layer and can be operated by a nonlinear activation function (such as ReLU).

[0087] In another implementation manner, the pedestrian trajectory bounding box can be preprocessed to extract its statistical motion features, such as average speed, acceleration variance, trajectory direction angle, bounding box size change rate, etc., to form a feature vector with physical meaning. The feature vector is input into the full connection layer, and the bounding box embedding feature is output after the full connection layer transformation. Compared with the original coordinate input, this method can highlight the key motion pattern and improve the detection accuracy of the pedestrian motion.

[0088] In yet another implementation manner, the pedestrian trajectory bounding box can contain multi-modal features such as geometric coordinates, confidence of each frame bounding box, appearance features, or relative distance from the host vehicle. The multi-modal features can be standardized and input into the full connection layer, which uniformly maps them into the bounding box embedding feature in the unified semantic space. This way can make full use of multi-source information, so that the bounding box embedding feature can reflect the spatial motion, visual identity and environmental context of the pedestrian at the same time, enhancing its expression ability and discriminability.

[0089] For example, referring to Figure 2 , the bounding box sequence (i.e. pedestrian trajectory bounding box) can more intuitively and explicitly reflect the kinematic information of the pedestrian, and the full connection layer network can be used to embed the bounding box position and size to a high-dimensional feature space to obtain the bounding box embedding feature. The bounding box embedding feature can be represented as: ; wherein, represents the bounding box embedding feature, represents the full connection layer network. For the training parameters in the fully connected layer network, the training parameters can be adjusted and optimized through the back propagation of the training samples to the network.

[0090] In the above manner, the pedestrian trajectory frame is processed by the fully connected layer to generate the detection frame embedding feature, so that the obtained embedding feature can more stably and accurately depict the behavior characteristics of the pedestrian, effectively alleviate the interference caused by the motion of the host vehicle, occlusion or detection jitter, and improve the perception accuracy and decision reliability of the pedestrian dynamics in a complex urban traffic environment, so as to optimize the traffic efficiency under the premise of ensuring safety.

[0091] In order to ensure the accuracy of pedestrian feature extraction, after processing the pedestrian trajectory frame to obtain the detection frame embedding feature, the detection frame embedding feature can be processed based on the full-scale network to obtain the updated detection frame embedding feature.

[0092] The full-scale network (Omni-Scale Network, OSNet) can be a deep neural network architecture based on multi-scale feature learning, which can capture feature representations under different receptive fields through a parallel multi-branch structure to enhance and refine the features.

[0093] In this embodiment, after obtaining the detection frame embedding feature, it can be input into the pre-trained or fine-tuned full-scale network (OSNet). The full-scale network can perform multi-scale context modeling on the detection frame embedding feature through its multi-branch convolutional structure, and output the processed detection frame embedding feature. The processed detection frame embedding feature is used to replace the original detection frame embedding feature. In order to determine the target motion feature based on the vehicle state data, the detection frame embedding feature and the processed pedestrian optical flow embedding feature, and ensure the accuracy of pedestrian motion recognition.

[0094] For example, the OSNet network with full-scale feature learning capability can be introduced to re-extract the target appearance feature in each pedestrian trajectory frame to obtain the updated detection frame embedding feature.

[0095] S250, processing the pedestrian optical flow map to obtain a pedestrian optical flow embedding feature.

[0096] The pedestrian optical flow embedding feature refers to a feature vector obtained by extracting and encoding the optical flow map (i.e. pixel-level motion vector) of the pedestrian region in the video sequence. For example, the pedestrian optical flow map can be subjected to feature extraction, dimension reduction, encoding or semantic abstraction, etc. to obtain a low-dimensional dense vector as the pedestrian optical flow embedding feature. The pedestrian optical flow embedding feature can represent the fine-grained dynamic behavior of the pedestrian in time sequence (such as walking direction, limb swing, sudden turning, etc.).

[0097] In this embodiment, statistical feature extraction can be performed on the pedestrian optical flow map, for example, calculating statistical features such as optical flow amplitude mean, direction histogram, principal motion axis or spatiotemporal gradient distribution, and splicing these statistical features into pedestrian optical flow embedding features. Alternatively, the pedestrian optical flow map can be input into a convolutional neural network (such as several layers of convolution plus global average pooling) to extract hierarchical motion features and output pedestrian optical flow embedding features. Alternatively, the most relevant regions to pedestrian motion in consecutive pedestrian optical flow maps can be weighted and enhanced based on an attention mechanism (such as a channel or spatiotemporal attention module), spatiotemporal features can be extracted through a three-dimensional convolution or a Transformer encoder, and pedestrian optical flow embedding features with context awareness can be generated.

[0098] It should be noted that the pedestrian optical flow map is a local optical flow sub-map cropped according to the pedestrian detection frame, reflecting the pixel-level motion vector of a specific pedestrian between adjacent frames, but the size of the pedestrian optical flow map may differ due to differences in pedestrian distance, posture or detection frame size. In order to ensure the consistency of the input data structure and eliminate the inconsistency of feature detection caused by the difference in the size of the pedestrian, the alignment of all pedestrian optical flow maps can be performed based on the pooling layer during the process of processing the pedestrian optical flow map to obtain the pedestrian optical flow embedding feature, and a plurality of pedestrian optical flow features can be obtained. The pedestrian optical flow embedding feature can be obtained by processing the pedestrian optical flow feature based on the first fully connected layer.

[0099] Among them, the pooling layer refers to a module for reducing the spatial dimension of the feature map, preserving key information and enhancing translation invariance. For example, the pooling layer includes maximum pooling and average pooling, etc. The pedestrian optical flow feature refers to an optical flow feature representation with the same size obtained after the pedestrian optical flow map is aligned by the pooling layer, still retaining the spatial structure. The first fully connected layer can be a fully connected (Fully Connected) layer for feature transformation, specifically used to map the input features to a new semantic space, introduce non-linear activation, and enhance the feature expression ability. The pedestrian optical flow embedding feature refers to a low-dimensional dense vector output after the pedestrian optical flow feature is processed by the first fully connected layer.

[0100] In this embodiment, fixed-size max-pooling or average-pooling operations (e.g., uniform pooling to 4x4 or 8x8) can be performed on all pedestrian optical flow maps to force the spatial dimensions to be aligned, generating pedestrian optical flow features of consistent size. Alternatively, adaptive pooling techniques (such as adaptive average pooling) can be used to automatically adjust the size and stride of the pooling kernel according to a preset target output size, without the need for manual parameter setting to map pedestrian optical flow maps of arbitrary size to pedestrian optical flow features of fixed dimensions. This operation can preserve the overall motion energy distribution while avoiding the loss of details that may be caused by pooling. Alternatively, a channel attention mechanism can be introduced before pooling to weight different channels of the pedestrian optical flow map, highlighting optical flow components (such as channels dominated by horizontal or vertical directions) that are more sensitive to motion modeling; and performing a pooling operation on the weighted optical flow image to obtain pedestrian optical flow features with more focused semantics. Through the attention-guided pooling strategy, key dynamic clues of pedestrians can be preserved during alignment, improving the expressiveness and robustness of the embedded features.

[0101] Further, continuing to refer to Figure 2 These aligned pedestrian optical flow features can be input into the first fully connected layer for nonlinear transformation and dimension compression, outputting pedestrian optical flow embedding features of uniform dimensions, extracting motion semantic features most valuable for trajectory prediction, and improving the accuracy of understanding pedestrian dynamic behavior.

[0102] For example, a pedestrian trajectory box is composed of fixed parameters such as center position and size, which can be directly input into a neural network for processing to obtain a bounding box embedding feature. In contrast, since different pedestrian trajectory boxes have different sizes, the number of elements in the cropped pedestrian optical flow map is different, so normalization is needed to ensure the consistency of the input data structure. The pedestrian optical flow maps of different sizes can be cropped to a uniform fixed dimension through a pooling layer (such as the region of interest pooling technique). The cropped pedestrian optical flow feature can be represented as: ; wherein, represents the pedestrian optical flow feature of the th pedestrian optical flow map at the th time point after normalization (i.e., alignment); represents a pooling layer. The specific implementation is as follows: the cropped pedestrian optical flow map is divided into equal-sized sub-optical flow regions, the original size is transformed from to , and the average value of the pixels in each sub-optical flow region is calculated to replace the overall pixel value of the sub-optical flow. After this operation, the size of the pedestrian optical flow map is uniform to , and the pedestrian optical flow feature , are pixel values of different sub-optical flow regions. Further, the pedestrian optical flow features are also embedded to ascend to a high-dimensional feature space to obtain pedestrian optical flow embedding features. The pedestrian optical flow embedding features can be expressed as: . denotes the pedestrian optical flow embedding features, are model parameters trained in the first fully connected layer.

[0103] S260, based on the vehicle state data, the bounding box embedding features and the pedestrian optical flow embedding features, determine the target motion features.

[0104] The bounding box embedding features can represent the macro motion patterns of the pedestrian (such as moving direction, speed change, trajectory curvature, etc.). The pedestrian optical flow embedding features can represent the micro dynamic details of the pedestrian (such as limb swing, gait rhythm, sudden turning, etc.).

[0105] In this embodiment, the vehicle state data, the bounding box embedding features and the pedestrian optical flow embedding features can be normalized to eliminate dimensional differences. Then, the three types of features are spliced in the dimension to form a joint feature vector. The joint feature vector is input into a fusion network composed of one or more fully connected layers to learn the interaction relationship between the modalities through nonlinear transformation, and output the target motion features. An attention mechanism can also be introduced to dynamically weight the three types of input features, for example, taking the vehicle state data as the query (Query), and calculating the correlation weights between it and the bounding box embedding features and the pedestrian optical flow embedding features, respectively, so as to adaptively emphasize the information source with more predictive value under the current vehicle motion state (for example, paying more attention to the trajectory trend when driving at high speed, and paying more attention to local actions when following at low speed). The weighted features are then integrated by the fusion network to generate the target motion features. This method can dynamically adjust the perception focus according to the driving scene, and improve the adaptability and robustness of the model under different working conditions. The bounding box embedding features and the pedestrian optical flow embedding features can also be preliminarily fused in the visual semantic level to form pedestrian dynamic features; then, the pedestrian dynamic features and the vehicle state data are secondarily fused in the motion level to obtain the target motion features. For example, the preliminary fusion can use a lightweight convolution or a Transformer module to fuse the spatio-temporal visual clues, and the second fusion introduces kinematic constraints (such as relative speed compensation) and integrates the vehicle state through a gating mechanism. The output target motion features not only retain the rich details of the pedestrian behavior, but also explicitly model the relative motion relationship between the host vehicle and the pedestrian.

[0106] The target motion feature is determined by fusing the vehicle state data, the bounding box embedding feature and the pedestrian optical flow embedding feature, realizing pedestrian trajectory detection from single observation to multi-dimensional collaborative perception. The target motion feature not only comprehensively integrates the macro trajectory trend and micro motion details of the pedestrian, but also explicitly incorporates the influence of the motion state of the host vehicle on the observation results, thereby effectively compensating for the perception deviation caused by relative motion, improving the prediction ability of the future behavior of the pedestrian, enabling the decision system to achieve a better balance between safe avoidance and efficient traffic, and enhancing the running safety and user experience of the vehicle in complex urban traffic scenarios.

[0107] In the embodiment, based on the vehicle state data, the bounding box embedding feature and the pedestrian optical flow embedding feature, the target motion feature is determined, including: based on the second fully connected layer, processing the pedestrian optical flow embedding feature according to a preset region size to obtain a fixed region optical flow embedding feature; respectively encoding the fixed region optical flow embedding feature, the bounding box embedding feature and the pedestrian optical flow embedding feature to obtain a fixed region optical flow enhanced feature, a bounding box enhanced feature and a pedestrian optical flow enhanced feature; and determining the target motion feature based on the fixed region optical flow enhanced feature, the bounding box enhanced feature, the pedestrian optical flow enhanced feature and the vehicle state data.

[0108] The second fully connected layer is used for nonlinear transformation and dimension adjustment of the pedestrian optical flow embedding feature. The preset region size can be a pre-defined unified spatial scale or feature dimension standard, used to indicate which features in certain regions of the pedestrian optical flow embedding feature are extracted. The fixed region optical flow embedding feature refers to the feature output by the third fully connected layer after processing the pedestrian optical flow embedding feature according to the preset region size. The fixed region optical flow enhanced feature / bounding box enhanced feature / pedestrian optical flow enhanced feature: respectively refers to the enhanced version of the feature obtained by independently encoding the fixed region optical flow embedding feature, the bounding box embedding feature and the original pedestrian optical flow embedding feature, each of which retains and enhances the semantic information of a specific modality. The target motion feature is a comprehensive feature generated by fusing multiple source enhanced features and vehicle state data, used to accurately represent the motion state and future trend of the pedestrian under the perspective of the host vehicle.

[0109] In this embodiment, the pedestrian optical flow embedding feature can be input into the second fully connected layer, and the processed optical flow embedding feature is output. Then, the fixed region optical flow embedding feature is extracted from the processed optical flow embedding feature according to the preset region size. Alternatively, the embedding feature of the fixed region can be extracted from the pedestrian optical flow embedding feature according to the preset region size. Then, the embedding feature of the fixed region is input into the second fully connected layer, and the fixed region optical flow embedding feature is output. When the feature extraction is performed according to the preset region size, the center position of the pedestrian optical flow embedding feature can be taken as the midpoint, and the features in the preset region size (such as a rectangular pixel size of 180x180) around the midpoint in the pedestrian optical flow embedding feature are taken as the embedding features of the fixed region. It should be noted that the pedestrian optical flow map can also be processed according to the preset region size based on the second fully connected layer to obtain the fixed region optical flow embedding feature. Further, the fixed region optical flow embedding feature, the bounding box embedding feature, and the pedestrian optical flow embedding feature are respectively sent into three independent encoding modules (such as a fully connected network, a Transformer, an MLP with a residual connection, or a gated recurrent unit) for encoding to obtain the fixed region optical flow enhanced feature, the bounding box enhanced feature, and the pedestrian optical flow enhanced feature. Further, the fixed region optical flow enhanced feature, the bounding box enhanced feature, the pedestrian optical flow enhanced feature, and the vehicle state data can be spliced in the feature dimension to input into the fusion network to output the target motion feature. Alternatively, the fixed region optical flow enhanced feature, the bounding box enhanced feature, the pedestrian optical flow enhanced feature, and the vehicle state data are integrated into the target motion feature through a gating fusion mechanism or a cross-attention mechanism (such as adjusting the weight of each feature based on the vehicle state).

[0110] For example, the fixed part embedding feature around the center of the pedestrian optical flow embedding feature can be framed. This part of the fixed size optical flow embedding feature can reflect the motion of the host vehicle to a certain extent, and the motion of the host vehicle will cause the relative motion of the pedestrian, which will bring negative disturbance to the trajectory prediction.

[0111] The preset region size can be set to a rectangular pixel size of 180x180, and the feature embedding is performed through the second fully connected layer to obtain the fixed region optical flow embedding feature. The fixed region optical flow embedding feature can be represented as: ; The fixed region optical flow embedding feature is represented as The fixed rectangular region optical flow in the pedestrian optical flow embedding feature is represented as The second fully connected layer The trained parameters in the second fully connected layer.

[0112] The embedded features are effective for expressing the time-dependent ability, and to further extract the time dependence of the detection box embedded features and the optical flow embedded features, a Gated Recurrent Unit (GRU) can be used to encode the fixed region optical flow embedded features, the detection box embedded features and the pedestrian optical flow embedded features respectively to obtain the fixed region optical flow enhanced features, the detection box enhanced features and the pedestrian optical flow enhanced features.

[0113] The detection box enhanced features can be expressed as: ;

[0114] The pedestrian optical flow enhanced features can be expressed as: ;

[0115] The fixed region optical flow enhanced features can be expressed as: ;

[0116] wherein, is the hidden state of the Gated Recurrent Unit at the previous time step obtained by encoding; is the Gated Recurrent Unit, is the parameter learned by the Gated Recurrent Unit.

[0117] Referring to Figure 2 , each Gated Recurrent Unit can be composed of three GRUs connected in series. For example, the fixed region optical flow embedded features are input into the first GRU to output the first embedded features; the first embedded features and the fixed region optical flow embedded features are input into the second GRU to output the second embedded features; and the second embedded features and the fixed region optical flow embedded features are input into the third GRU to output the detection box enhanced features.

[0118] The technical solution provided by the embodiment introduces a second fully connected layer to extract the fixed region optical flow embedded features from the pedestrian optical flow embedded features, and independently or cooperatively enhances the fixed region optical flow embedded features, the detection box embedded features and the pedestrian optical flow embedded features, thereby realizing the fine feature extraction of the multi-granularity and multi-view pedestrian dynamic information. In combination with the fusion strategy of the vehicle state data, the influence of the host vehicle motion on the pedestrian observation is further explicitly modeled, the perception deviation caused by the relative motion is effectively suppressed, and the extraction accuracy of the pedestrian target motion features is ensured.

[0119] In this embodiment, based on the fixed region optical flow enhanced feature, the bounding box enhanced feature, the pedestrian optical flow enhanced feature and the vehicle state data, the target motion feature is determined, including: determining the fusion attention feature based on the bounding box enhanced feature and the pedestrian optical flow enhanced feature; determining the host vehicle motion feature based on the fixed region optical flow enhanced feature and the vehicle state data; fusing the fusion attention feature and the host vehicle motion feature to obtain the target motion feature.

[0120] The fusion attention feature refers to a joint representation generated by modeling the bounding box enhanced feature and the pedestrian optical flow enhanced feature through an attention mechanism, which can adaptively highlight the motion features more discriminative for the current scene. The host vehicle motion feature refers to an intermediate feature generated by fusing the fixed region optical flow enhanced feature and the vehicle state data, which explicitly embeds the environmental dynamic perception result from the perspective of the host vehicle. The target motion feature refers to a comprehensive feature obtained by deeply fusing the fusion attention feature and the host vehicle motion feature, which comprehensively depicts the motion state and future behavior trend of the pedestrian in the host vehicle coordinate system.

[0121] In this embodiment, the trajectory box is closely related to its corresponding optical flow map, and respectively describes the position and motion trend of the pedestrian. Continuing to refer to Figure 2 , the bounding box enhanced feature and the pedestrian optical flow enhanced feature can be fused through a cross-modal cross-attention mechanism to obtain the fusion attention feature. For example, the bounding box enhanced feature and the pedestrian optical flow enhanced feature are input into a cross-attention module (for example, taking the bounding box enhanced feature as the query and the pedestrian optical flow enhanced feature as the key value), the correlation weight between the two is calculated, and the fusion attention feature is generated. The vehicle state data is encoded to obtain the vehicle state feature; the vehicle state feature and the fixed region optical flow enhanced feature are spliced based on the full connection layer to obtain the spliced feature; the spliced feature can be taken as the host vehicle motion feature, or the spliced feature can be input into a conditional fusion network (such as a full connection layer with a gating mechanism) to generate the host vehicle motion feature. The fusion attention feature and the host vehicle motion feature are spliced again or fused through a residual connection to output the target motion feature.

[0122] For example, the bounding box enhanced feature can be represented as: ; represents the encoded feature in the mthdimension of the bounding box enhanced feature corresponding to the nthpedestrian at the nthmoment. ; represents the encoded feature in the mthdimension of the pedestrian optical flow enhanced feature corresponding to the nthpedestrian at the nthmoment. ​​​​​​

[0123] The fusion attention feature can be represented as: is represented as an activation function. represents the fusion attention feature; the query matrix is calculated by the bounding box enhanced feature ; the key matrix is calculated by the pedestrian optical flow enhanced feature and the value matrix represents the feature dimension. is represented as a transpose symbol. Both are learned matrix parameters. The bounding box enhanced feature belongs to kinematics data, and the pedestrian optical flow enhanced feature belongs to image data. Cross-attention fusion of the two helps to fully stimulate the complementary properties of the two, and improves the pedestrian trajectory prediction accuracy.

[0124] The vehicle itself is in a state of motion, and even a stationary pedestrian will have relative motion with the host vehicle, resulting in a timing change in its bounding box. When the vehicle is running, it may mistakenly think that the pedestrian is moving, and thus make an incorrect prediction of the pedestrian trajectory. To solve this problem, the technical solution provided in the embodiment splices the host vehicle displacement feature and the heading angle feature in the vehicle state data with the fixed region optical flow enhanced feature to obtain the host vehicle motion trend feature (i.e., the host vehicle motion feature), which compensates for the motion disturbance.

[0125] The host vehicle motion feature can be represented as: is the host vehicle motion feature, is a feature splicing operation; and are the encoding features of the host vehicle displacement and heading angle, respectively. The vehicle state data can be collected from the vehicle CAN bus, and its encoding features can be calculated by a gated recurrent unit (GRU) network.

[0126] Further, the pedestrian cross-modal cross-feature (i.e., the fusion attention feature) and the host vehicle motion feature are further spliced to form a target motion feature for pedestrian trajectory prediction. The target motion feature can be represented as: is the target motion feature.

[0127] S270, based on the target motion feature, determining the predicted position of the pedestrian.

[0128] The GRU model can be used to decode the target motion feature, and the correlation between each predicted trajectory point is reflected by step-by-step decoding, and the predicted position of the pedestrian is output. ​​​​​​​​

[0129] The predicted position of the pedestrian may be expressed as: are parameters learned by the GRU model.

[0130] On the basis of the above scheme, a pedestrian trajectory prediction model applied to a vehicle can be pre-trained to obtain multiple frames of to-be-processed images of the surrounding environment of the vehicle during driving of the vehicle; the multiple frames of to-be-processed images are input into the pedestrian trajectory prediction model, and the pedestrian trajectory prediction model can perform steps S120 to S150 to output a predicted position of the pedestrian. A loss function for training the pedestrian trajectory prediction model can be a mean squared error (MSE) function, and the training target is to minimize the deviation between the predicted trajectory and the real trajectory of the pedestrian.

[0131] The mean squared error function can be expressed as:

[0132] wherein, represents a pedestrian trajectory prediction value output by the pedestrian trajectory prediction model; represents a real value of a pedestrian trajectory labeled in a training sample, is the number of pedestrians in the training sample in a single prediction process.

[0133] After training is completed, a pedestrian trajectory frame can be extracted based on the pedestrian trajectory prediction model and a corresponding pedestrian optical flow map can be calculated, and then a target motion feature of the pedestrian can be determined based on vehicle state data, the pedestrian trajectory frame, and the pedestrian optical flow map, and accurate prediction of the pedestrian trajectory can be realized based on the target motion feature.

[0134] The technical scheme provided in this embodiment extracts a detection frame embedding feature by performing deep processing on a pedestrian trajectory frame, analyzes an optical flow map reflecting the dynamic behavior of the pedestrian to obtain a pedestrian optical flow embedding feature, and then combines the running state data of the vehicle itself to accurately determine the target motion feature of the pedestrian by comprehensively integrating three aspects of information. This multi-level and multi-modal information fusion method not only improves the detail and accuracy of the description of the motion state of the pedestrian, but also captures the instantaneous changes and long-term trends of the behavior of the pedestrian, ensuring the accuracy of the prediction of the future position of the pedestrian. At the same time, this method can effectively distinguish the actual movement of the pedestrian from the relative movement difference caused by the movement of the vehicle itself, avoiding the misjudgment risk brought by the traditional reliance on a single data source, enhancing the safety of driving, reducing potential traffic accident risks, and optimizing the driving path planning of the vehicle through more accurate prediction of the behavior of the pedestrian, improving the efficiency and smoothness of the overall traffic system.

[0135] Figure 4 ​​Fig. 1 is a structural schematic diagram of a pedestrian trajectory prediction device applied to a vehicle according to an embodiment of the present application. As shown in the figure, the device comprises an image acquisition module 310, a pedestrian trajectory box determination module 320, a pedestrian optical flow map determination module 330, a target motion feature determination module 340, and a predicted position determination module 350. Figure 4

[0136] The image acquisition module 310 is configured to acquire a plurality of frames of to-be-processed images of the surrounding environment of the vehicle during the driving of the vehicle. The pedestrian trajectory box determination module 320 is configured to detect a detection box of at least one pedestrian in the plurality of frames of to-be-processed images, and associate the detection boxes of the same pedestrian to obtain a pedestrian trajectory box. The pedestrian optical flow map determination module 330 is configured to determine a pedestrian optical flow map based on adjacent frames of to-be-processed images and the pedestrian trajectory box. The target motion feature determination module 340 is configured to acquire vehicle state data of the vehicle, and determine a target motion feature of the pedestrian based on the vehicle state data, the pedestrian trajectory box, and the pedestrian optical flow map. The predicted position determination module 350 is configured to determine a predicted position of the pedestrian based on the target motion feature.

[0137] The technical scheme of the present embodiment comprises the following steps: acquiring a plurality of frames of to-be-processed images of the surrounding environment of the vehicle during the driving of the vehicle, detecting a detection box of at least one pedestrian in the plurality of frames of to-be-processed images, and associating the detection boxes of the same pedestrian to obtain a pedestrian trajectory box; determining a pedestrian optical flow map based on adjacent frames of to-be-processed images and the pedestrian trajectory box; acquiring vehicle state data of the vehicle, and determining a target motion feature of the pedestrian based on the vehicle state data, the pedestrian trajectory box, and the pedestrian optical flow map; and determining a predicted position of the pedestrian based on the target motion feature. The present embodiment solves the problem in the prior art that the prediction method based only on historical trajectory coordinates cannot distinguish between the real motion of the pedestrian and the observation deviation caused by relative motion, which easily leads to inaccurate prediction, thereby resulting in low driving safety and low traffic efficiency, and achieves the technical effects of acquiring a plurality of frames of to-be-processed images of the surrounding environment of the vehicle during the driving of the vehicle, detecting and associating the detection boxes of the same pedestrian to construct a pedestrian trajectory box, then combining adjacent frames of to-be-processed images and the pedestrian trajectory box to generate a pedestrian optical flow map focusing on the pedestrian area, and further fusing vehicle state data, the pedestrian trajectory box, and the pedestrian optical flow map to comprehensively model the macro motion trend, the micro dynamic details, and the influence of the relative motion of the host vehicle of the pedestrian, thereby accurately extracting the target motion feature of the pedestrian and predicting its future position, so that the real motion of the pedestrian and the observation deviation caused by the motion of the host vehicle can be clearly distinguished, and the accuracy and robustness of trajectory prediction are improved. Thus, the behavior of the pedestrian can be more reliably predicted in a complex urban traffic scene, and the technical effect of optimizing traffic efficiency while ensuring pedestrian safety by avoiding overly conservative or aggressive decisions is achieved.

[0138] ​On the basis of the above-mentioned device, optionally, the pedestrian optical flow map determination module 330 comprises:

[0139] A pixel optical flow map determination unit is configured to determine a pixel optical flow map based on the to-be-processed image of the adjacent frame.

[0140] A pedestrian optical flow map determination unit is configured to crop the pixel optical flow map based on the pedestrian trajectory frame to obtain a pedestrian optical flow map.

[0141] On the basis of the above-mentioned device, optionally, the target motion feature determination module 340 comprises:

[0142] A pedestrian optical flow embedding feature determination unit is configured to process the pedestrian trajectory frame to obtain a detection frame embedding feature, and process the pedestrian optical flow map to obtain a pedestrian optical flow embedding feature.

[0143] A target motion feature determination unit is configured to determine a target motion feature based on the vehicle state data, the detection frame embedding feature, and the pedestrian optical flow embedding feature.

[0144] On the basis of the above-mentioned device, optionally, the pedestrian optical flow embedding feature determination unit comprises:

[0145] A pedestrian optical flow feature determination unit is configured to align all the pedestrian optical flow maps based on a pooling layer to obtain a plurality of pedestrian optical flow features.

[0146] A pedestrian optical flow embedding feature determination subunit is configured to process the pedestrian optical flow features based on a first fully connected layer to obtain a pedestrian optical flow embedding feature.

[0147] On the basis of the above-mentioned device, optionally, the target motion feature determination unit comprises:

[0148] A fixed region optical flow embedding feature determination unit is configured to process the pedestrian optical flow embedding feature according to a preset region size based on a second fully connected layer to obtain a fixed region optical flow embedding feature.

[0149] An enhanced feature determination unit is configured to respectively encode the fixed region optical flow embedding feature, the detection frame embedding feature, and the pedestrian optical flow embedding feature to obtain a fixed region optical flow enhanced feature, a detection frame enhanced feature, and a pedestrian optical flow enhanced feature.

[0150] A target motion feature determination subunit is configured to determine a target motion feature based on the fixed region optical flow enhanced feature, the detection frame enhanced feature, the pedestrian optical flow enhanced feature, and the vehicle state data.

[0151] On the basis of the above-mentioned device, optionally, the target motion feature determination subunit comprises:

[0152] The fusion attention feature determination unit is configured to determine a fusion attention feature based on the detection frame enhanced feature and the pedestrian optical flow enhanced feature.

[0153] The host vehicle motion feature determination unit is configured to determine a host vehicle motion feature based on the fixed area optical flow enhanced feature and the vehicle state data.

[0154] The fusion unit is configured to fuse the fusion attention feature and the host vehicle motion feature to obtain a target motion feature.

[0155] On the basis of the above device, optionally, the device further comprises:

[0156] The detection frame embedded feature update unit is configured to process the detection frame embedded feature based on a full-scale network to obtain an updated detection frame embedded feature.

[0157] The pedestrian trajectory prediction device for a vehicle provided in the embodiments of the present application can execute the pedestrian trajectory prediction method for a vehicle provided in any of the embodiments of the present application, and has the function modules and beneficial effects corresponding to the execution method.

[0158] Figure 5 is a structural schematic diagram of an electronic device for implementing the pedestrian trajectory prediction method for a vehicle in the embodiments of the present application. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown in the figure, their connections, and relationships, and their functions, are merely examples, and are not intended to limit the implementations of the present application described and / or claimed herein.

[0159] As shown in Figure 5 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory 12, a random access memory 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory 12 or loaded from the storage unit 18 into the random access memory 13. In the random access memory 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the read-only memory 12, and the random access memory 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0160] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0161] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the pedestrian trajectory prediction method applied to a vehicle.

[0162] In some embodiments, the pedestrian trajectory prediction method applied to a vehicle can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the read-only memory 12 and / or the communication unit 19. When the computer program is loaded onto the random access memory 13 and executed by the processor 11, one or more steps of the pedestrian trajectory prediction method applied to a vehicle described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the pedestrian trajectory prediction method applied to a vehicle by any other appropriate means, such as by means of firmware.

[0163] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0164] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, enables the functions / acts specified in the flowcharts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0165] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0166] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0167] The systems and techniques described herein can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), blockchain network, and the Internet.

[0168] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0169] In particular, the processes described above with reference to the flow charts can be implemented in computer software programs according to embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from a network by the communication unit 19, or installed from the storage unit 18, or installed from the read-only memory 12. When the computer program is executed by the processor 11, the above-mentioned functions defined in the methods of embodiments of the present application are performed.

[0170] Embodiments of the present application also provide a computer program product comprising a computer program which, when executed by a processor, implements the pedestrian trajectory prediction method applied to a vehicle as provided by any of the embodiments of the present application.

[0171] The computer program code can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce the computer implemented process such that the

[0172] It should be understood that the various forms of flow shown in the figures are illustrative examples of implementing the steps of the application. Several steps have been described as being performed by a single device. It will be understood that these steps can be performed by a single device or multiple devices. It will also be understood that the steps can be performed in a different order than that shown in the figures. It will also be understood that the steps can be performed concurrently or sequentially. It will also be understood that the steps can be performed by different entities. It will also be understood that the steps can be performed by a combination of hardware and software. It will also be understood that the steps can be performed by a combination of one or more devices and one or more computers.

[0173] The specific embodiments have been shown and described for the purposes of illustrating the physiological principles of the application. It will be understood that various modifications, combinations, sub-combinations, and alternatives can be made to the specific embodiments without departing from the spirit and principles of the application. Any modifications, equivalent substitutions, improvements, combinations or the like not described above are also encompassed within the scope of the application.

Claims

1. A method for predicting pedestrian trajectories applied to vehicles, characterized in that, include: During vehicle operation, acquire multiple frames of images of the surrounding environment of the vehicle to be processed; In the multiple frames of the image to be processed, at least one detection box of a pedestrian is detected, and the detection boxes of the same pedestrian are associated to obtain the pedestrian trajectory box; Based on the image to be processed in adjacent frames and the pedestrian trajectory bounding box, determine the pedestrian optical flow map; The vehicle status data of the vehicle is acquired, and the target motion features of the pedestrian are determined based on the vehicle status data, the pedestrian trajectory box, and the pedestrian optical flow map. Based on the target's motion characteristics, the predicted location of the pedestrian is determined.

2. The method according to claim 1, characterized in that, The step of determining the pedestrian optical flow map based on the image to be processed in adjacent frames and the pedestrian trajectory bounding box includes: Based on the images to be processed in adjacent frames, determine the pixel optical flow map; Based on the pedestrian trajectory bounding box, the pixel optical flow map is cropped to obtain the pedestrian optical flow map.

3. The method according to claim 1, characterized in that, The step of determining the target motion features of the pedestrian based on the vehicle state data, the pedestrian trajectory bounding box, and the pedestrian optical flow map includes: The pedestrian trajectory bounding box is processed to obtain the detection box embedding feature, and the pedestrian optical flow map is processed to obtain the pedestrian optical flow embedding feature; Based on the vehicle state data, the detection box embedding features, and the pedestrian optical flow embedding features, the target motion features are determined.

4. The method according to claim 3, characterized in that, The process of processing the pedestrian optical flow map to obtain pedestrian optical flow embedding features includes: Aligning all the pedestrian optical flow maps based on the pooling layer yields multiple pedestrian optical flow features; The pedestrian optical flow features are processed based on the first fully connected layer to obtain pedestrian optical flow embedding features.

5. The method according to claim 3, characterized in that, The determination of target motion features based on the vehicle state data, the detection box embedding features, and the pedestrian optical flow embedding features includes: Based on the second fully connected layer, the pedestrian optical flow embedding features are processed according to the preset region size to obtain fixed region optical flow embedding features; The fixed region optical flow embedding feature, the detection box embedding feature, and the pedestrian optical flow embedding feature are encoded respectively to obtain the fixed region optical flow enhancement feature, the detection box enhancement feature, and the pedestrian optical flow enhancement feature; Based on the optical flow enhancement features of the fixed region, the detection box enhancement features, the pedestrian optical flow enhancement features, and the vehicle state data, the target motion features are determined.

6. The method according to claim 5, characterized in that, The determination of target motion features based on the fixed region optical flow enhancement features, the detection box enhancement features, the pedestrian optical flow enhancement features, and the vehicle state data includes: Based on the enhanced features of the detection box and the enhanced features of pedestrian optical flow, the fused attention features are determined; Based on the optical flow enhancement features of the fixed region and the vehicle state data, the motion characteristics of the main vehicle are determined; The fused attention features and the main vehicle motion features are fused to obtain the target motion features.

7. The method according to claim 3 or 5, characterized in that, The method further includes: The detection box embedding features are processed based on a full-scale network to obtain updated detection box embedding features.

8. A pedestrian trajectory prediction device for vehicles, characterized in that, include: The image acquisition module is used to acquire multiple frames of images of the surrounding environment of the vehicle during vehicle operation. The pedestrian trajectory box determination module is used to detect at least one pedestrian's detection box in multiple frames of the image to be processed, and associate the detection boxes of the same pedestrian to obtain the pedestrian trajectory box. The pedestrian optical flow map determination module is used to determine the pedestrian optical flow map based on the image to be processed in adjacent frames and the pedestrian trajectory box; The target motion feature determination module is used to acquire the vehicle status data of the vehicle, and determine the target motion features of the pedestrian based on the vehicle status data, the pedestrian trajectory box and the pedestrian optical flow map; The predicted location determination module is used to determine the predicted location of the pedestrian based on the target motion characteristics.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to said at least one processor; wherein, The memory stores a computer program executable by the at least one processor, which enables the at least one processor to perform the pedestrian trajectory prediction method for vehicles as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the pedestrian trajectory prediction method for vehicles as described in any one of claims 1-7.

Images