A method, model training method and device for predicting surrounding multi-target trajectories based on monocular visual motion estimation

Abstract

The present invention discloses a method, a model training method and an apparatus for predicting the trajectories of multiple targets in the surrounding area based on monocular visual motion estimation. The vehicle motion estimation and prediction part establishes a complete vehicle visual odometer to achieve independent modeling of the vehicle motion. Secondly, the proposed target trajectory initialization method normalizes the target bounding box coordinates at different times to the current moment based on the established visual odometer and depth map information. The vehicle motion estimation and prediction model completes the decoupling of the target motion and the vehicle motion. Finally, the normalized trajectory coordinates are used in the future trajectory prediction part to complete the prediction of the target's future trajectory under the current camera perspective of the vehicle. The data required for training the model at each stage include the video sequence captured by the on-board camera and the position information and identity information of each target in the sequence.

Description

Technical Field

[0001] The present invention belongs to the field of vehicle intelligent driving, and in particular relates to a method for predicting the trajectories of multiple targets in the surrounding area based on monocular visual motion estimation, a model training method and a device. Background Art

[0002] Smart cars have become a strategic direction for the development of the global automotive industry and a historic opportunity for my country to become a leading automotive power. They are also an innovative and integrated vehicle for strategic emerging industries such as next-generation information technology, high-end equipment manufacturing, new materials, and new energy. Environmental perception is a crucial foundation for achieving automotive intelligence and a bottleneck in achieving high-level autonomous driving. Trajectory prediction, a high-level behavioral perception technology based on object detection and tracking, provides effective information for smart car decision-making and control, improving the safety and rationality of driving trajectory planning. Effective trajectory prediction is particularly important for improving driving safety in complex traffic scenarios such as intersections and highway merging ramps.

[0003] Existing trajectory prediction methods need to be based on target detection, tracking, positioning, and map matching to complete the modeling of the relative motion relationship with surrounding vehicles within a certain range. This places extremely high demands on the vehicle's environmental perception system and has a slow inference speed, which to a certain extent restricts the practical application of such methods.

[0004] To address the shortcomings of such trajectory prediction methods, this paper develops a multi-target trajectory prediction technology based on a low-cost monocular perception system for intelligent vehicles. Under this perspective, the position changes of the target bounding box captured by the onboard camera are not only due to the target's motion, but are also closely related to the vehicle's motion. Decoupling target motion from the vehicle's motion presents a new challenge, the core of which lies in the depth recognition of the target and the pose estimation of the vehicle under monocular vision. Summary of the Invention

[0005] The trajectory prediction method proposed in the present invention includes three parts: ego vehicle motion estimation and prediction, target trajectory initialization and future trajectory prediction. First, the ego vehicle motion estimation and prediction part establishes a complete vehicle visual odometry to realize independent modeling of the ego vehicle motion. Secondly, the proposed target trajectory initialization normalizes the target bounding box coordinates at different times to the current moment based on the established visual odometry and depth map information. The ego vehicle motion estimation and prediction model completes the decoupling of target motion and ego vehicle motion. Finally, the normalized trajectory coordinates are used in the future trajectory prediction part to complete the prediction of the target's future trajectory under the current camera perspective of the ego vehicle. The data required for training the model at each stage include the video sequence captured by the on-board camera and the position information and identity information of each target in the sequence. The models used in the above method include the ego vehicle motion estimation network, the ego vehicle motion prediction network and the multi-target trajectory prediction network.

[0006] Beneficial effects of the present invention:

[0007] (1) The present invention proposes a self-vehicle motion estimation network. The network is a self-supervised training model composed of a depth estimation network and a pose estimation network. The model calculates the visual odometry of the self-vehicle motion by inputting a complete sequence of past and future images of the self-vehicle. Both networks use a codec structure, where the encoder can be a convolutional neural network model with any number of layers, extracting low-dimensional large-scale image features and converting them into high-dimensional small-scale feature maps. The network can effectively solve the deficiency of existing prediction methods that do not fully consider the motion state of the self-vehicle, and convert the pixel coordinates of surrounding targets associated with the self-vehicle motion into a normalized static camera coordinate system.

[0008] (2) The present invention proposes a self-vehicle motion prediction network. When the proposed self-vehicle motion estimation network performs inference in an actual scene, it can only obtain image information at historical moments through the on-board camera. Therefore, at this time, the model can only calculate the pose transformation matrix between the camera coordinate systems during the historical motion of the self-vehicle, but cannot directly calculate the pose transformation matrix during the future motion of the self-vehicle. In order to solve the above problem, the proposed method adds a new self-vehicle motion prediction network. The network introduces a codec model composed of a long short-term memory network (LSTM) to predict the future pose changes of the self-vehicle. The network input only includes the pose transformation parameters between the historical frames output by the self-vehicle estimation network. The self-vehicle motion prediction network is combined with the self-vehicle motion estimation network to form an interactive calculation model, which realizes the calculation of the complete visual odometry of the self-vehicle.

[0009] (3) The present invention proposes a multi-target trajectory prediction network. The trajectory prediction network is also composed of a codec structure. The encoder is divided into two parts. The first part is a motion feature encoder, and the model used is a long short-term memory network (LSTM), which is used to extract the normalized historical target bounding box information; the second part is a regional feature encoder, which encodes the pixel block of the target at the current moment and the pixel block sequence of the position in the past image. The network used by the encoder is a convolutional LSTM (ConvLSTM), which is a neural network that combines convolution operations with the gating mechanism of LSTM. While using convolution to extract image features, the network also controls the input ratio of the current image feature information and the retention ratio of the historical image feature information. The trajectory prediction network proposed by the present invention establishes a network model for calculating the mapping relationship between the target image and the bounding box position sequence and the future trajectory. The future trajectory predicted by this model is close to the true trajectory with extremely small pixel error.

[0010] (4) The present invention proposes a multi-stage model training method, provides the necessary hyperparameters for the recommended neural network training, as well as the appropriate input and output dimensions and number of layers for each neural network layer, thereby improving the effects of motion estimation and trajectory prediction.

[0011] (5) This paper proposes a method for predicting the trajectory of multiple targets around a vehicle using a low-cost environment perception device (monocular camera). Driven by image and target identity position data, this method can accurately predict the trajectory of the future bounding boxes of multiple targets around the vehicle, with a minimum prediction error of 10 pixels. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 The pose transformation of the ego-vehicle between the t-th frame image and the t′-th frame image

[0013] Figure 2 Overall structure of the egocar motion estimation network

[0014] Figure 3 Overall structure of the ego vehicle motion prediction network

[0015] Figure 4 Overall structure of multi-target trajectory prediction network

[0016] Figure 5 Typical test results under the KITTI Tracking dataset DETAILED DESCRIPTION

[0017] The present invention will be further described below with reference to the accompanying drawings.

[0018] Step 1: Build the ego vehicle visual odometry

[0019] The present invention establishes a visual odometer of the vehicle's motion process through the video captured by the vehicle-mounted camera, and completes the independent modeling of the vehicle's motion. The first step in decoupling the vehicle's motion and the motion of surrounding targets is to establish a visual odometer of the vehicle's motion process through the video captured by the vehicle-mounted camera, and complete the independent modeling of the vehicle's motion. The video is decomposed into an image sequence consisting of multiple frames, and the targets captured in each frame are in the camera coordinate system of the vehicle at the current moment. During the vehicle's driving process, changes in the vehicle's motion direction and position will cause continuous transformation of the camera coordinate system, that is, vehicle posture transformation. As Figure 1 As shown in Figure 1, the purpose of ego vehicle motion estimation is to calculate the pose transformation between the historical frame th,…,t-1 and the current frame t, and to predict the pose transformation between the future frame t+1,…,t+f and t. The inter-frame transformation process can be regarded as the rotation of the camera coordinate system of the two frames around the axis a by θ. This process can be expressed by the transformation matrix T t→t′ express:

[0020]

[0021] Where: is a number field, the superscript 4x4 indicates the dimension; R∈SO(3) is the rotation matrix, and is the offset of the origin of the camera coordinate axis. SE(3) and SO(3) are groups that satisfy the properties of Lie groups, where SE(3) is a special Euclidean group and SO(3) is a special orthogonal group.

[0022] Step 2: Build a vehicle motion estimation network

[0023] The present invention uses a self-supervised training model composed of a depth estimation network and a posture estimation network to complete the vehicle motion estimation. The network structure is as follows: Figure 2 First, the depth estimation network is used to output the current frame image I t Depth The transformation matrix T predicted by the pose estimation network t′→t And the intrinsic parameter matrix of the vehicle camera The image I at time t′ can be t′ All pixels in are projected to the corresponding positions in the pixel coordinate system at time t, thereby reconstructing the image I at time t t′→t :

[0024] I t′→t =proj(I t )

[0025] proj:p t =KT t′→t Dt K -1 p t′

[0026] Where: proj is the reprojection operation, p t and p t′ are the coordinates of each pixel in the image at time t and t′ respectively.

[0027] Specifically, the depth estimation network is a codec structure. Enc Input is the current frame image I t , extract the image feature information F through the ResNet-18 structure t 。 t Input to the depth decoder D Dec In, D Dec The depth D of each pixel position in the current frame image will be calculated. The value range of D is (0.1, 100), 0.1 is a depth unit, and the upper limit of the depth value is 100. The process is expressed as:

[0028] F t =D Enc (I t )

[0029] D t =D Dec (F t )

[0030] The pose estimation network has the same encoder-decoder structure, and the pose encoder P Enc With D Enc The same ResNet-18 network structure is used as the feature extractor. Enc The difference is that P Enc The input is an image sequence S consisting of images at time t, t+1, and t-1 hist ={I t-1 , I t}or{I t , I t+1}. First, S is connected through cascade operation. hist The channel dimension is merged to obtain the channel dimension that matches P Enc 6-channel image tensor of input dimension P Enc The model output is a high-dimensional feature map after 32 times downsampling Pose decoder P Dec It is composed of multiple layers of convolution and activation function ReLU stacked together. The decoder outputs 6 channels, representing the axis angle φ t→t′ ∈so(3), and offset Decomposition φ t→t′Get the rotation axis a and rotation angle θ, where φ i The element representing the i-th position of the axis angle:

[0031]

[0032]

[0033]

[0034] According to the Rodriguez formula, the exponential mapping between so(3) and SO(3) is calculated, and then combined with the offset d to obtain the pose transformation matrix T:

[0035] T t+1→t =Combine(R, d)

[0036] R = exp(φ^) = exp(θa^)

[0037] exp(θa^)=cosθE+(1+cosθ)aa T +sinθa^

[0038] Where: ^ is the antisymmetric symbol. The final posture transformation matrix T represents I t+1 The camera coordinate system to I t The transformation between the camera coordinate system.

[0039] In the actual model training process, the image sequence from the historical time t-h+1 to the future time t+f-1 will be input into multiple pose estimation codecs with adjacent frames for calculation, and finally the complete visual odometry VO of the sequence is obtained. t-h+1→t-h ,...,T t+f→t+f-1}.

[0040] Step 3: Build a vehicle motion prediction network

[0041] like Figure 3 As shown, the vehicle motion prediction network P pred It consists of an LSTM encoder-decoder structure, where the input F of the LSTM encoder is emb P Dec The output historical moment part of φ, d sequence Seq φ，d ={(φ t-h , d t-h ),...,(φ t-1 , d t-1 )}:

[0042] F emb =LeakyReLU(f1(Seq φ，d ))

[0043] O t，(φ，d) , H ht，(φ，d) =LSTM([H t-1，(φ，d) , F emb ],W enc )

[0044] Where: f1 is a linear layer, which acts as an embedding layer to map the low-dimensional features of the input sequence to high dimensions. LeakyReLU is the activation function, with α = 0.1. This part takes the hidden state H of the last time step t′(θ，d) As the feature vector summarizing the historical poses, W enc is the weight parameter of the LSTM encoder.

[0045] The LSTM decoder uses a non-autoregressive (NAR) method to t′(φ，d) Decode and output the predicted future vehicle motion:

[0046] O t , H t =LSTM([H t-1 , H t′(φ，d) ],W dec )

[0047] φ pred , d pred =f out (O t )

[0048] Where: f out It is a linear layer, the output of which is the predicted rotation axis and origin offset φ for each future time step pred , d pred ={(φ t+1 , d t+1 ),...,(φ t+f , d t+f )}, and finally use the formula φ pred , d pred Converted into pose transformation matrix T pred , W dec is the weight parameter of the LSTM decoder.

[0049] Step 4: Target trajectory normalization

[0050] According to the visual odometry output by the ego-vehicle motion estimation network, the pose transformation matrix set {T t→t′, t′∈[th, t+f]}. The perspective of the final predicted target future trajectory will be the perspective of the camera at time t, so when the target history bounding box B at any time t′ is t′ Before inputting into the prediction model, the bounding box B t′ The pixel coordinates of the midpoint (p tl , p br ) t′ Convert to pixel coordinates at time t to form a new bounding box Bounding box B at time t′ t′ The coordinates of any corner point p in t′ Convert to coordinates at time t It can be expressed as:

[0051]

[0052]

[0053] Where: d(B t′ ) is the bounding box B t′ The average depth within the area. h B , w B For B t′ The height and width, D ij is the depth of a single pixel position in the depth map, i∈[p tl，x , p br，x ],j∈[p tl，y , p br，y ], where x, y represent the x and y axes respectively. I has been predicted in the vehicle motion estimation network. t′ to I t Therefore, when converting the camera coordinate system of any historical frame i to the camera coordinate system at time t, the transformation matrix can be obtained by matrix multiplication: T t-i→t =T t-i→t-1 T t-1→t .

[0054] Step 5: Predict future trajectory

[0055] After the target trajectory normalization process, the target bounding box at the historical time t′ has been converted to the current time t, and the bounding box is obtained. Therefore, when predicting the future trajectory, the target's historical motion process will all be carried out within the perspective of the static camera, and the predicted future trajectory will also be within this perspective. In order to better characterize the target's historical motion process, this network will calculate more feature information through the historical bounding box sequence. In addition, this network also pays attention to the pixel transformation process in the area where the target is located in the image sequence, and uses cropping to obtain image information in the corresponding area of ​​the target. Finally, the mapping relationship between the target's historical feature sequence and the future target's kinematic parameters can be calculated by any neural network model, such as Figure 4 . Including the following:

[0056] (1) Target motion feature encoding

[0057] The width h of the bounding box of any target in the static view is different due to different movement directions and movement speeds. B and Gao W B The initial bounding box coordinates cannot directly represent the changes of this pair of parameters, so it is necessary to use the bounding box corner coordinates p tl , p br To calculate h B and w B , and add it to the initial training parameters. In summary, the historical motion information input by the network is All parameters in x will be normalized with respect to the image size to ensure that the input parameters are within the interval (0, 1).

[0058] Then, the historical motion information sequence X = {x t-h ,...,x t-1 , x t} is input into the LSTM encoder:

[0059] O t , H t =LSTM enc ([X t , H t-1 ],W enc )

[0060] Finally, the hidden state H output by the tth LSTMCell is used t As a feature vector summarizing the characteristics of the target motion sequence.

[0061] (2) Regional image feature coding

[0062] Current frame image I t The background pixel blocks corresponding to the area where the observed target i is located have a continuous change process in the historical perspective. t The target bounding box center point p in c As the center, use the proj formula to put pc Projected into the historical perspective coordinate system, using a uniform size (h crop , w crop ) to crop the target surrounding area, h crop , w crop For any custom value, the resulting cropped image sequence The input feature P of the region image encoder can be expressed as:

[0063]

[0064] Among them, Concatnate is a cascade operation that merges the channel dimensions of the image.

[0065] Input P into ConvLSTM, the network output is H t，P As the image sequence feature of the region:

[0066] O t，P , H t，P =ConvLSTM([P, H t-1，P ],W P )

[0067] F P =MaxPooling(H t，P )

[0068] H t，P It is regarded as a number of neatly arranged nodes containing high-dimensional features, and the maximum pooling operation (MaxPooling) is used to aggregate the overall features F of the area P , W P is the weight parameter of ConvLSTM.

[0069] (3) Future trajectory prediction

[0070] In complex traffic scenarios, the motion patterns and properties of different categories of targets are quite different, so it is unreasonable to use a single model to model all categories of targets. This prediction network uses an LSTM decoder to receive the encoded information F of different targets. enc =(H t , F p ), LSTM decoders are all NAR decoding methods. Finally, the multi-layer perceptron MLP is used to predict the kinematic parameters m={v x , v y , a x , a y}:

[0071]

[0072] O t , H t，out =LSTM dec ([F enc , H t-1 ],W dec )

[0073] Where: H t，out It is LSTM dec The hidden state of the last time step of the output, is the predicted future target kinematic parameters, M pred ={m t+1 ,...,m t+f}. The target bounding box calculated by the predicted parameters is in the static perspective t, so before calculating the training Loss, it needs to be converted into the bounding box position and size of the corresponding future moment:

[0074]

[0075]

[0076] Here, the superscript or subscript t represents time t. Similarly, using the proj formula and the pose transformation matrix of the future vehicle motion predicted in Step 1, the bounding box sequence of view t is projected to its view t′, t′∈{t+1,...,t+f}, and finally the predicted bounding box center point coordinate sequence B is obtained. t′，pred .

[0077] Step 6: Multi-stage model training method

[0078] (1) Phase 1

[0079] The primary goal of the ego-vehicle motion estimation network is to accurately compute the pose transformation between images. Therefore, the first phase of model training aims to preserve the weights of the image feature extraction network and fine-tune the decoder network that outputs depth and pose. The depth estimation network uses a pre-trained model trained on the KITTI RAW dataset using the monodepth2 dataset with an image size of 640×192, where h represents height and w represents width.

[0080] The ego-vehicle estimation network is trained using the KITTI Tracking dataset, freezing the depth estimation encoder D during training. enc and pose estimation encoder P enc The parameters of the decoder D do not participate in the gradient descent method update. de and P deSimilar to the existing self-supervised depth estimation method, this model minimizes the average photometric reprojection loss function L p To train the pose estimation network:

[0081]

[0082]

[0083] Where: pe is the photometric reconstruction error, which is obtained by weighted addition of the structural similarity loss function (SSIM) and the L1 loss function, with the hyperparameter α = 0.85.

[0084] (2) Second stage

[0085] In the second phase of overall model training, the ego-motion estimation network will load the model weights saved from the epoch with the lowest validation set loss during fine-tuning in the first phase, and will not be trained in this round.

[0086] Ego-car motion prediction network P pred During the training process, the axis angle and offset sequence Seq between historical frames output by the vehicle motion estimation network is used. φ，d As input to the network, the model uses the predicted future frames φ pred , d pred and the future frame φ output by the ego vehicle motion estimation module tgt , d tgt {(φ t+1→t , d t+1→t ),...,(φ t+f→t+f-1 , d t+f→t+f-1 )} Calculate the MSE loss function to train the network P pred :

[0087]

[0088] Where i represents the sequence length and j represents the output data dimension.

[0089] (3) The third stage

[0090] The trajectory prediction network is trained using the KITTI Tracking dataset, and the predicted coordinate point (x pred ,y pred ) and the real coordinate point (x gt ,y gt )’s Euclidean distance error:

[0091]

[0092] Where: n is the number of targets contained in the current image.

[0093] (4) Training details

[0094] The present invention uses the KITTI dataset as a dataset for model training and verification. The KITTI dataset was jointly researched and produced by the Karlsruhe Institute of Technology in Germany and Toyota Research Institute of America, and is one of the largest multi-scenario autonomous driving algorithm evaluation datasets currently available. The KITTI dataset contains real image data collected from scenes such as urban areas, rural areas, and highways. Each image contains up to 15 vehicles and 30 pedestrians, sampled and synchronized at a frequency of 10Hz. The present invention uses the KITTI tracking dataset (KITTI Tracking) to provide the target position information and identity information required for trajectory prediction. The dataset contains video data for 20 scenes and target position information and identity information in sequence form in each video, and gives the intrinsic parameter matrix of each on-board camera. The method proposed in the present invention takes the vehicle as the target during training and uses a single-frame retrieval sequence to extract training data. The overall dataset contains a total of 4041 vehicle trajectories, 70% of the dataset is used as a training set, 10% as a validation set, and the remaining 20% ​​as a test set.

[0095] During the model training process, the loss function of the first stage is L p , Adam optimizer is selected as the parameter update optimizer, and the learning rate lr1=10 -5 20 epochs of training are performed, where lr1 is reduced to 1 / 10 of its original value after the 10th epoch. The training parameters of the second stage are the same as those of the first stage, with the initial learning rate lr2 = 10 -4 In the third stage, the optimizer and learning rate remain unchanged, and a total of 30 epochs are trained, with lr3 = 1×10 -3 Every 10 epochs, it is reduced to 1 / 2 of the original value.

[0096] (5) Trajectory prediction visualization

[0097] Figure 5The figure shows the test results of the model proposed in the present invention in different scenarios, including single-vehicle and multi-vehicle scenarios. In the single-vehicle scenario, the target vehicle maintains a small relative motion with the ego vehicle, and the proposed model has accurate prediction effects in all three key frames; in the multi-vehicle scenario, each vehicle maintains a certain speed of relative motion with the ego vehicle, where the target vehicle in the second column is in motion and the target vehicle in the third column is stationary. The green box representing the true value and the red box representing the predicted value have a high overlap rate, which reflects the good prediction accuracy of the proposed method. On the other hand, the prediction error of the proposed method for the surrounding stationary vehicles is higher than that for the moving vehicles. The reason may be that the relative motion speed of the stationary vehicle and the ego vehicle is large and in opposite directions, which affects the model prediction.

[0098] The above prediction method, as well as the designed models and model training methods can all be set in the controller device or the storage device.

[0099] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent methods or changes that do not deviate from the technology of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for predicting the trajectory of multiple targets based on monocular visual motion estimation, characterized in that: include: S1: Establish the ego vehicle visual odometry and complete the independent modeling of the ego vehicle motion; S2: Build the vehicle motion estimation network, including the depth estimation network and the pose estimation network. The depth estimation network outputs the current frame image I t Depth The pose estimation network predicts the transformation matrix T t′→t And the intrinsic parameter matrix of the vehicle camera The image I at time t′ t′ All pixels in are projected to the corresponding positions in the pixel coordinate system at time t, and the image I at time t is reconstructed. t′→t ; S3: Build a vehicle motion prediction network using an LSTM encoder-decoder structure; S4: Normalization of target trajectory; S5: Establish a trajectory prediction network to predict future trajectories; The trajectory prediction network of S5 includes a target motion feature encoder, a regional image feature encoder and a future trajectory predictor; wherein, Object motion feature encoder: Enter historical exercise information All parameters in x will be normalized with respect to the image size to ensure that the input parameters are within the interval (0,1); The historical motion information sequence X={x t-h ,…,x t-1 ,x t } is input into the LSTM encoder: O t ,H t =LSTM enc ([X t ,H t-1 ],W enc ) Finally, the hidden state H output by the tth LSTMCell is used t as a feature vector summarizing the characteristics of the target motion sequence; Regional image feature encoder: Current frame image I t The background pixel block corresponding to the area where the observed target i is located has a continuous change process in the historical perspective, with I t The target bounding box center point p in c As the center, use the proj formula to put p c Projected into the historical perspective coordinate system, using a uniform size (h crop ,w crop ) to crop the target surrounding area, and the resulting cropped image sequence The input feature P of the region image encoder is expressed as: Among them, Concatnate is a cascade operation that merges the channel dimensions of the image; Input P into ConvLSTM, the network output is H t,P As the image sequence feature of the region: O t,P ,H t,P =ConvLSTM([P,H t-1,P ],W P ) F P =MaxPooling(H t,P ) H t,P It is regarded as a number of neatly arranged nodes containing high-dimensional features, and the maximum pooling operation is used to aggregate the overall features F of the area P ; Future trajectory predictor: Use LSTM decoder to receive the encoded information F of different targets enc =(H t ,F p ), LSTM decoders are all NAR decoding methods, and finally the multi-layer perceptron MLP is used to predict the moving speed and acceleration of the center point of the target bounding box in the viewing angle at time t on the x and y axes. x ,v y ,a x ,a y }: O t ,H t,out =LSTM dec ([F enc ,H t-1 ],W dec ) Where: is the predicted future target kinematic parameters, M pred ={m t+1 ,…,m t+f }, The target bounding box calculated by the predicted parameters is in the static perspective t, so before calculating the training Loss, it needs to be converted to the bounding box position and size of the corresponding future moment: Similarly, using the proj formula and the predicted future vehicle motion pose transformation matrix, the bounding box sequence of view t is projected to its view t′, t′∈{t+1,…,t+f}, and finally the predicted bounding box center point coordinate sequence B is obtained t′,pred .

2. The method for predicting the trajectory of multiple targets based on monocular visual motion estimation according to claim 1, characterized in that: The specific process of S1 includes: decomposing the video into an image sequence consisting of multiple frames, so that the target captured in each frame is in the camera coordinate system of the vehicle at the current moment. During the driving process, the change of the vehicle's movement direction and position will cause the continuous transformation of the camera coordinate system, that is, the vehicle posture transformation. The purpose of the vehicle motion estimation is to calculate the posture transformation between the historical frame th,...,t-1 and the current frame t of the vehicle, and predict the posture transformation between the future frame t+1,...,t+f and t of the vehicle. The inter-frame transformation process can be regarded as the rotation of the camera coordinate system of the two frames around the axis a by θ. This process can be expressed by the transformation matrix T t→t′ express: Where: t′∈[th,t-1]∪[t+1,t+f]; R∈SO(3) is the rotation matrix, and is the offset of the camera coordinate axis origin.

3. The method for predicting the trajectory of multiple targets based on monocular visual motion estimation according to claim 1, characterized in that: In S2, the image I at time t is reconstructed. t′→t Specific methods: I t′→t =proj(I t ) proj:p t =KT t′→t D t K -1 p t′ Where: proj is the reprojection operation, p t and p t′ are the coordinates of each pixel in the image at time t and t′ respectively.

4. The method for predicting the trajectory of multiple targets based on monocular visual motion estimation according to claim 1, characterized in that: In S2, the depth estimation network is a codec structure, where the depth encoder D Enc Input is the current frame image I t , extract the image feature information F through the ResNet-18 structure t , F t Input to the depth decoder D Dec In, D Dec The depth D of each pixel position in the current frame image will be calculated. The value range of D is (0.1, 100), 0.1 is a depth unit, and the upper limit of the depth value is 100. The process is expressed as: F t =D Enc (I t ) D t =D Dec (F t ) The pose estimation network has the same encoder-decoder structure, and the pose encoder P Enc With D Enc Using the same ResNet-18 network structure as the feature extractor, P Enc The input is an image sequence S consisting of images at time t, t+1, and t-1 hist ={I t-1 ,I t }or{I t ,I t+1 }, first, through the cascade operation, S hist The channel dimension is merged to obtain the channel dimension that matches P Enc Input dimension P Enc The model output is a high-dimensional feature map after 32 times downsampling Pose decoder P Dec It is composed of multiple layers of convolution and activation function ReLU stacked together. The decoder outputs 6 channels, representing the axis angle φ t→t′ ∈so(3), and offset Decomposition φ t→t′ Get the rotation axis a and rotation angle θ: According to the Rodriguez formula, the exponential mapping between so(3) and SO(3) is calculated, and then combined with the offset d to obtain the pose transformation matrix T: T t+1→t =Combine(R,d) R = exp(φ^) = exp(θa^) exp(θa^)=cosθE+(1+cosθ)aa T +sinθa^ Where: ^ is the antisymmetric symbol, and the final posture transformation matrix T represents I t+1 The camera coordinate system to I t The transformation between the camera coordinate system.

5. The method for predicting the trajectory of multiple targets based on monocular visual motion estimation according to claim 4, characterized in that: During the training process of the vehicle motion estimation network, the image sequence from the historical time t-h+1 to the future time t+f-1 will be input into multiple pose estimation codecs with adjacent frames for calculation, and the complete visual odometry VO of the sequence is finally obtained. t-h+1→t-h ,…,T t+f→t+f-1 }.

6. The method for predicting the trajectory of multiple targets based on monocular visual motion estimation according to claim 4, characterized in that: In S3, the input of the LSTM encoder is the pose decoder P Dec The output historical moment part of the φ,d sequence Seq φ,d ={(φ t-h ,d t-h ),…,(φ t-1 ,d t-1 )}: F emb =LeakyReLU(f1(Seq φ,d )) O t,(φ,d) ,H t,(φ,d) =LSTM([H t-1,(φ,d) ,F emb ],W enc ) Where: f1 is a linear layer, which is used as an embedding layer to map the low-dimensional features of the input sequence to high dimensions. LeakyReLU is the activation function, α = 0.1, and this part takes the hidden state H of the last time step. t,(θ,d) As a feature vector summarizing historical poses; The LSTM decoder uses a non-autoregressive approach to t,(φ,d) Decode and output the predicted future vehicle motion: O t ,H t =LSTM([H t-1 ,H t,(φ,d) ],W dec ) f pred ,d pred =f(O t ) Where: f2 is a linear layer, the output of which is the predicted rotation axis and origin offset φ for each future time step pred ,d pred ={(φ t+1 ,d t+1 ),…,(φ t+f ,d t+f )}, and finally use the formula φ pred ,d pred Converted into pose transformation matrix T pred .

7. The method for predicting the trajectory of multiple targets based on monocular visual motion estimation according to claim 1, characterized in that: The specific process of S4: Calculate the pose transformation matrix set {T t→t′ ,t′∈[th,t+f]}, the perspective of the final prediction of the target’s future trajectory will be the perspective of the camera at time t, so when the target history bounding box B at any time t′ is t′ Before inputting into the prediction model, the bounding box B t′ The pixel coordinates of the midpoint (p tl ,p br ) t′ Convert to pixel coordinates at time t to form a new bounding box Bounding box B at time t′ t′ The coordinates of any corner point p in t′ Convert to coordinates at time t It can be expressed as: Where: d(B t′ ) is the bounding box B t′ The average depth within the region, h B ,w B For B t′ The height and width, D ij is the depth of a single pixel position in the depth map, i∈[p tl,x ,p br,x ],j∈[p tl,y ,p br,y ], when converting the camera coordinate system of any historical frame i to the camera coordinate system at time t, the transformation matrix can be obtained by matrix multiplication: T t-i→t =T t-i→t-1 T t-1→t .

8. A method for training a surrounding multi-target trajectory prediction model based on monocular visual motion estimation, characterized in that: The model comprises the ego-vehicle motion estimation network, the ego-vehicle motion prediction network, and the trajectory prediction network according to any one of claims 1 to 7; and the training comprises three stages: (1) Phase 1 The main purpose of the ego-vehicle motion estimation network is to accurately calculate the pose transformation between images. Therefore, the purpose of the first stage of model training is to retain the weights of the image feature extraction network and fine-tune the decoder network that outputs depth and pose. The depth estimation network and pose estimation network are pre-trained using the monodepth2 model trained on the KITTI RAW dataset with an image size of 640×192. The ego-vehicle estimation network is trained using the KITTI Tracking dataset, freezing the depth estimation encoder D during training. enc and pose estimation encoder P enc The parameters of the decoder D do not participate in the gradient descent method update. dec and P dec Training is performed by minimizing the average photometric reprojection loss function L p To train the pose estimation network: Where: pe is the photometric reconstruction loss, which is obtained by weighted addition of structural similarity loss and L1 loss, where α = 0.85; (2) Second stage In the second phase of overall model training, the ego-motion estimation network will load the model weights saved from the epoch with the lowest validation set loss during fine-tuning in the first phase, and will not be trained in this round. Ego-car motion prediction network P pred During the training process, the axis angle and offset sequence Seq between historical frames output by the vehicle motion estimation network is used. φ,d As input to the network, the model uses the predicted future frames φ pred ,d pred and the future frame φ output by the ego vehicle motion estimation module tgt ,d tgt {(φ t+1→t ,d t+1→t ),…,(φ t+f→t+f-1 ,d t+f→t+f-1 )} Calculate the MSE loss function to train the network P pred : Where i represents the sequence length and j represents the output data dimension; (3) The third stage The trajectory prediction network is trained using the KITTI Tracking dataset. The Euclidean distance error between the predicted coordinates and the true coordinates is taken into account when calculating the trajectory prediction error: Where: n is the number of targets contained in the current image; In the above training process, the loss function of the first stage is L p , Adam optimizer is selected as the parameter update optimizer, and the learning rate lr1=10 -5 20 epochs of training were performed, where lr1 was reduced to 1 / 10 of its original value after the 10th epoch; the training parameters of the second stage were the same as those of the first stage, with the initial learning rate lr2 = 10 -4 In the third stage, the optimizer and learning rate remain unchanged, and a total of 30 epochs are trained, with lr3 = 1 × 10 -3 Every 10 epochs, it is reduced to 1 / 2 of the original value.

9. A device for predicting the trajectory of multiple targets based on monocular visual motion estimation, characterized in that: The device is provided with a program carrier of the prediction method described in any one of claims 1 to 7, or at least one of the program carriers corresponding to the ego-vehicle motion estimation network, ego-vehicle motion prediction network, and trajectory prediction network described in any one of claims 1 to 7, and / or a program carrier of the model training method described in claim 8.

Images