Moving target trajectory prediction method and device based on visual transformation

Abstract

The present invention discloses a method and device for predicting the trajectory of a moving target based on visual transformation, which comprises the following steps: S1. Real-time acquisition of road environment images acquired by a vehicle-mounted image acquisition device, and detection of moving targets to obtain a 2D bounding box of the moving target; S2. Using the 2D bounding box of the moving target to predict the target trajectory and track the target to obtain the predicted coordinates of the moving target trajectory; S3. According to the coordinate conversion relationship between the image pixels and the vehicle body obtained by pre-calibration, the predicted coordinates of the moving target trajectory are converted to the vehicle body coordinate system to obtain the predicted coordinates of the moving target trajectory in the vehicle body coordinate system. The present invention can realize the prediction of the target trajectory of vehicle-mounted environment perception, and has the advantages of high prediction efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of vehicle environment perception technology, and in particular to a moving target trajectory prediction method and device based on visual transformation. Background Art

[0002] Unmanned vehicles achieve automatic driving on public roads through perception, prediction, planning and control technologies. They often need to use on-board sensors such as on-board cameras, lidar, IMU (inertial measurement unit) and GNSS (global navigation satellite system) to perceive and locate the vehicle's surrounding environment, as well as the vehicle's own positioning and navigation information, and then make real-time decisions and plans based on map information and obstacle information. Environmental perception, as an important part of unmanned driving, is the key to the interaction between unmanned vehicles and external environmental information. Through environmental perception, unmanned vehicles can better simulate the perception ability of human drivers, so as to understand their own and surrounding driving situations. For unmanned vehicle decision-making and planning, it is very important to accurately predict the movement trajectory of obstacles. The accuracy of trajectory prediction will directly affect the safety and reliability of unmanned vehicle driving.

[0003] The existing technology usually uses the following methods to predict the motion trajectory of unmanned vehicles:

[0004] 1. For videos captured by on-board cameras, a rule-based prediction algorithm or a machine learning-based unmanned driving moving target prediction algorithm is used. However, this type of method may have problems such as missed detection due to the instability of the detection algorithm, or tracking failure when the target is blocked. Therefore, the prediction accuracy is limited and it is not suitable for realizing real-time moving target trajectory prediction in unmanned vehicles with high prediction accuracy requirements.

[0005] 2. Use the moving target trajectory prediction method based on laser point cloud data, that is, to predict the moving target trajectory based on laser point cloud data. However, the equipment used by this type of method is expensive and has high requirements for the working environment. In addition, due to the low resolution of its data, it cannot accurately distinguish the category of the predicted target. When the targets are close or intersecting, it is easy to produce inaccurate or wrong trajectory prediction. The calculation time and accuracy of a single-frame laser radar scan are also unstable, which cannot meet the safety requirements (stable, small variance) in unmanned driving application scenarios.

[0006] Patent application CN202110331661.2 discloses a method for predicting the trajectory of unmanned neighboring vehicles. The solution uses LKDBSCAN (Limit KDBSCAN) to extract the vehicle set and road condition information around the unmanned neighboring vehicle in the video data and point cloud data, and then uses the long short-term memory neural network LSTM to predict the real-time behavior of the vehicle. Finally, the vehicle behavior predicted by LSTM and the historical behavior data of the vehicle are combined to predict the trajectory of the vehicle through BLSTM (Behavior based LSTM). The above scheme can only realize the trajectory prediction of neighboring vehicles, and the vehicle may encounter moving targets such as people and animals during driving. The above scheme cannot realize the trajectory prediction of moving targets other than neighboring vehicles, and has great limitations in applicability. In addition, the above scheme needs to process video data and point cloud data at the same time. At the same time, the prediction of vehicle trajectory needs to rely on the prediction of vehicle behavior and historical behavior data. It is not only complex to implement and has a large amount of data processing, but also leads to problems such as low prediction efficiency. Summary of the invention

[0007] The technical problem to be solved by the present invention is: in view of the technical problems existing in the prior art, the present invention provides a method and device for predicting the trajectory of a moving target based on visual transformation, which has a simple implementation method, fast prediction speed, high efficiency and accuracy, and can realize real-time trajectory prediction of various surrounding moving targets during vehicle operation.

[0008] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0009] A moving target trajectory prediction method based on visual transformation is applied to vehicle-mounted environment perception, characterized in that the method comprises the following steps:

[0010] S1. Real-time acquisition of road environment images collected by the vehicle-mounted image acquisition device, and detection of moving targets to obtain a 2D bounding box of the moving target;

[0011] S2. Use the 2D bounding box of the moving target to predict the target trajectory and track the target to obtain the predicted coordinates of the moving target trajectory;

[0012] S3. According to the coordinate conversion relationship between the image pixels and the vehicle body obtained in advance, the predicted coordinates of the moving target trajectory are converted to the vehicle body coordinate system to obtain the predicted coordinates of the moving target trajectory in the vehicle body coordinate system.

[0013] Furthermore, in step S1, the moving target is detected by inputting the road environment image acquired in real time into the target detection network model, and the target detection network model is trained using a historical road environment image data set acquired by a vehicle-mounted image acquisition device.

[0014] Furthermore, step S1 also includes identifying the target category based on the detected motion trajectory, inertial characteristics, and speed change characteristics of the moving target, wherein when the motion trajectory of the moving target tends to a straight line parallel to the lane line boundary and does not exceed the lane line boundary, and the inertial characteristics of the moving target are greater than a preset threshold and the speed change rate is less than a preset threshold, it is determined to be a vehicle.

[0015] Furthermore, in step S2, a Kalman filter is used to predict the target trajectory of the 2D bounding box of the moving target; the depth features of the 2D bounding box are extracted by using a pre-trained deep network model, and then the 2D bounding box and the extracted depth features are used to track the moving target, and the mass m and speed v are used as physical characteristics of the moving target to match and correct the results of target recognition during the tracking process.

[0016] Furthermore, in step S3, by taking the laser radar arranged on the vehicle as the reference coordinate system, the vehicle body and the laser radar, as well as the laser radar and the vehicle-mounted camera are jointly calibrated respectively to determine the change relationship between the vehicle body coordinate system and the laser radar coordinate system, and between the laser radar coordinate system and the vehicle-mounted camera. The external parameter matrix from the laser radar to the vehicle body and the external parameter matrix from the laser radar to the vehicle-mounted camera are obtained by joint calibration. The external parameter matrix from the laser radar to the vehicle body obtained by the joint calibration is inverted and multiplied with the external parameter matrix from the laser radar to the vehicle-mounted camera to obtain the external parameter transformation matrix from the vehicle body to the vehicle-mounted camera. According to the internal parameter matrix of the vehicle-mounted camera and the external parameter transformation matrix from the vehicle body to the vehicle-mounted camera, the coordinate transformation relationship between the image pixels and the vehicle body is constructed.

[0017] Furthermore, the internal parameter matrix of the vehicle-mounted image acquisition device is Where fx, fy, cx, and cy are respectively the scale factor of the vehicle-mounted image acquisition device in the u-axis direction in the pixel coordinate system, the scale factor of the vehicle-mounted image acquisition device in the v-axis direction in the pixel coordinate system, the number of horizontal pixels that differ between the center pixel coordinate of the image and the pixel coordinate of the image origin, and the number of vertical pixels that differ between the center pixel coordinate of the image and the pixel coordinate of the image origin. The external parameter matrix between the vehicle body and the vehicle-mounted image acquisition device is where r 11 ~r 33 are the elements of the orthogonal rotation matrix, t1-t3 are the elements of the translation matrix, and the coordinate transformation relationship between the image pixels and the vehicle body is constructed as follows:

[0018]

[0019] Among them, u, v are the coordinates of the pixel coordinate point, w is the scale factor, x camera ,ycamera 、z camera They are the x, y, and z axis coordinates in the vehicle body coordinate system respectively;

[0020] Configure z according to the distance h between the vehicle body and the ground camera , i.e. zcamera = -h, and the internal parameter matrix and the external parameter matrix are multiplied and converted into:

[0021]

[0022] Among them, A00~A23 are the elements of the matrix A obtained by multiplying the internal parameter matrix and the external parameter matrix respectively;

[0023] The final coordinate transformation relationship between the image pixels and the vehicle body is obtained as follows:

[0024]

[0025] Among them, x and y are the x-axis and y-axis coordinates in the vehicle coordinate system respectively.

[0026] Furthermore, in step S3, the coordinates of the bottom center point of the 2D bounding box obtained by each trajectory prediction are projected from above using the coordinate transformation relationship to obtain the predicted trajectory coordinates of the moving target in the current frame corresponding to the vehicle body coordinate system, thereby completing the trajectory prediction of the moving target.

[0027] Furthermore, the step S3 further includes predicting the position deviation between the coordinates of the moving target in the two frames before and after the moving target to obtain the moving target's moving speed, and calculating the moving target's moving speed according to the formula Estimate the motion speed of the next frame, where vx' and vy' are the velocity components on the x-axis and y-axis of the motion speed of the next frame, vx and vy are the velocity components on the x-axis and y-axis of the motion speed of the current frame, and k is the preset threshold.

[0028] A moving target trajectory prediction device based on visual transformation, comprising:

[0029] The detection module is used to acquire the road environment image collected by the vehicle-mounted image acquisition device in real time, detect the moving target, and obtain the 2D bounding box of the moving target;

[0030] A prediction and tracking module, used to use the 2D bounding box of the moving target to predict the target trajectory and track the target to obtain the predicted coordinates of the moving target trajectory;

[0031] The visual transformation module is used to transform the moving target trajectory prediction coordinates into the vehicle body coordinate system according to the coordinate transformation relationship between the image pixels and the vehicle body obtained in advance, so as to obtain the moving target trajectory prediction coordinates in the vehicle body coordinate system.

[0032] A computer device comprises a processor and a memory, wherein the memory is used to store a computer program, the processor is used to execute the computer program, and the processor is used to execute the computer program to perform the above method.

[0033] Compared with the prior art, the advantages of the present invention are:

[0034] 1. The present invention acquires the road environment image in real time by the on-board camera of the unmanned vehicle, detects the 2D bounding box of each running target on the road, and then performs target prediction and target tracking based on the 2D bounding box. For the predicted moving target trajectory prediction coordinates, projection is performed according to the coordinate conversion relationship to obtain the moving target trajectory prediction coordinates in the vehicle body coordinate system. The target prediction tracking and visual transformation can be combined to achieve reliable and accurate moving target trajectory prediction, avoid missed detection problems caused by unstable detection and tracking failure problems caused by obstructed targets. When the targets are close to each other or intersecting, accurate trajectory prediction can still be achieved. Not only is the prediction accuracy high, but the prediction speed is fast and the flexibility is strong. It can achieve fast and accurate detection and trajectory prediction of various moving targets in the road environment.

[0035] 2. The present invention further fully considers the inertia factor of the object's movement, the characteristics of the moving object, and the difference in the movement speed of objects of different categories, and combines the inertia and characteristics of the moving target to consider the target category identification, so as to solve the problems of inaccurate identification of individual targets and frequent switching of tracking IDs.

[0036] 3. The present invention can further effectively construct a coordinate conversion relationship between pixels and the vehicle body by continuously converting between the three coordinate systems of vehicle body-vehicle radar-vehicle camera, and realize the conversion from low dimension (pixel coordinate system) to high dimension (vehicle body coordinate system) by setting the height of the vehicle body coordinate system to a fixed value and solving the equations.

[0037] 4. The present invention further obtains the coordinate conversion relationship between pixels and the vehicle body by jointly calibrating the vehicle body and the laser radar as well as the vehicle-mounted camera and the laser radar, and then calculates the displacement of the projection coordinates of the bottom center point of the moving target bounding box in adjacent frames based on the top-down projection, so as to quickly and accurately detect the speed of various moving targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the implementation flow of the moving target trajectory prediction method based on visual transformation in this embodiment.

[0039] Figure 2 It is a detailed flow chart of realizing moving target trajectory prediction in a specific application embodiment of the present invention.

[0040] Figure 3 It is a schematic diagram of the layout principle of the vehicle-mounted camera and laser radar in this embodiment.

[0041] Figure 4 It is a schematic diagram of the principle of coordinate transformation in this embodiment. DETAILED DESCRIPTION

[0042] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0043] like Figure 1 As shown, the steps of the moving target trajectory prediction method based on visual transformation in this embodiment include:

[0044] S1. Acquire the road environment image collected by the vehicle-mounted image acquisition device in real time, detect the moving target, and obtain the 2D (Dimensional) bounding box of the moving target. The 2D bounding box is the minimum detection rectangular frame of the detected moving target;

[0045] S2. Use the 2D bounding box of the moving target to predict the target trajectory and track the target to obtain the predicted coordinates of the moving target trajectory;

[0046] S3. According to the coordinate conversion relationship between the image pixels and the vehicle body obtained in advance, the predicted coordinates of the moving target trajectory are converted to the vehicle body coordinate system to obtain the predicted coordinates of the moving target trajectory in the vehicle body coordinate system.

[0047] This embodiment acquires the road environment image collected by the on-board camera of the unmanned vehicle in real time, detects the 2D bounding box of each running target on the road, and then performs target prediction and target tracking based on the 2D bounding box. For the predicted moving target trajectory prediction coordinates, the moving target trajectory prediction coordinates in the vehicle body coordinate system are obtained by projection according to the coordinate conversion relationship. The target prediction tracking and visual transformation can be combined to achieve reliable and accurate moving target trajectory prediction, which can avoid the problem of missed detection caused by unstable detection and the problem of tracking failure caused by occlusion of the target. When the targets are close to each other or intersecting, accurate trajectory prediction can still be achieved. Not only is the prediction accuracy high, but the prediction speed is fast and the flexibility is strong, which can achieve rapid and accurate detection and trajectory prediction of various moving targets in the road environment. The predicted moving target trajectory can be further provided to the decision-making end of the unmanned vehicle to provide decision support for the decision-making planning of the unmanned vehicle.

[0048] In this embodiment, the vehicle-mounted image acquisition device in step S1 is specifically a vehicle-mounted camera, and the vehicle-mounted camera is specifically arranged on the top of the unmanned vehicle, such as Figure 3As shown, the vehicle-mounted camera faces the ground to collect real-time road environment information. The layout position of the vehicle-mounted camera and the type of camera used can be configured and selected according to actual needs.

[0049] In this embodiment, in step S1, the moving target is detected by inputting the road environment image acquired in real time into the target detection network model, and the target detection network model is trained using the historical road environment image data set collected by the vehicle-mounted image acquisition device. The historical road environment image data set is various road environment images previously adopted by the vehicle-mounted camera, and the historical road environment image data set contains various moving targets, and the moving targets can specifically be various types of targets to be identified, such as vehicles, pedestrians, and animals. The target detection network model is obtained by marking the moving targets in each image in the historical road environment image data set and then training, and the target detection network model can be used to realize the detection and identification of the target later.

[0050] The detailed steps of building the above target detection network model include:

[0051] S101. Use the on-board camera of the vehicle to record the video data of the surrounding road traffic environment, and divide it into training set data TRAIN and test set data TEST.

[0052] The video data collected by the above-mentioned vehicle-mounted camera can be in MP4 format but is not limited to it. The video data is segmented and saved in the form of pictures through a video processing program, and then the pictures are divided into a training set TRAIN and a test set TEST according to a predetermined ratio.

[0053] S102: Annotate the images in the training set TRAIN, including the location information of all objects and their category vectors, and generate two parts, namely, a training image set pa and a verification image set pb.

[0054] When annotating the images in the training set TRAIN, a 2D bounding box can be generated. The 2D bounding box is the minimum detection rectangular frame of the detected moving target. The upper left corner coordinates (x, y) and the width and height (w, h) of the 2D bounding box are the location information of the moving target. Then the annotated images are randomly divided into a training image set pa and a verification image set pb according to a certain ratio.

[0055] S103: Use the prepared training picture set pa to input the initial target detection network model for training, and obtain the trained target detection model M1.

[0056] The above target detection network model can be a deep neural network model YOLOv5 or other types of neural network models.

[0057] S104: collect images from the vehicle camera in real time and publish them in the form of topic TP1 through ROS.

[0058] Further explanation: the topic TP1 format can be customized, such as " / minibus / camera / image" and so on.

[0059] S105: Use the target detection model M1 obtained in S103 to subscribe to the topic TP1 of step S104 and convert it into a picture, then detect the converted picture to obtain the 2D bounding box and category of the moving target in the picture.

[0060] The above-mentioned topic TP1 can be converted into a picture form through tool libraries such as OpenCV.

[0061] This embodiment uses a deep neural network for target detection based on the road environment image captured by the vehicle-mounted camera. It can not only quickly and accurately detect various moving targets and obtain the 2D bounding box of the target, but also identify the category to which each object belongs.

[0062] The inertia factors of different moving objects, the characteristics of moving objects, and the moving speed characteristics of different categories of objects are different. For example, the moving speed of a vehicle is different from that of a pedestrian, and the moving trajectory of a vehicle is different from that of a pedestrian. In this embodiment, the inertia factors of the moving objects, the characteristics of the moving objects, and the difference in the moving speeds of different categories of objects are fully considered, and the inertia and characteristics of the moving targets are combined to achieve the classification of the moving targets.

[0063] When a vehicle moves along the lane boundary, it will have the following characteristics: ① The motion trajectory satisfies the requirement of being a straight line roughly parallel to the lane boundary and will not exceed the lane boundary ② The inertia of the vehicle's motion is large, and the speed changes are relatively uniform, while the speed of pedestrians is unpredictable. This embodiment takes the above characteristics into consideration, and assists in distinguishing the target category based on the motion trajectory, inertia characteristics, and speed change characteristics of the moving target. When the motion trajectory of the moving target is a straight line tending to be parallel to the lane boundary and will not exceed the lane boundary, and the inertia characteristics of the moving target are greater than a preset threshold, and the speed change rate is less than a preset threshold, it is determined to be a vehicle. By combining the inertia and characteristics of the moving target for target category recognition, the category to which each object on the road belongs can be efficiently identified, and the problems of inaccurate identification of individual targets and frequent switching of tracking IDs can be well solved.

[0064] In step S2 of this embodiment, a Kalman filter is used to predict the target trajectory of the 2D bounding box of the moving target. The Kalman filter is used to track the X and Y directions of the center point of the bottom of the bounding box of the moving target. Of course, the specific target point of the bounding box tracked can also be other position points besides the center point.

[0065] In step S2 of this embodiment, the deep features of the 2D bounding box are extracted by using a pre-trained deep network model, and then the 2D bounding box and the extracted deep features are used to track the moving target. The deep network model uses a feature extraction network to extract the deep feature Feature of the target corresponding to the 2D bounding box. The feature extraction network can be a network structure such as ResNet (residual network). The detected 2D bounding box and the deep feature Feature corresponding to the 2D bounding box extracted by the feature extraction network are used to track the target to form a target tracking module. This embodiment uses a deep learning detection network to detect the category and 2D bounding box of each object on the road, and then further sends the 2D bounding box and the features extracted by the deep network to the target tracking module for target tracking, which can well solve the problem of frequent switching of tracking labels.

[0066] When the present embodiment uses the Kalman filter algorithm to predict the trajectory of the detected moving target, the target tracking module of the deep convolution re-identification appearance model fused with the cascade matching algorithm in the multi-target tracking algorithm Deepsort is specifically used to obtain the motion trajectory of each target individual, and then the trajectory information predicted by the Kalman filter is converted to the world coordinate system through visual transformation. And the inertia principle of the object is used to combine the trajectory with the re-identification (re-identification) algorithm, and the strategic judgment is adopted to reduce the influence of occlusion and appearance changes, with high reliability, good accuracy and stronger robustness. According to the law of inertia, inertia is related to the mass m and speed v of the detected moving target. Therefore, according to Newton's first law, the larger the mass m of the detected moving target, that is, the larger the 2D bounding box, the larger the speed v, and the less likely the inertia of the object is to change. In this embodiment, the mass m and speed v of the moving target are expressed as the physical characteristics of the moving target by using the re-id algorithm for target recognition, and the re-id result is matched and corrected, thereby effectively solving the problem of mismatching in the previous and next frames caused by occlusion and the appearance change of the same moving target.

[0067] In this embodiment, after detecting a moving target based on the image taken by the vehicle camera and completing tracking, the moving target trajectory prediction coordinates are further converted to the vehicle body coordinate system according to the coordinate conversion relationship between the image pixels and the vehicle body obtained in advance, and the moving target trajectory prediction coordinates in the vehicle body coordinate system are obtained, and the moving target trajectory prediction is realized by using visual transformation.

[0068] In this embodiment, a laser radar is also arranged on the unmanned vehicle. The laser radar can be arranged on the left, right, front and rear sides of the vehicle. Figure 3As shown. The coordinate conversion relationship between the above-mentioned image pixels and the vehicle body is specifically achieved by jointly calibrating the vehicle body and the laser radar, and the laser radar and the vehicle-mounted camera (vehicle-mounted image acquisition device), that is, using the laser radar as the reference coordinate system, the vehicle body and the laser radar, and the laser radar and the vehicle-mounted camera are jointly calibrated twice to determine the change relationship between the vehicle body coordinate system and the laser radar coordinate system, and the laser radar coordinate system and the vehicle-mounted camera. The external parameter matrix from the laser radar to the vehicle body and the external parameter matrix from the laser radar to the vehicle-mounted camera are obtained by joint calibration. The external parameter matrix includes a rotation matrix and a translation matrix. The external parameter matrix from the laser radar to the vehicle body obtained by the joint calibration is inverted and multiplied with the external parameter matrix from the laser radar to the vehicle-mounted camera to obtain the external parameter transformation matrix from the vehicle body to the vehicle-mounted camera, that is, the projection transformation matrix between the vehicle body coordinates and the image pixel points is obtained. The external parameter transformation matrix from the vehicle body to the vehicle-mounted camera is calculated according to the internal parameter matrix of the vehicle-mounted camera and the two external parameter transformation matrices obtained by the joint calibration. The coordinate conversion relationship between the image pixels and the vehicle body is constructed, and the coordinate conversion between the image pixels and the vehicle body can be realized.

[0069] In this embodiment, the conversion from pixel coordinates to vehicle body coordinates is specifically performed in two steps: first, converting from vehicle body coordinates to vehicle camera coordinates, and then converting from pixel coordinates to vehicle camera coordinates, specifically:

[0070] The first step is to calculate the external parameter transformation matrix from the vehicle body to the vehicle camera: the external parameter matrix of the joint calibration result is to realize the transformation from the vehicle camera to the lidar, so it is necessary to invert it to get the transformation from the lidar to the vehicle camera. At the same time, it is necessary to use tf monitoring to obtain the external parameters from the vehicle body to the lidar. The two external parameter transformation matrices from the vehicle body to the lidar and the lidar to the vehicle camera are multiplied to get the external parameter transformation matrix from the vehicle body to the vehicle camera.

[0071] Step 2: Coordinate conversion from pixel to vehicle-mounted camera: The pixel coordinate system is two-dimensional, and the vehicle-mounted camera coordinate system is three-dimensional. Under normal circumstances, coordinate transformation can be performed if the dimensions are the same, or conversion from high dimension to low dimension can be performed. However, conversion from low dimension to high dimension cannot be performed due to missing information. This embodiment achieves conversion by solving a column equation given a certain dimension value.

[0072] The coordinate transformation relationship between the image pixels and the vehicle body in this embodiment is specifically obtained by constructing the intrinsic parameter matrix of the vehicle-mounted image acquisition device and the extrinsic parameter matrix between the vehicle body and the vehicle-mounted camera. The intrinsic parameter matrix of the vehicle-mounted camera is Where fx, fy, cx, and cy are the scale factors of the vehicle camera in the u-axis direction in the pixel coordinate system, the scale factors of the vehicle camera in the v-axis direction in the pixel coordinate system, the number of horizontal pixels between the center pixel coordinate of the image and the pixel coordinate of the origin of the image (i.e., the horizontal coordinate of the camera principal point), and the number of vertical pixels between the center pixel coordinate of the image and the pixel coordinate of the origin of the image (i.e., the vertical coordinate of the camera principal point). The external parameter matrix from the vehicle body to the vehicle camera is where r 11 ~r 33 are the elements of the orthogonal rotation matrix, and t1~t3 are the elements of the translation matrix, that is, the coordinate transformation relationship between the image pixel and the vehicle camera is constructed as follows:

[0073]

[0074] Among them, u and v are the pixel coordinates, w is the scale factor, and x is the pixel coordinates. camera ,y camera 、z camera They are the x, y, and z axis coordinates in the vehicle coordinate system respectively;

[0075] The origin of the vehicle coordinate system is at the center of the rear axle of the vehicle, and the rear axle center is h from the ground. According to this h, zcamera is configured, that is, zcamera = -h (the plane where the ground is located is the plane of z = -h in the vehicle coordinate system). Then according to zcamera = -h, the intrinsic parameter matrix and the extrinsic parameter matrix are multiplied and converted to:

[0076]

[0077] Among them, A00~A23 are the elements of the matrix A obtained by multiplying the internal parameter matrix and the external parameter matrix;

[0078] After matrix multiplication, we get the system of equations:

[0079]

[0080] The final coordinate transformation relationship between the image pixels and the vehicle body is:

[0081]

[0082] Among them, x and y are the x-axis and y-axis coordinates in the vehicle coordinate system respectively.

[0083] By cross multiplication, the coordinates (x, y) of any pixel coordinate point (u, v) in the vehicle coordinate system can be obtained.

[0084] In step S3 of this embodiment, the coordinates of the bottom center point of the 2D bounding box obtained by each trajectory prediction are projected from above using a coordinate transformation relationship to obtain the predicted trajectory coordinates of the moving target in the current frame corresponding to the vehicle body coordinate system, thereby completing the trajectory prediction of the moving target.

[0085] Conventional coordinate transformation is often a transformation between two different types of coordinate systems. For example, the coordinate transformation of the laser and the vehicle body can obtain the coordinates of the laser coordinate system in the vehicle body coordinate system. This embodiment can finally construct the coordinate transformation relationship between the pixel and the vehicle body by continuously transforming between the three coordinate systems of the vehicle body, the on-board radar, and the on-board camera, and then multiplying by the internal parameters of the on-board camera. In addition, since the pixel coordinate system is two-dimensional and the vehicle body coordinate system is three-dimensional, the transformation from low dimension to high dimension can be achieved by setting the height of the vehicle body coordinate system to a fixed value and solving the equation. The transformation principle is as follows: Figure 4 As shown in the figure, self-drivingcar represents the driverless car, lsliadar represents the front Leishen solid-state laser radar, camera represents the front camera on the vehicle, R1 and T1 represent the rotation matrix and translation matrix from the laser radar to the vehicle body, R2 and T2 represent the rotation matrix and translation matrix from the vehicle camera to the laser radar, R3 and T3 are obtained by multiplying (R1, T1) and (R2, T2), and finally the transformation relationship between the vehicle body and the pixel coordinate system is obtained.

[0086] After step S3 of this embodiment, the position deviation of the coordinates is predicted based on the moving target trajectory of the two frames before and after to obtain the moving target's moving speed. The vehicle body and the laser radar, as well as the vehicle-mounted camera and the laser radar are calibrated jointly to obtain the coordinate conversion relationship between the pixel and the vehicle body, and then the displacement in the X direction and the Y direction of the projection coordinates of the bottom center point of the bounding box of the moving target in the adjacent frames are calculated based on the top view projection, and the speed in the x direction and the speed in the y direction of the tracked moving target are calculated respectively, so that the speed detection of various moving targets can be realized quickly and accurately.

[0087] This embodiment specifically calculates the position deviation between the projection coordinates (u1, v1) of the bottom center point of the 2D bounding box of the target in the current frame and the projection coordinates (u2, v2) of the center point of the 2D bounding box of the target in the previous frame, and substitutes them into the transformation equation group shown in the above formula (4) to obtain the coordinates (x1, y1) and (x2, y2) corresponding to the vehicle body coordinate system. Finally, the target's movement speed can be calculated according to the formula vx = (x2-x1) / Δt, vy = (y2-y1) / Δt, where Δt is the time interval between the previous and next two frames.

[0088] The speed of the vehicle in the previous frame can be decomposed into the speed vx in the x direction and the speed vy in the y direction along the lane line, so further speed estimation can be constrained using this feature, where the vx direction is the normal direction of the lane line and the vy direction is the tangent direction of the lane line. Since the unmanned vehicle will travel a large distance at a large normal speed vx in a very short time when the vehicle is driving close to the lane line boundary, it will exceed the lane line boundary. Therefore, the ratio of vx to vy must be less than the threshold k, otherwise if the ratio of vx to vy is greater than k, the unmanned vehicle will inevitably exceed the lane line boundary in a very short time. This embodiment further uses the following formula to update the speed of the next frame based on the ratio relationship between vx and vy, and the update formula is: The value of k can be determined based on experience. For example, if it is 0.09, the calculation equation is as follows:

[0089] This embodiment collects road traffic environment information in real time through the on-board camera, uses a deep learning detection network to effectively detect the category and 2D bounding box of each object on the road, and further sends the 2D bounding box and the features extracted by the deep network to the target tracking module for target tracking, which can well solve the problem of frequent switching of tracking labels. Further, through the joint calibration between the on-board camera and the lidar and the overhead projection of the camera to the ground, reliable and accurate target trajectory prediction coordinates can be finally obtained, and the trajectory prediction of various moving targets can be realized based on visual transformation under any on-board camera, thereby improving the accuracy of trajectory prediction of moving targets of unmanned vehicles and reducing the error of trajectory prediction, thereby providing accurate information for upper-level decision-making planning, which can greatly improve the safety and efficiency of unmanned driving, and at the same time can greatly reduce the cost and complexity of environmental perception.

[0090] The above method of this embodiment is applied to vehicle perception of unmanned vehicles to achieve trajectory prediction of moving targets. Of course, it can also be applied to ordinary vehicles with similar requirements according to actual needs, such as vehicle automatic braking function.

[0091] The moving target trajectory prediction device based on visual transformation in this embodiment includes:

[0092] The detection module is used to acquire the road environment image collected by the vehicle-mounted image acquisition device in real time, detect the moving target, and obtain the 2D bounding box of the moving target;

[0093] The prediction and tracking module is used to predict the target trajectory and track the target using the 2D bounding box of the moving target to obtain the predicted coordinates of the moving target trajectory;

[0094] The visual transformation module is used to transform the predicted coordinates of the moving target trajectory into the vehicle body coordinate system according to the coordinate transformation relationship between the image pixels and the vehicle body obtained in advance, so as to obtain the predicted coordinates of the moving target trajectory in the vehicle body coordinate system.

[0095] The detection module of this embodiment specifically includes a target detection network model, which detects the moving target by inputting the road environment image acquired in real time into the target detection network model. The target detection network model is trained using a historical road environment image data set collected by a vehicle-mounted image acquisition device.

[0096] The detection module of this embodiment also includes a target category recognition unit, which is used to identify the target category based on the motion trajectory, inertial characteristics, and speed change characteristics of the detected moving target, wherein when the motion trajectory of the moving target tends to a straight line parallel to the lane line boundary and does not exceed the lane line boundary, and the inertial characteristics of the moving target are greater than a preset threshold and the speed change rate is less than a preset threshold, it is determined to be a vehicle.

[0097] The prediction and tracking module of this embodiment specifically includes a trajectory prediction unit and a target tracking unit. The trajectory prediction unit uses a Kalman filter to predict the target trajectory of the 2D bounding box of the moving target; the target tracking unit extracts the depth features of the 2D bounding box by using a pre-trained deep network model, and then uses the 2D bounding box and the extracted depth features to track the moving target.

[0098] In the visual transformation module of this embodiment, the vehicle body and the laser radar, as well as the laser radar and the vehicle-mounted camera are jointly calibrated respectively, and the coordinate transformation relationship between the image pixels and the vehicle body is constructed using the intrinsic parameter matrix of the vehicle-mounted image acquisition device and the extrinsic parameter matrix between the vehicle body and the vehicle-mounted image acquisition device. The coordinate transformation relationship is as described above.

[0099] In this embodiment, the above-mentioned vehicle-mounted image acquisition device is a vehicle-mounted camera. The modules in the moving target trajectory prediction device can be arranged on the vehicle to process the images collected by the vehicle-mounted camera in real time. They can also be arranged on a remote control terminal to transmit data to the remote control terminal and process the data at the control terminal to reduce the setting of modules on the vehicle. The specific setting method can be determined according to actual needs.

[0100] The moving target trajectory prediction device based on visual transformation in this embodiment corresponds one to one with the moving target trajectory prediction method based on visual transformation described above, and will not be described in detail here.

[0101] This embodiment also includes a computer device, including a processor and a memory, the memory is used to store a computer program, the processor is used to execute the computer program, and the processor is used to execute the computer program to perform the above method.

[0102] Those skilled in the art will appreciate that the above description of the computer device is merely an example and does not constitute a limitation on the computer device. The computer device may include more or fewer components than described above, or may combine certain components, or may include different components, such as input and output devices, network access devices, buses, etc. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect the various parts of the entire computer device.

[0103] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the computer device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0104] If the module / unit integrated in the computer device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned distributed crawler method embodiments based on template annotation can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, electrical signal and software distribution medium, etc.

[0105] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A moving target trajectory prediction method based on visual transformation, applied to vehicle-mounted environment perception, characterized in that: The steps of the method include: S1. Real-time acquisition of road environment images collected by the vehicle-mounted image acquisition device, and detection of moving targets to obtain a 2D bounding box of the moving target; S2. Use the 2D bounding box of the moving target to predict the target trajectory and track the target to obtain the predicted coordinates of the moving target trajectory; S3. According to the coordinate conversion relationship between the image pixels and the vehicle body obtained in advance, the predicted coordinates of the moving target trajectory are converted to the vehicle body coordinate system to obtain the predicted coordinates of the moving target trajectory in the vehicle body coordinate system; In the step S3, by taking the laser radar arranged on the vehicle as the reference coordinate system, the vehicle body and the laser radar, as well as the laser radar and the vehicle-mounted camera are jointly calibrated respectively to determine the change relationship between the vehicle body coordinate system and the laser radar coordinate system, and the laser radar coordinate system and the vehicle-mounted camera, and the laser radar to vehicle body external parameter matrix and the laser radar to vehicle-mounted camera external parameter matrix are obtained by joint calibration, and the external parameter matrix of the laser radar to vehicle body obtained by the joint calibration is inverted and multiplied with the external parameter matrix of the laser radar to vehicle-mounted camera to obtain the external parameter transformation matrix of the vehicle body to the vehicle-mounted camera, and the vehicle-mounted camera internal parameter matrix and the vehicle-mounted camera external parameter transformation matrix are obtained by setting the height of the vehicle body coordinate system to a fixed value. h ,according to The coordinate transformation relationship between the image pixels and the vehicle body is constructed, where ~ are the elements of matrix A obtained by multiplying the internal parameter matrix and the external parameter matrix, u and v are the coordinates of the pixel points, x,y They are the x and y axis coordinates in the vehicle coordinate system respectively.

2. The moving target trajectory prediction method based on visual transformation according to claim 1 is characterized in that: In step S1, the moving target is detected by inputting the road environment image acquired in real time into the target detection network model, and the target detection network model is trained using a historical road environment image data set collected by a vehicle-mounted image acquisition device.

3. The moving target trajectory prediction method based on visual transformation according to claim 1 is characterized in that: The step S1 also includes identifying the target category based on the detected motion trajectory, inertial characteristics, and speed change characteristics of the moving target, wherein when the motion trajectory of the moving target tends to a straight line parallel to the lane line boundary and does not exceed the lane line boundary, and the inertial characteristics of the moving target are greater than a preset threshold and the speed change rate is less than a preset threshold, it is determined to be a vehicle.

4. The moving target trajectory prediction method based on visual transformation according to claim 1 is characterized in that: In step S2, a Kalman filter is used to predict the target trajectory of the 2D bounding box of the moving target; the depth features of the 2D bounding box are extracted by using a pre-trained deep network model, and then the 2D bounding box and the extracted depth features are used to track the moving target, and the mass m and speed v are used as physical features of the moving target to match and correct the results of target recognition during the tracking process.

5. The moving target trajectory prediction method based on visual transformation according to claim 1 is characterized in that: The internal parameter matrix of the vehicle-mounted image acquisition device is: ,in , , , are respectively the scale factor of the vehicle-mounted image acquisition device in the u-axis direction in the pixel coordinate system, the scale factor of the vehicle-mounted image acquisition device in the v-axis direction in the pixel coordinate system, the number of horizontal pixels that differ between the central pixel coordinate of the image and the pixel coordinate of the image origin, and the number of vertical pixels that differ between the central pixel coordinate of the image and the pixel coordinate of the image origin. The external parameter matrix between the vehicle body and the vehicle-mounted image acquisition device is ,in ~ are the elements of the orthogonal rotation matrix, ~ are the elements of the translation matrix respectively, and the coordinate transformation relationship between the image pixels and the vehicle body is constructed as follows: Among them, u, v are the coordinates of the pixel coordinate point, w is the scale factor, , , They are the x, y, and z axis coordinates in the vehicle coordinate system respectively; Configure according to the distance h between the vehicle body and the ground , that is, zcamera=-h, and multiply the internal parameter matrix and the external parameter matrix to convert into: in, ~ are the elements in the matrix A obtained by multiplying the internal parameter matrix and the external parameter matrix respectively; The final coordinate transformation relationship between the image pixels and the vehicle body is obtained as follows: in, x,y They are the x and y axis coordinates in the vehicle coordinate system respectively.

6. The moving target trajectory prediction method based on visual transformation according to any one of claims 1 to 4, characterized in that: In step S3, the coordinates of the bottom center point of the 2D bounding box obtained by each trajectory prediction are projected from above using the coordinate transformation relationship to obtain the predicted trajectory coordinates of the moving target in the current frame corresponding to the vehicle body coordinate system, thereby completing the trajectory prediction of the moving target.

7. The moving target trajectory prediction method based on visual transformation according to any one of claims 1 to 4, characterized in that: The step S3 also includes predicting the position deviation between the coordinates of the moving target in the two frames before and after, obtaining the moving speed of the moving target, and calculating the moving speed according to the formula Estimate the motion speed of the next frame, where , They are the velocity components on the x-axis and y-axis in the next frame of motion speed, , They are the velocity components on the x-axis and y-axis of the current frame motion speed, is the preset threshold.

8. A moving target trajectory prediction device based on visual transformation, characterized in that: include: The detection module is used to acquire the road environment image collected by the vehicle-mounted image acquisition device in real time, detect the moving target, and obtain the 2D bounding box of the moving target; A prediction and tracking module, used to use the 2D bounding box of the moving target to predict the target trajectory and track the target to obtain the predicted coordinates of the moving target trajectory; A visual transformation module, used to transform the predicted coordinates of the moving target trajectory into the vehicle body coordinate system according to the coordinate transformation relationship between the image pixels and the vehicle body obtained in advance, so as to obtain the predicted coordinates of the moving target trajectory in the vehicle body coordinate system; In the visual transformation module, by taking the laser radar arranged on the vehicle as the reference coordinate system, the vehicle body and the laser radar, as well as the laser radar and the vehicle-mounted camera are jointly calibrated respectively to determine the change relationship between the vehicle body coordinate system and the laser radar coordinate system, and the laser radar coordinate system and the vehicle-mounted camera, and the external parameter matrix from the laser radar to the vehicle body and the external parameter matrix from the laser radar to the vehicle-mounted camera are obtained by joint calibration. The external parameter matrix from the laser radar to the vehicle body obtained by the joint calibration is inverted and multiplied with the external parameter matrix from the laser radar to the vehicle-mounted camera to obtain the external parameter transformation matrix from the vehicle body to the vehicle-mounted camera. According to the internal parameter matrix of the vehicle-mounted camera and the external parameter transformation matrix from the vehicle body to the vehicle-mounted camera, and by setting the height of the coordinate system of the vehicle body to a fixed value h ,according to The coordinate transformation relationship between the image pixels and the vehicle body is constructed, where ~ are the elements of matrix A obtained by multiplying the internal parameter matrix and the external parameter matrix, u and v are the coordinates of the pixel points, x,y They are the x and y axis coordinates in the vehicle coordinate system respectively.

9. A computer device comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program, wherein: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 7.

Images