A method and apparatus for determining a motion trajectory of a target object

Abstract

The application discloses a target object motion trajectory determination method and device, determines each target object based on a current video frame and a previous video frame collected; for any target object, determines geometric information of the target object according to image information of the target object in the current video frame; determines a current spatial position of the target object through consistency between projection under a spatial coordinate system and position under a camera coordinate system of the geometric information; and determines a predicted spatial position of the target object at a subsequent moment according to the current spatial position and a historical spatial position of the target object. The method can accurately track a visual scene target trajectory based on image time sequence texture correlation and a road surface target kinematic model, and can accurately predict a motion trajectory of a target in a specific time segment in the future through a time sequence long short-term memory network based on a three-dimensional spatial position and a motion relationship of the target to be predicted and surrounding targets, thereby providing an important basis for subsequent driving behavior decision of a vehicle.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and apparatus for determining the motion trajectory of a target object. Background Technology

[0002] Intelligentization is one of the key trends in the automotive industry today, with applications covering a variety of complex driving scenarios, including highways, cities, and parking. In terms of traffic scenarios, accurately tracking and predicting the trajectory of vehicles can effectively solve some of the "long-tail problems" that exist in various intelligent driving systems, such as lane changes, emergency braking, and pedestrian crossings. Summary of the Invention

[0003] This application provides a method and apparatus for determining the motion trajectory of a target object, which is used to accurately track and predict the motion trajectory of the target object, thereby improving the intelligence and safety of the vehicle during driving.

[0004] In a first aspect, embodiments of this application provide a method for determining the motion trajectory of a target object. The method includes: determining each target object based on a current video frame and previous video frames; for any target object, determining its geometric information based on its image information in the current video frame; determining the current spatial position of the target object at the current time by using the consistency relationship between the projection of the geometric information in a spatial coordinate system and its position in a camera coordinate system; and determining the predicted spatial position of the target object at subsequent times based on its current spatial position and its historical spatial position.

[0005] In the above scheme, for scenarios where vehicles are traveling on the road, following vehicles collect road information ahead using their onboard equipment. By analyzing the currently collected road information and road information collected at a slightly earlier time, various target objects with a driving relationship to the vehicle can be identified. Then, for any target object, its geometric information can be determined based on its image information in the current video frame. Subsequently, by projecting the geometric information into the spatial coordinate system and considering the consistency between its position in the camera coordinate system, the target object's current spatial position can be obtained. Finally, based on the target object's current spatial position and its spatial position at historical moments, its spatial position at subsequent moments can be predicted. This method, based on image temporal texture correlation and road target kinematics models, can accurately track the trajectory of targets in a visual scene. Based on the three-dimensional spatial position and motion relationship between the target to be predicted and its surrounding targets, a temporal long short-term memory network can accurately predict the target's motion trajectory within a specific future time segment, providing an important basis for subsequent vehicle driving behavior decisions.

[0006] In one possible implementation method, determining each target object based on the acquired current video frame and the preceding video frames includes: performing target detection on any acquired video frame; if the region overlap between a first target detection result in the current video frame and a second target detection result in a video frame acquired before the current video frame meets a first preset threshold, then the first target detection result is determined to be a target object; otherwise, the correlation matching confidence between a first feature description of the first target detection result and a second feature description of the second target detection result is determined; if the correlation matching confidence meets a second preset threshold, then the first target detection result is determined to be a target object. In the above scheme, by combining the region overlap of the same target detection result in the current video frame and the preceding video frame, as well as their correlation matching confidence, it is possible to accurately determine whether the same target detection result is indeed the same target object. This method, by accurately tracking the target object first, is of great significance for predicting the target object's trajectory in subsequent moments.

[0007] In one possible implementation, the relevance matching confidence is determined by a weighted loss function based on binary cross-entropy and a category center loss function of the target object's category.

[0008] In the above scheme, when determining the correlation matching confidence between the first feature description in the current video frame and the second feature description in the previous video frame based on the same target detection result, the correlation matching confidence between the first feature description and the second feature description is jointly determined by the binary cross-entropy weight loss function and the category center loss function of the target object's category. This is of great significance for accurately determining whether the same target detection result is really the same target object.

[0009] In one possible implementation, the target objects include vehicles; determining the geometric information of the target objects based on their image information in the current video frame includes: for any target vehicle among the target objects, processing the image information of the target vehicle through a first backbone and a second backbone of a convolutional neural network to obtain first geometric information and second geometric information of the target vehicle respectively; the first geometric information includes the rear information of the target vehicle, the orientation information of the target vehicle, and the position information of the key points of the target vehicle in the current video frame; the second geometric information includes the depth information and the size information of the target vehicle.

[0010] In the above scheme, when determining the geometric information of the target object based on the image information of the target object in the current video frame, considering the strong correlation between different target semantics, a geometric analysis network sharing backbone features is designed under the multi-task training framework. Among them, the vehicle's tail information, vehicle's orientation information, and vehicle's key point information only contain two-dimensional information, so their corresponding sub-networks can be controlled to share the same backbone. Since the vehicle's depth information and size information contain three-dimensional information, their corresponding sub-networks can be controlled to share the same backbone, thus improving the extraction efficiency of the same type of geometric information.

[0011] In one possible implementation method, determining the current spatial position of the target object at the current video frame time by using the consistency relationship between the geometric information projected in the spatial coordinate system and the position in the camera coordinate system includes: obtaining the rear feature line of the target vehicle based on the rear information of the target vehicle, obtaining the orientation feature line of the target vehicle based on the orientation information of the target vehicle; and obtaining the current spatial position of the target vehicle at the current video frame time by using the corresponding correction weights in the energy function based on the rear feature line of the target vehicle, the orientation feature line of the target vehicle, the position information of the key points of the target vehicle in the current video frame, and the size information of the target vehicle.

[0012] In the above scheme, after acquiring several types of geometric information about the target vehicle, since this geometric information belongs to the planar category, it needs to be converted into spatial information to provide effective computational data for subsequent prediction of the target vehicle's trajectory. In this process, by using the vehicle's rear end, orientation, key points, and dimensions, combined with weights that correct for each data point, the vehicle's current spatial position can be accurately defined.

[0013] In one possible implementation method, the energy function is determined by: for a sample vehicle, determining the coordinates of each vehicle vertex of the sample vehicle in a spatial coordinate system constructed based on the center point of the bottom surface of the sample vehicle; obtaining the projection point coordinates of each vehicle vertex of the sample vehicle in the camera coordinate system based on the coordinates of each vehicle vertex of the sample vehicle in the spatial coordinate system; determining each set of correspondences between the projection point coordinates of each vehicle vertex and the coordinates of each vehicle vertex in the camera coordinate system; for the same vehicle vertex of the sample vehicle indicated by each set of correspondences, determining the energy function based on the deviation of the same vehicle vertex, the size deviation of the sample vehicle, the orientation line deviation of the sample vehicle, and the rear deviation of the sample vehicle.

[0014] In the above scheme, by pre-determining the energy function based on the sample vehicles, the spatial position information of the vehicles can be quickly obtained based on the vehicle's geometric information.

[0015] In one possible implementation method, determining the predicted spatial position of the target object at a subsequent time based on the target object's current spatial position and its historical spatial position includes: using a Long Short-Term Memory (LSTM) neural network to determine the predicted spatial position of the target object at a subsequent time, taking the target object's spatial motion trajectory, the spatial motion trajectories of surrounding objects, and lane line information; the target object's spatial motion trajectory is determined by the target object's current spatial position and its historical spatial position; the surrounding objects' spatial motion trajectories are determined by the surrounding objects' current spatial positions and their historical spatial positions; the surrounding objects are those among the target objects that satisfy a distance threshold with the target object.

[0016] In the above scheme, by using LSTM to process the spatial motion trajectory of the target object itself, the spatial motion trajectory of the surrounding objects, and the lane line information, the spatial position of the target object at subsequent times can be predicted more accurately.

[0017] In one possible implementation, the energy function is:

[0018]

[0019] in, Represents the free variables to be optimized. Represents the i-th vertex The x-coordinate or y-coordinate of the projected point in the image. This represents the corresponding coordinate values ​​obtained from the 2D detection results based on the correspondence relationship. This represents the prior values ​​of the vehicle's length, width, and height obtained from the neural network. This represents the straight line parameters of the orientation line determined based on key points and target orientation, where Let G represent the unit normal vector of the line, and let G represent the set of target points on the line. Points corresponding to the feature lines of the rear of the car The x-coordinate of the projection point in the image. This represents the x-coordinate of the feature line at the rear of the vehicle, determined by the neural network. These represent the weighting coefficients for the coordinate projection error term, the vehicle size error term, the orientation line error, and the rear error, respectively.

[0020] Secondly, embodiments of this application provide a motion trajectory determination device for a target object. The device includes: a target object determination unit, configured to determine each target object based on a current video frame and preceding video frames; and a motion trajectory prediction unit, configured to, for any target object, determine its geometric information based on image information of the target object in the current video frame; determine the current spatial position of the target object at the current time by using the consistency relationship between the projection of the geometric information in a spatial coordinate system and its position in a camera coordinate system; and determine the predicted spatial position of the target object at subsequent times based on its current spatial position and its historical spatial position. The energy function is determined by the consistency between the projection of the geometric information of the same object in a spatial coordinate system and its position in a camera coordinate system.

[0021] Thirdly, embodiments of this application provide a computing device, including:

[0022] Memory, used to store computer programs;

[0023] A processor is configured to invoke a computer program stored in the memory and execute the method described in any of the first aspects according to the obtained program.

[0024] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program for causing a computer to perform the method described in any of the first aspects. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram illustrating a method for determining the motion trajectory of a target object according to an embodiment of this application;

[0027] Figure 2 A schematic diagram of a convolutional neural network provided in an embodiment of this application;

[0028] Figure 3 A schematic diagram illustrating the relative positional relationship between a target coordinate system and a camera coordinate system, provided for an embodiment of this application;

[0029] Figure 4 A schematic diagram illustrating the construction of a target coordinate system provided in an embodiment of this application;

[0030] Figure 5 A schematic diagram of a target object motion trajectory determination device provided in an embodiment of this application;

[0031] Figure 6 This is a schematic diagram of a computing device provided in an embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] The current lack of tracking of targets on the road ahead and prediction of their trajectories greatly contributes to the "long tail problem" during driving.

[0034] To address the aforementioned technical problems, this application provides a method for determining the trajectory of a target object. Based on this method, following vehicles can accurately track the target object on the road surface and predict its trajectory, greatly facilitating their timely planning and adjustment of their driving progress.

[0035] like Figure 1 The diagram shown is a schematic representation of a method for determining the motion trajectory of a target object according to an embodiment of this application. The method includes the following steps:

[0036] Step 101: Determine each target object based on the current video frame and the preceding video frames.

[0037] Optionally, the current video frame and the previous video frame are captured by the vehicle in motion, specifically by video capture equipment installed on the vehicle; each target object is a target object that has a driving relationship with the vehicle that captures the video frame.

[0038] The vehicle's movement is not limited to a continuously moving state, but also includes a temporarily stopped state, such as waiting at a traffic light. By installing video capture equipment on the vehicle, images can be captured to form a video stream. Since objects in front of the vehicle are more instructive from a driving perspective than the vehicle itself, controlling the video capture equipment to capture images of these objects and form a video stream is crucial for guiding safe driving. In this application, by controlling the video capture equipment to capture images of objects in front of the vehicle in real time, and based on the current video frame and all preceding video frames in the captured video stream, target objects related to the vehicle's movement can be identified. The current video frame is simply called the current video frame, and all preceding video frames are called the pre-current video frames.

[0039] Step 102: For any target object among the target objects, determine the geometric information of the target object based on the image information of the target object in the current video frame; determine the current spatial position of the target object at the current time frame by using the consistency relationship between the projection of the geometric information in the spatial coordinate system and the position in the camera coordinate system; determine the predicted spatial position of the target object at subsequent time points based on the current spatial position and the historical spatial position of the target object; the energy function is determined by the consistency between the projection of the geometric information of the same object in the spatial coordinate system and the position in the camera coordinate system.

[0040] In this step, taking one target object as an example, its geometric information can be determined based on its image information in the current video frame. Then, by projecting this information into the spatial coordinate system and verifying the consistency between its position in the camera coordinate system, the target object's current spatial position can be obtained. Finally, based on the target object's current spatial position and its historical spatial positions, its spatial position in subsequent moments can be predicted. This method allows following vehicles to accurately track target objects on the road and predict their trajectories, greatly facilitating their timely planning and adjustment of their driving progress.

[0041] This method is based on image temporal texture correlation and road surface target kinematic model, which can accurately track the target trajectory in visual scene; based on the three-dimensional spatial position and motion relationship between the target to be predicted and its surrounding targets, the temporal long short-term memory network can accurately predict the target's motion trajectory in a specific time segment in the future, providing an important basis for subsequent vehicle driving behavior decisions.

[0042] The following will provide a detailed explanation of some of the steps mentioned above, with examples.

[0043] In one implementation of step 101 above, determining each target object based on the current video frame and the preceding video frames includes: performing target detection on any video frame; if the region overlap between the first target detection result in the current video frame and the second target detection result in the previous video frame (whose acquisition time is before the current video frame) satisfies a first preset threshold, then the first target detection result is determined to be a target object; otherwise, the correlation matching confidence between the first feature description of the first target detection result and the second feature description of the second target detection result is determined; if the correlation matching confidence satisfies a second preset threshold, then the first target detection result is determined to be a target object. For example, for a video stream captured by a video capture device installed in a vehicle, embodiments of this application can apply a target detection algorithm to each video frame in the video stream in real time. Through the target detection algorithm, each object in any video frame can be effectively detected, and each object is a target detection result. The target detection algorithm in this application can be any target detection algorithm currently used in the industry, such as the YOLO algorithm; this application does not limit or elaborate on it.

[0044] In order to reduce the amount of data calculation and improve the efficiency of data processing, this application tracks the detection results of the same target in the two video frames (i.e., the current video frame) and the video frame preceding the current video frame, to determine whether the same target detection results identified by the target detection algorithm in the two video frames are indeed the same target detection results. The specific details are as follows:

[0045] First, an object detection algorithm is applied to the current video frame. Let's say the algorithm detects a vehicle X in the current video frame, and let the region where vehicle X is located be region X. Furthermore, by applying a target detection algorithm to the previous video frame of the current video frame, let's assume that the target detection algorithm detects a vehicle in the previous video frame, denoted as vehicle X', and let the region where vehicle X' is located be region X'. Therefore, to determine whether vehicle X in the current video frame is the same as vehicle X' in the previous video frame, we can first calculate it using the region overlap calculation method, that is, calculate the region overlap. and region overlap According to the degree of overlap Match the corresponding target. Among these, the overlap rate... The calculation formula is as follows:

[0046]

[0047] Among them, if If the value is greater than the preset threshold, it indicates that the temporally adjacent targets have been successfully matched, such as determining that vehicle X in the current video frame is the same as vehicle X' in the previous video frame; otherwise, it indicates that the target matching has failed, such as determining that vehicle X in the current video frame and vehicle X' in the previous video frame are not the same vehicle.

[0048] Next, in the case where suspected detection results of the same target are confirmed as matching failures in two adjacent video frames by the above-mentioned region overlap calculation method, the following method can be used to recalculate to determine whether the two are indeed the same target detection results.

[0049] In some embodiments of this application, the relevance matching confidence is determined by a weighted loss function based on binary cross-entropy and a category center loss function of the category to which the target object belongs.

[0050] For example, for suspected same target detection results that failed to match based on region overlap calculation, the feature descriptions used in target detection can be reused for temporal feature correlation matching. A bilinear interpolation feature pooling module can be used to extract and concatenate features from target regions corresponding to preceding and following video frames to obtain the target correlation matching confidence score, while simultaneously outputting refined target bounding box regression results. During training, an adaptive weight loss based on binary cross-entropy is employed. and category center distance loss The loss function for controlling the confidence level of the correlation between the constraint target and the target is as follows:

[0051]

[0052] in, This represents a hyperparameter, which can be used to adjust the weights of the two losses; This represents adaptive weight loss, which can be controlled in a manner similar to Focal Loss to manage the loss ratio between easy and difficult samples; This represents the class center distance loss, which can be used to effectively reduce the intra-class distance of multiple targets.

[0053] Furthermore, for unobstructed targets that can be successfully matched using either of the above two methods and whose bounding box size falls within a preset threshold range, this application can utilize kinematic-based trackless Kalman filtering for target tracking in image coordinates. Specifically, the trackless Kalman filter updates the target trajectory based on a specific kinematic prior model and corrects the trajectory according to a preset observation equation. The state-space equation used in this process is as follows:

[0054]

[0055]

[0056] Where x is the system state variable, including the vehicle's position and position change information, i.e. y represents system observations, including vehicle location information, i.e. ω and ρ represent process noise and measurement noise, respectively.

[0057] Based on the above method, the overlap of suspected identical target detection results in adjacent video frames is first calculated. Since the acquisition frequency of video acquisition devices is generally high, this method of calculating overlap is effective for tracking targets with small displacement changes, such as pedestrians on the road. However, for targets with large displacement changes in a short period of time, such as vehicles encountering emergency braking, this method of tracking targets using overlap is not effective. Therefore, this application improves the accuracy of target tracking in this scenario by calculating the correlation matching confidence of suspected identical target detection results.

[0058] In one implementation of step 102 above, the target objects include vehicles; determining the geometric information of the target objects based on the image information of the target objects in the current video frame includes: for any target vehicle among the target objects, processing the image information of the target vehicle through the first and second backbones of a convolutional neural network respectively to obtain the first geometric information and the second geometric information of the target vehicle respectively; the first geometric information includes the tail information of the target vehicle, the orientation information of the target vehicle, and the position information of the key points of the target vehicle in the current video frame, and the second geometric information includes the depth information and the size information of the target vehicle.

[0059] For example, continuing the previous example, by calculating the region overlap or correlation matching confidence of suspected same target detection results in the current video frame and the previous video frame of the video stream captured by the video acquisition device, for target detection results that meet the region overlap requirement or the correlation matching confidence requirement, based on the target bounding box of the target detection result in the current video frame, by using... Figure 2 The convolutional neural network shown performs target geometry analysis, where, Figure 2 This is a schematic diagram of a convolutional neural network provided in an embodiment of this application, thereby obtaining the corresponding geometric semantics and depth information. Wherein:

[0060] (1) When the target object is a vehicle, its geometric semantic analysis may include the following geometric semantic information: rear of the vehicle, orientation, projection position of vehicle key points in the image, projection position of vehicle center point in the image, vehicle center depth, and vehicle size. Among them, vehicle key points may include 22 key points on the vehicle, namely: 1. Left front wheel point; 2. Right front wheel point; 3. Left rear wheel point; 4. Right rear vehicle point; 5. Left front headlight; 6. Right front headlight; 7. Center point of front license plate; 8. Left front vehicle vertex; 9. Right front vehicle vertex; 10. Left rear vehicle vertex; 11. Right rear vehicle vertex; 12. Left rear headlight; 13. Right rear headlight; 14. Left rear fog light; 15. Right rear fog light; 16. Left rear panel; 17. Right rear panel; 18. Rear vehicle logo; 19. Center point of rear license plate; 20. Front vehicle logo; 21. Left rearview mirror; 22. Right rearview mirror.

[0061] Considering the strong correlation between different geometric semantics, this application can design a geometric analysis network with shared backbone features under a multi-task training framework. Since the rear frame, key points, orientation, and center point only contain two-dimensional information, their corresponding subnetworks will share the same backbone; vehicle depth and vehicle size, because they contain three-dimensional information, will also have their corresponding subnetworks share the same backbone.

[0062] (2) Depth information calculation adopts an unsupervised depth prediction technique based on video stream structure and motion recovery. It is assumed that there exists a current frame... and the next frame The depth network output includes camera output from... Frame to Frame rotation matrix Translation amount and Scene depth of a frame In the known , and Based on this, it can be used to linearly interpolate from Frame recovery Frame, after recovery Frame is denoted as .by and The difference between two frames is used to establish a loss function to achieve unsupervised training of the depth network. During training, scene semantic analysis is incorporated to remove moving targets in the scene. Furthermore, by jointly training the scene depth subnetwork, target depth and target size subnetworks, the depth depth subnetwork can successfully predict the absolute depth of the scene by leveraging the inherent absolute size information of the target depth and target size.

[0063] In one implementation of step 102 above, determining the current spatial position of the target object at the current video frame time by using the consistency relationship between the projection of the geometric information in the spatial coordinate system and the position in the camera coordinate system includes: obtaining the rear feature line of the target vehicle based on the rear information of the target vehicle, obtaining the orientation feature line of the target vehicle based on the orientation information of the target vehicle; and obtaining the current spatial position of the target vehicle at the current video frame time by using the corresponding correction weights in the energy function based on the rear feature line of the target vehicle, the orientation feature line of the target vehicle, the position information of the key points of the target vehicle in the current video frame, and the size information of the target vehicle.

[0064] In some embodiments of this application, the energy function is determined by the following methods: for a sample vehicle, determining the coordinates of each vehicle vertex of the sample vehicle in a spatial coordinate system constructed based on the center point of the bottom surface of the sample vehicle; obtaining the projection point coordinates of each vehicle vertex of the sample vehicle in the camera coordinate system based on the coordinates of each vehicle vertex of the sample vehicle in the spatial coordinate system; determining each set of correspondences between the projection point coordinates of each vehicle vertex and the coordinates of each vehicle vertex in the camera coordinate system; for the same vehicle vertex of the sample vehicle indicated by each set of correspondences, determining the energy function based on the deviation of the same vehicle vertex, the size deviation of the sample vehicle, the orientation line deviation of the sample vehicle, and the rear deviation of the sample vehicle.

[0065] For example, by abstracting the geometric information present in the target semantics, an energy function is constructed based on the consistency of geometric projection, and the three-dimensional information of the target object is solved by iterative optimization.

[0066] like Figure 3 The diagram illustrates the relative positional relationship between a target coordinate system and a camera coordinate system according to an embodiment of this application. The coordinate system identified by "c" represents the camera coordinate system, and the coordinate system identified by "o" represents the target coordinate system. Clearly, the coordinates of the same target object in the target coordinate system will differ from its coordinates in the camera coordinate system. Therefore, it is necessary to convert the target object's coordinates in the camera coordinate system to its coordinates in the target coordinate system. This is crucial for describing the spatial position information of the target object.

[0067] When the target object is a vehicle, there is no need to consider the vehicle's roll angle; that is, the vehicle will have a yaw angle in the camera coordinate system. , are constants The pitch angle and the roll angle are both 0.

[0068] like Figure 4The diagram shown is a schematic representation of the construction of a target coordinate system according to an embodiment of this application. The origin of the target coordinate system is fixed at the center of the bottom surface of the cuboid corresponding to the sample vehicle. For details regarding the orientation of the coordinate system and its relative position to the sample vehicle, please refer to [link to relevant documentation]. Figure 4 ,in, The axle points from the center of the bottom surface of the cuboid towards the front of the vehicle. The axle points from the center of the bottom surface of the cuboid to the right side of the vehicle. The axis points from the center of the bottom surface of the cuboid towards the chassis. Assume the sample vehicle's length, width, and height are respectively... Then the homogeneous coordinates of the 8 vertices of the cuboid corresponding to the sample vehicle are: , , , .

[0069] The coordinates of the projection points of the vertices of the cuboid in the image are: , .

[0070] in, and They are the first The depth and projection coordinates of each vertex in the camera coordinate system It's the camera's internal parameters. , , which are the coordinates of the center point of the base of the vehicle's corresponding cuboid in the camera coordinate system. It is the rotation matrix from the target coordinate system to the camera coordinate system, and its definition is as follows:

[0071] .

[0072] Based on the target orientation information, the correspondence between the 3D bounding box and the 2D bounding box is determined. Specifically, each of the four edges of the 2D bounding box corresponds to a vertex of the 3D bounding box. The resulting set of matching pairs is... Establish the following energy function:

[0073]

[0074] in, Represents the free variables to be optimized. Represents the i-th vertex The x-coordinate or y-coordinate of the projected point in the image. This represents the corresponding coordinate values ​​obtained from the 2D detection results based on the correspondence relationship. This represents the prior values ​​of the vehicle's length, width, and height obtained from the neural network. This represents the straight line parameters of the orientation line determined based on key points and target orientation, where Let G represent the unit normal vector of the line, and let G represent the set of target points on the line. Points corresponding to the feature lines of the rear of the car The x-coordinate of the projection point in the image. This represents the x-coordinate of the feature line at the rear of the vehicle, determined by the neural network. These represent the weighting coefficients for the coordinate projection error term, the vehicle size error term, the orientation line error, and the rear error, respectively.

[0075] In the above scheme, the coordinates of the eight vertices of the rectangular prism corresponding to the sample vehicle are projected and compared with the coordinates of the eight vertices in the image of the sample vehicle. A correspondence is established between vertices in the same orientation under both coordinate systems. Based on these eight established correspondences, the energy function is determined. Each of their respective parameter values, thus in parsing After obtaining the corresponding values, the geometric semantic information of the target object obtained through target tracking can be substituted into the energy function to solve for the three-dimensional spatial information of the target object's rear, orientation, and key points.

[0076] In one implementation of step 102 above, determining the predicted spatial position of the target object at a subsequent time based on the target object's current spatial position and its historical spatial position includes: using a Long Short-Term Memory (LSTM) neural network to determine the predicted spatial position of the target object at a subsequent time, taking the target object's spatial motion trajectory, the spatial motion trajectories of surrounding objects, and lane line information; the target object's spatial motion trajectory is determined by the target object's current spatial position and its historical spatial position; the surrounding objects' spatial motion trajectories are determined by the surrounding objects' current spatial positions and their historical spatial positions; the surrounding objects are those among the target objects that satisfy a distance threshold with the target object.

[0077] For example, continuing from the previous example, when the target object is a target vehicle, based on solving the current geometric information of the target vehicle, by combining it with a predetermined energy function, the current spatial position of the target vehicle can be obtained. Based on the same principle, the spatial position of the target vehicle within a certain time period before the current moment can be solved. Thus, based on the spatial position information of the target vehicle within a certain time period, the spatial motion trajectory of the target vehicle can be obtained.

[0078] For a target vehicle, other target objects within its set range, i.e., surrounding objects, may also affect the driving of vehicles behind it. Therefore, based on the same calculation logic as the target vehicle, the spatial movement trajectory of surrounding objects can be determined.

[0079] In addition, lane markings will also have a certain impact on the vehicle's driving process. For example, whether the road ahead changes from 4 lanes to 3 lanes, or whether there is a turn ahead, will affect the safe driving of the vehicle.

[0080] Based on the above analysis, this application comprehensively considers the spatial movement trajectory of the target object, the spatial movement trajectory of surrounding objects, and lane line information, including:

[0081] First, an LSTM encoder and a fully connected layer are used to encode the dynamic information of the target object. Then, an LSTM encoder and fine-grained convolutional feature descriptors are used to encode the target's position distribution and motion information within a specific distance range around the target object. Finally, an LSTM is used to encode the lane line information. These three encoded information are cascaded to construct a scene target trajectory encoding module.

[0082] Next, based on the feature encoding in the previous step, the probability distribution of the target object's longitudinal and lateral target motion behavior under the current conditions is obtained through the softmax operation;

[0083] Finally, combining the comprehensive information of the target object and its related target probability distributions, the LSTM decoding unit outputs the predicted motion trajectory of the target object.

[0084] Based on the same concept, embodiments of this application also provide a device for determining the motion trajectory of a target object, such as... Figure 5 The diagram shown is a schematic of a target object motion trajectory determination device provided in an embodiment of this application. The device includes a target object determination unit 501 and a motion trajectory prediction unit 502, wherein:

[0085] The target object determination unit 501 is used to determine each target object based on the current video frame and the preceding video frame.

[0086] The motion trajectory prediction unit 502 is used to determine the geometric information of any target object among the target objects based on the image information of the target object in the current video frame; determine the current spatial position of the target object at the current time by using the consistency relationship between the projection of the geometric information in the spatial coordinate system and the position in the camera coordinate system; and determine the predicted spatial position of the target object at subsequent times based on the current spatial position of the target object and the historical spatial position of the target object.

[0087] Furthermore, for this device, the target object determination unit 501 is specifically used for: performing target detection on any acquired video frame; if the region overlap between the first target detection result in the current video frame and the second target detection result in the previous video frame whose acquisition time is before the current video frame meets a first preset threshold, then the first target detection result is determined to be a target object; otherwise, the correlation matching confidence between the first feature description of the first target detection result and the second feature description of the second target detection result is determined; if the correlation matching confidence meets a second preset threshold, then the first target detection result is determined to be a target object.

[0088] Furthermore, for this device, the relevance matching confidence is determined by a weighted loss function based on binary cross-entropy and a category center loss function of the category to which the target object belongs.

[0089] Furthermore, for this device, the motion trajectory prediction unit 502 is specifically used to: for any target vehicle among the target objects, process the image information of the target vehicle through the first backbone and the second backbone of the convolutional neural network respectively to obtain the first geometric information and the second geometric information of the target vehicle respectively; the first geometric information includes the tail information of the target vehicle, the orientation information of the target vehicle, and the position information of the key points of the target vehicle in the current video frame; the second geometric information includes the depth information and the size information of the target vehicle.

[0090] Furthermore, for this device, the motion trajectory prediction unit 502 is specifically used to: obtain the rear feature line of the target vehicle based on the rear information of the target vehicle, and obtain the orientation feature line of the target vehicle based on the orientation information of the target vehicle; and obtain the current spatial position of the target vehicle at the current time of the current video frame based on the rear feature line of the target vehicle, the orientation feature line of the target vehicle, the position information of the key points of the target vehicle in the current video frame, and the size information of the target vehicle, through the corresponding correction weights in the energy function.

[0091] Furthermore, the device also includes an energy function determination unit 503; the energy function determination unit 503 is used to: determine the coordinates of each vehicle vertex of the sample vehicle in a spatial coordinate system constructed based on the center point of the bottom surface of the sample vehicle for the sample vehicle; obtain the projection point coordinates of each vehicle vertex of the sample vehicle in the camera coordinate system based on the coordinates of each vehicle vertex of the sample vehicle in the spatial coordinate system; determine each set of correspondences between the projection point coordinates of each vehicle vertex and the coordinates of each vehicle vertex in the camera coordinate system; and for the same vehicle vertex of the sample vehicle indicated by each set of correspondences, determine the energy function based on the deviation of the same vehicle vertex, the size deviation of the sample vehicle, the orientation line deviation of the sample vehicle, and the rear deviation of the sample vehicle.

[0092] Furthermore, for this device, the motion trajectory prediction unit 502 is specifically used to: determine the predicted spatial position of the target object at subsequent times by using a Long Short-Term Memory (LSTM) neural network to analyze the spatial motion trajectory of the target object, the spatial motion trajectories of the surrounding objects, and lane line information; the spatial motion trajectory of the target object is determined by the current spatial position and the historical spatial position of the target object; the spatial motion trajectory of the surrounding objects is determined by the current spatial position and the historical spatial position of the surrounding objects; the surrounding objects are the target objects that satisfy a distance threshold with respect to the target object.

[0093] This application also provides a computing device, which may specifically be a desktop computer, portable computer, smartphone, tablet computer, personal digital assistant (PDA), etc. The computing device may include a central processing unit (CPU), memory, input / output devices, etc. Input devices may include a keyboard, mouse, touchscreen, etc., and output devices may include display devices, such as liquid crystal displays (LCDs) and cathode ray tubes (CRTs).

[0094] The memory may include read-only memory (ROM) and random access memory (RAM), and provides the processor with program instructions and data stored in the memory. In the embodiments of this application, the memory may be used to store program instructions for a method of determining the motion trajectory of a target object;

[0095] The processor is used to call the program instructions stored in the memory and execute the method for determining the motion trajectory of the target object according to the obtained program.

[0096] like Figure 6 The diagram shown is a schematic representation of a computing device provided in an embodiment of this application. The computing device includes:

[0097] The processor 601, memory 602, transceiver 603, and bus interface 604 are included; wherein the processor 601, memory 602, and transceiver 603 are connected via bus 605.

[0098] The processor 601 is used to read the program in the memory 602 and execute the above-mentioned method for determining the motion trajectory of the target object;

[0099] Processor 601 can be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. It can also be a hardware chip. The aforementioned hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0100] The memory 602 is used to store one or more executable programs and can store data used by the processor 601 when performing operations.

[0101] Specifically, the program may include program code, which includes computer operation instructions. Memory 602 may include volatile memory, such as random-access memory (RAM); memory 602 may also include non-volatile memory, such as flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 602 may also include combinations of the above types of memory.

[0102] Memory 602 stores the following elements: executable modules or data structures, or subsets thereof, or extended sets thereof:

[0103] Operation instructions: This includes various operation instructions used to perform various operations.

[0104] Operating system: includes various system programs used to implement various basic business functions and handle hardware-based tasks.

[0105] The 605 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0106] Bus interface 604 can be a wired communication interface, a wireless bus interface, or a combination thereof. The wired bus interface can be, for example, an Ethernet interface. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. The wireless bus interface can be a WLAN interface.

[0107] This application also provides a computer-readable storage medium storing computer-executable instructions for causing a computer to execute a method for determining the motion trajectory of a target object.

[0108] Those skilled in the art will understand that embodiments of this application can be provided as methods or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0113] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for determining the motion trajectory of a target object, characterized in that, include: Based on the current video frame and the preceding video frames, each target object is determined. For any target object among the target objects, the geometric information of the target object is determined based on the image information of the target object in the current video frame; the geometric information is used to determine the current spatial position of the target object at the current time of the current video frame by the consistency relationship between the projection in the spatial coordinate system and the position in the camera coordinate system. The current spatial position of the target object includes the three-dimensional spatial information of the target object's rear, orientation, and key points. Based on the current spatial location and the historical spatial location of the target object, determine the predicted spatial location of the target object at subsequent times; The step of determining each target object based on the current video frame and the preceding video frames includes: performing target detection on any video frame; if the region overlap between the first target detection result in the current video frame and the second target detection result in the previous video frame (whose acquisition time is before the current video frame) meets a first set threshold, then the first target detection result is determined to be a target object; otherwise, a bilinear interpolation feature pooling module is used to extract features from the first target detection result and the second target detection result and concatenate them to determine the correlation matching confidence of the first feature description of the first target detection result and the second feature description of the second target detection result; if the correlation matching confidence meets a second set threshold, then the first target detection result is determined to be a target object. The step of determining the current spatial position of the target object at the current video frame by projecting the geometric information into the spatial coordinate system and determining the consistency between the position in the camera coordinate system and the spatial coordinate system includes: obtaining the rear feature line of the target vehicle based on the rear information of the target vehicle, and obtaining the orientation feature line of the target vehicle based on the orientation information of the target vehicle; and obtaining the current spatial position of the target vehicle at the current video frame by using the rear feature line of the target vehicle, the orientation feature line of the target vehicle, the position information of the key points of the target vehicle in the current video frame, and the size information of the target vehicle, through the corresponding correction weights in the energy function.

2. The method as described in claim 1, characterized in that, The relevance matching confidence is determined by a weighted loss function based on binary cross-entropy and a category center loss function based on the category to which the target object belongs.

3. The method as described in claim 1, characterized in that, The target objects include vehicles; Determining the geometric information of the target object based on its image information in the current video frame includes: For any target vehicle among the target objects, the image information of the target vehicle is processed by the first backbone and the second backbone of the convolutional neural network to obtain the first geometric information and the second geometric information of the target vehicle respectively. The first geometric information includes the tail information of the target vehicle, the orientation information of the target vehicle, and the position information of the key points of the target vehicle in the current video frame. The second geometric information includes the depth information and the size information of the target vehicle.

4. The method as described in claim 1, characterized in that, The energy function is determined in the following ways, including: For a sample vehicle, determine the coordinates of each vehicle vertex in a spatial coordinate system constructed based on the center point of the bottom surface of the sample vehicle; Based on the coordinates of each vehicle vertex of the sample vehicle in the spatial coordinate system, the projection point coordinates of each vehicle vertex of the sample vehicle in the camera coordinate system are obtained; and the correspondence between the projection point coordinates of each vehicle vertex and the coordinates of each vehicle vertex in the camera coordinate system is determined. For the same vehicle vertex of the sample vehicle indicated by each set of correspondences, the energy function is determined based on the deviation of the same vehicle vertex, the size deviation of the sample vehicle, the orientation line deviation of the sample vehicle, and the rear deviation of the sample vehicle.

5. The method as described in claim 1, characterized in that, Determining the predicted spatial position of the target object at subsequent times based on its current spatial position and historical spatial position includes: The spatial trajectory of the target object, the spatial trajectories of the surrounding objects, and lane line information are used to determine the predicted spatial position of the target object at subsequent time points through a Long Short-Term Memory (LSTM) neural network. The spatial trajectory of the target object is determined by its current spatial position and its historical spatial position. The spatial trajectories of the surrounding objects are determined by their current spatial positions and their historical spatial positions. The surrounding objects are those that meet a distance threshold from the target object.

6. A device for determining the motion trajectory of a target object, characterized in that, include: The target object determination unit is used to determine each target object based on the current video frame and the preceding video frame. The motion trajectory prediction unit is used to determine the geometric information of any target object among the target objects based on the image information of the target object in the current video frame; and to determine the current spatial position of the target object at the current time of the current video frame by using the consistency relationship between the projection of the geometric information in the spatial coordinate system and the position in the camera coordinate system. The current spatial position of the target object includes the three-dimensional spatial information of the target object's rear, orientation, and key points. Based on the current spatial location and the historical spatial location of the target object, determine the predicted spatial location of the target object at subsequent times; The step of determining each target object based on the current video frame and the preceding video frames includes: performing target detection on any video frame; if the region overlap between the first target detection result in the current video frame and the second target detection result in the previous video frame (whose acquisition time is before the current video frame) meets a first set threshold, then the first target detection result is determined to be a target object; otherwise, a bilinear interpolation feature pooling module is used to extract features from the first target detection result and the second target detection result and concatenate them to determine the correlation matching confidence of the first feature description of the first target detection result and the second feature description of the second target detection result; if the correlation matching confidence meets a second set threshold, then the first target detection result is determined to be a target object. The step of determining the current spatial position of the target object at the current video frame by projecting the geometric information into the spatial coordinate system and determining the consistency between the position in the camera coordinate system and the spatial coordinate system includes: obtaining the rear feature line of the target vehicle based on the rear information of the target vehicle, and obtaining the orientation feature line of the target vehicle based on the orientation information of the target vehicle; and obtaining the current spatial position of the target vehicle at the current video frame by using the rear feature line of the target vehicle, the orientation feature line of the target vehicle, the position information of the key points of the target vehicle in the current video frame, and the size information of the target vehicle, through the corresponding correction weights in the energy function.

7. A computer device, characterized in that, include: Memory, used to store computer programs; A processor is configured to invoke a computer program stored in the memory and execute the method as described in any one of claims 1-5 according to the obtained program.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-5.

Images