A vehicle trajectory recovery method, device and computer equipment in an occlusion state

Abstract

The application relates to a vehicle trajectory recovery method, device and computer equipment in a shielding state, wherein the vehicle trajectory recovery method in the shielding state comprises the following steps: acquiring a target video frame of a vehicle driving shot by a UAV, clustering lane pixel coordinates in the target video frame to obtain an intersection, taking the intersection as a node, and constructing a lane topology graph according to the connection relationship between the intersections; in the lane topology graph, at least two to-be-bound lanes are determined according to the driving state of the vehicle, the binding score of the vehicle trajectory and each to-be-bound lane is calculated, and the binding lane corresponding to the vehicle trajectory is determined according to the binding score; when the UAV detects that the vehicle is shielded, at least two candidate trajectories are generated according to the binding lane and the lane information in the lane topology graph, a target trajectory is determined in the candidate trajectories through a decision model, and if there are at least two target trajectories and the target trajectories meet a preset separation condition, the target trajectories are fused to obtain a complete trajectory. Through the application, the accuracy of trajectory recovery is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of vehicle trajectory, and particularly relates to a vehicle trajectory recovery method and device in a shielding state and a computer device. BACKGROUND

[0002] In the process of tracking a vehicle by a UAV, the vehicle may be shielded by buildings, trees and other obstacles, which may cause the tracking target to be lost in the tracking process. In the process of losing the tracking target, the UAV may predict the trajectory of the vehicle to identify the shielded tracking target again. However, in the shielding process, if the tracking target changes the driving direction or changes lanes, the actual driving trajectory of the vehicle may be obviously different from the predicted trajectory, so that the accuracy of the UAV in re-identifying the tracking target after shielding is reduced. Therefore, how to improve the accuracy of the UAV in predicting the trajectory of the vehicle after the vehicle is shielded has become a problem to be solved. SUMMARY

[0003] Embodiments of the present application provide a vehicle trajectory recovery method and device in a shielding state and an electronic device to at least solve the problem of low accuracy of a UAV in predicting the trajectory of a vehicle after the vehicle is shielded in the related art.

[0004] In a first aspect, embodiments of the present application provide a vehicle trajectory recovery method in a shielding state. The method is applied to a UAV carrying a camera component, and the method comprises the following steps.

[0005] Obtaining a target video frame of a vehicle driving captured by the UAV, clustering lane pixel coordinates in the target video frame to obtain an intersection, and constructing a lane topology graph with the intersection as a node and a connection relationship between the intersections as an edge.

[0006] In the lane topology graph, determining at least two to-be-bound lanes according to a driving state of the vehicle, calculating a binding score of the vehicle trajectory and each to-be-bound lane, and determining a bound lane corresponding to the vehicle trajectory according to the binding score.

[0007] When the UAV detects that the vehicle is shielded, generating at least two candidate trajectories according to the bound lane and lane information in the lane topology graph, and determining a target trajectory in the candidate trajectories through a decision model.

[0008] If there are at least two target trajectories, and the target trajectories meet a preset separation condition, fusing the target trajectories to obtain a complete trajectory.

[0009] In an embodiment, the driving state comprises position information, a heading angle and a driving speed of the vehicle, and the calculation of the binding score of the vehicle trajectory and each to-be-bound lane comprises:

[0010] For any one of the to-be-bound lane, according to the distance between the position information and the center of the to-be-bound lane, the distance feature is determined;

[0011] According to the lane direction of the to-be-bound lane and the heading angle, the angle feature is determined;

[0012] According to the lane speed limit of the to-be-bound lane and the driving speed, the speed feature is determined;

[0013] Based on the current time, the binding score of the vehicle and the corresponding bound lane at the last time is obtained, and the historical feature is determined;

[0014] Based on the distance feature, the angle feature, the speed feature and the historical feature, the binding score of the vehicle trajectory and the to-be-bound lane is obtained.

[0015] In an embodiment, the to-be-bound lane corresponding to the vehicle trajectory is determined according to the binding score, further comprising:

[0016] If the binding score of each target video frame is greater than a first preset threshold in a first preset number of continuous first target video frames, an intermediate pairing set is obtained, and the intermediate pairing set is a pairing of the vehicle trajectory and the to-be-bound lane;

[0017] In the intermediate pairing set, if the binding score of each continuous second target video frame is greater than a second preset threshold in a second preset number of continuous second target video frames, the current to-be-bound lane is determined as the bound lane of the vehicle, wherein the first preset number is less than the second preset number, and the first preset threshold is less than the second preset threshold;

[0018] If the vehicle trajectory meets the lane changing condition, the binding relationship between the vehicle trajectory and the bound lane is released, and the vehicle is bound to the lane again.

[0019] In an embodiment, the lane pixel coordinates in the target video frame are clustered to obtain an intersection, and a lane topology graph is constructed with the intersection as a node and the connection relationship between the intersections as an edge, comprising:

[0020] The lane pixel coordinates are clustered according to a preset lane requirement to obtain a structured road network, and the intersection is determined in the structured road network according to a preset curvature requirement;

[0021] According to a preset lane design requirement, a lane topology graph is constructed with the intersection as a node and the connection relationship between the intersections as an edge.

[0022] In an embodiment, when the unmanned aerial vehicle detects that the vehicle is blocked, comprising:

[0023] In the third preset number of continuous video frames, a confidence degree of identifying the vehicle according to each frame is obtained, an average confidence degree is obtained according to the average confidence degree and a confidence degree of the last frame of the continuous video frames, and a confidence degree drop rate is obtained according to the average confidence degree and the confidence degree of the last frame of the continuous video frames.

[0024] If the unmanned aerial vehicle does not detect the vehicle in the continuous video frames, and the confidence degree drop rate is greater than a preset confidence degree threshold, the unmanned aerial vehicle detects that the vehicle is blocked.

[0025] In an embodiment, the generating at least two candidate trajectories according to the bound lane and lane information in the lane topology map comprises:

[0026] According to lane speed limit information and lane turning information in each lane in the lane topology map, a to-be-matched lane to which the vehicle can drive from the bound lane is determined.

[0027] A driving distance of the vehicle in a blocked time is obtained, and at least two candidate trajectories are generated according to the driving distance and the to-be-matched lane.

[0028] In an embodiment, the determining a target trajectory in the candidate trajectories by a decision model comprises:

[0029] For each candidate trajectory, a predicted speed of the vehicle in the candidate trajectory and a distance between the candidate trajectory and an adjacent lane are obtained.

[0030] According to an actual speed of the vehicle and the predicted speed, a kinematic score is obtained.

[0031] According to a distance between the candidate trajectory and a center of an adjacent lane, a topological consistency score is obtained.

[0032] When the vehicle reappears, feature and position information of the vehicle are obtained, a re-identification score is determined according to a similarity between the feature of the vehicle and a feature of the candidate trajectory, and a norm of the position information and position information of the candidate trajectory.

[0033] According to the kinematic score, the topological consistency score and the re-identification score, an optimization score is obtained.

[0034] The candidate trajectory with the highest optimization score is taken as the target trajectory.

[0035] In an embodiment, after the determining an optimization score according to the kinematic score, the topological consistency score and the re-identification score, the method further comprises:

[0036] if a difference between the optimization scores of at least two of the candidate trajectories is less than a preset difference threshold, the candidate trajectory of straight driving is taken as the target trajectory;

[0037] if any two of the candidate trajectories intersect, the candidate trajectory of the vehicle with a predicted speed greater than a preset speed threshold is taken as the target trajectory.

[0038] In an embodiment, when the preset separation condition is satisfied, the following configuration is met:

[0039] a difference between the end time and the start time of each target trajectory is within a preset time range;

[0040] the types of vehicles driving on different target trajectories are consistent.

[0041] In an embodiment, the fusing of the target trajectories to obtain a complete trajectory comprises:

[0042] obtaining a running speed of the vehicle before being blocked and a driving duration of the vehicle when being blocked;

[0043] obtaining a motion continuity score according to a predicted acceleration at the end of the target trajectory and an actual acceleration of the vehicle;

[0044] if the driving duration is within a driving duration range of the running speed and the motion continuity score meets a continuity requirement, determining a fusion order of the target trajectories, and fusing the target trajectories according to the fusion order by using the decision model to obtain a complete trajectory.

[0045] In a second aspect, an embodiment of the present application provides a vehicle trajectory recovery device in a blocked state. The device is applied to a UAV carrying a camera component. The device comprises:

[0046] a lane topology graph construction module, configured to obtain a target video frame of vehicle driving captured by the UAV, cluster lane pixel coordinates in the target video frame to obtain an intersection, and construct a lane topology graph with the intersection as a node and a connection relationship between the intersections as an edge;

[0047] a lane and vehicle trajectory binding module, configured to determine at least two to-be-bound lanes in the lane topology graph according to a driving state of the vehicle, calculate a binding score of the vehicle trajectory and each to-be-bound lane, and determine a binding lane corresponding to the vehicle according to the binding score;

[0048] a target trajectory determination module, configured to generate at least two candidate trajectories according to the binding lane and lane information in the lane topology graph when the UAV detects that the vehicle is blocked, and determine a target trajectory in the candidate trajectories by using a decision model.

[0049] fusing the target trajectories to obtain a complete trajectory if there are at least two target trajectories and the target trajectories satisfy a preset separation condition.

[0050] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the vehicle trajectory recovery method in a shielding state according to the first aspect.

[0051] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, having a computer program stored thereon, and the program is executable on a processor to implement the vehicle trajectory recovery method in a shielding state according to the first aspect.

[0052] The vehicle trajectory recovery method, device and computer device in a shielding state provided by the embodiments of the present application have at least the following technical effects.

[0053] The target video frames of vehicle driving are photographed by the unmanned aerial vehicle, and a lane topology graph is constructed. The vehicle trajectory is bound with the corresponding bound lane by calculating the binding score between the bound lane and the vehicle trajectory, and the vehicle trajectory is bound with the bound lane, physical constraints are provided between the vehicle trajectory and the bound lane, so that the vehicle trajectory is continuous in the shielding scene under the constraint condition of the lane information in the bound lane; when the vehicle is detected to be shielded, at least two candidate trajectories are generated according to the bound lane and the lane information in the lane topology graph, wherein the bound lane and the lane information provide lane geometric constraints and traffic rule constraints for generating candidate trajectories, so as to ensure that the candidate trajectories conform to the actual situation, so that the target trajectory in the candidate trajectory can ensure that the target trajectory is close to the actual trajectory of the vehicle, and the accuracy of the target trajectory is ensured. The target trajectories satisfying the preset separation condition are fused to obtain a complete trajectory consistent globally, so as to overcome the problem of trajectory fragmentation caused by frame-by-frame processing.

[0054] Details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0055] The drawings described herein are intended to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0056] Figure 1 is a flowchart of a vehicle trajectory recovery method in a shielding state according to an exemplary embodiment;

[0057] Figure 2 is a schematic diagram of vehicle and lane binding according to an example embodiment;

[0058] Figure 3 is a flowchart of determining a bound lane according to another example embodiment;

[0059] Figure 4 is a flowchart of binding a lane according to another example embodiment;

[0060] Figure 5 is a flowchart of a vehicle trajectory recovery method in an occlusion state according to another example embodiment;

[0061] Figure 6 is a flowchart of a vehicle trajectory recovery method in an occlusion state according to another example embodiment;

[0062] Figure 7 is a block diagram of a vehicle trajectory recovery apparatus in an occlusion state according to an example embodiment;

[0063] Figure 8 is a block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0064] In order to make the purposes, technical solutions, and advantages of the present application clearer, the present application is described and explained below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0065] It is obvious that the accompanying drawings in the following description are only some examples or embodiments of the present application, and those of ordinary skill in the art can apply the present application to other similar scenarios without creative labor on the basis of these drawings. In addition, it can be understood that although the efforts made in this development process can be complex and lengthy, some designs, manufacturing or production changes made on the basis of the technical content disclosed in the present application are only routine technical means for those of ordinary skill in the art related to the content disclosed in the present application, and should not be understood as insufficient disclosure of the present application.

[0066] Reference to an "example" in this application means that a particular feature, structure, or characteristic described in connection with the example can be included in at least one example of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same example, nor are they necessarily mutually exclusive of one another. As used in this application, the term "exemplary" is intended to mean serving as an instance or illustration. Any implementation having "exemplary" characteristics is preferred only insofar as it serves as an instance or illustration.

[0067] Unless otherwise defined, technical terms and scientific terms used in the present application shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. Unless otherwise defined, the terms "one", "a", "an", "the" and like terms refer to both the singular and the plural, unless the context clearly indicates otherwise. The terms "comprises", "comprising", "includes", "including" and the like are to be construed as open-ended terms, meaning that the comprising of steps or the inclusion of elements is not limited to the elements recited, but can include other steps or elements not expressly listed or inherent to such process, method, system, product or apparatus. The terms "connected", "coupled", "coupling" and like terms are not limited to direct or physical connections, but can include indirect or wireless connections. The term "plurality" means two or more. The term "and / or" describes association between name objects. For example, "A and / or B" can mean A alone, A and B together, or B alone. The character " / " generally means "or". The terms "first", "second", "third" and the like are used to distinguish like elements, not to designate a particular order.

[0068] In a first aspect, the embodiments of the present application provide a vehicle trajectory recovery method in an occlusion state, Figure 1 is a flow chart of a vehicle trajectory recovery method in an occlusion state according to an example embodiment, as shown in Figure 1 The vehicle trajectory recovery method in an occlusion state includes the following steps:

[0069] In step S101, a target video frame in which a vehicle travels is captured by a UAV, lane pixel coordinates in the target video frame are clustered to obtain an intersection, and a lane topology graph is constructed with the intersection as a node and a connection relationship between intersections as an edge.

[0070] The unmanned aerial vehicle captures the vehicle driving to obtain video frames of the unmanned aerial vehicle. In the video frames of the unmanned aerial vehicle, a Scale-Invariant Feature Transform (SIFT) feature of each frame is obtained, and a difference degree between each frame is obtained. The difference degree satisfies the following formula:

[0071]

[0072] wherein f t is a video frame at time t, f t-1 is a video frame at time t-1, Δ(f t , f t-1 ) is the difference degree, SIFT(f t ) is the SIFT feature of the video frame at time t, and SIFT(f t-1 ) is the SIFT feature of the video frame at time t-1.

[0073] If the difference between the continuous frames exceeds the preset difference threshold, the video frames of the unmanned aerial vehicle are captured according to the specific situation of the video frames of the unmanned aerial vehicle to obtain target video frames. For example, if the video frames of the unmanned aerial vehicle are 30 frames, the video frames of the unmanned aerial vehicle are captured to obtain 15 frames of target video frames.

[0074] The target video frames are obtained by judging that the difference between the continuous frames exceeds the preset difference threshold, so as to ensure that the scene is obviously changed before analysis.

[0075] Continuing to refer to step S101, the target video frames are processed by the double-branch hybrid neural network to obtain pixel coordinates of the lane lines. The pixel coordinates are scored for confidence, and the pixel coordinates with a confidence score greater than a preset score threshold are retained. The lane pixel coordinates are clustered according to preset lane requirements by a clustering algorithm to obtain a structural road network. The preset lane requirements include a length requirement of the lane lines and a lane obstacle setting. The length requirement of the lane lines specifically includes that the aggregated lane length is greater than or equal to a preset length threshold, so as to ensure that the lane length formed by clustering meets the actual requirement. The lane obstacle setting specifically includes an obstacle setting of the road due to construction and temporary changes of the lane, etc.

[0076] In the structural road network, the pixel point coordinates are sampled in the structural road network according to a preset interval distance and a same direction, a three-point curvature calculation is performed on the pixel coordinates, and the curvature is quickly obtained through the discrete pixel coordinates. The curvature is adaptively adjusted by a smoothing coefficient, so that the curvature retains the real characteristics and suppresses the noise. Based on all the curvatures, the extreme values of the curvatures are determined. The extreme values of the curvatures include a maximum value and a minimum value. The maximum value represents that the bending degree of the curve near the pixel coordinates is larger than that of the surrounding area, so that the maximum value can accurately identify the vertex of the bend and the entry point of the intersection. The minimum value represents that the bending degree of the pixel coordinates is smaller than that of the adjacent area. Therefore, the pixel coordinates are part of a straight line segment. A lower limit of the curvature is set, so that the curvature is greater than the lower limit of the curvature, and the lane satisfies the minimum bend radius. The three-point curvature calculation satisfies:

[0077]

[0078] where k is the curvature, (x １ ,y1), (x ２ ,y ２ ), and (x ３ ,y ３ ) are three different pixel coordinates.

[0079] After the maximum value and the minimum value of the curvature are determined, the Gaussian smoothing coefficient is dynamically adjusted to cope with the pixel coordinate jitter caused by the shaking of the unmanned aerial vehicle.

[0080] The extreme values of the curvature can accurately position the bend vertex and the intersection entry, and provide curvature continuity constraints for subsequent vehicle trajectory and lane binding. The dynamic adjustment of the Gaussian smoothing coefficient can reduce the error rate of constructing the lane topology graph, and provide curvature constraints for subsequent accurate trajectory generation to improve the rationality of the generated trajectory.

[0081] The pixel coordinates corresponding to the maximum value of the curvature are determined as first nodes, and the pixel coordinates corresponding to the minimum value are determined as second nodes. If there are at least three edges connected with adjacent pixel points at the intersection node, and the included angle between any two edges satisfies that it is greater than a first preset angle and less than a second preset angle, the first node satisfying the condition is determined as an intersection node. If the curvature change rate between the second node and the adjacent pixel points is less than a change threshold, the second node is an ordinary node.

[0082] The fitted curve is obtained by fitting the lane pixel coordinates. Equidistant sampling is performed in the fitted curve to obtain a plurality of sampling points. The Hausdorff distance between each adjacent sampling point is calculated. If the number of Hausdorff distances greater than the preset distance threshold reaches a preset proportion in all Hausdorff distances, the lane pixel point satisfies the first connection condition. In the fitted curve, a vector formed by adjacent lane pixel points is determined, and an angle formed by the lane pixel points is determined according to the vector. If the angle is less than a third preset angle, the lane pixel point satisfies the second connection condition, that is, the direction of the lane pixel point is the same direction.

[0083] When the lane pixel points simultaneously satisfy the first connection condition and the second connection condition, it is indicated that the lane pixel points can be connected. Therefore, the intersection node and the ordinary node can be connected. Further, based on the ordinary node and the intersection node, the connection relationship between the ordinary node and the intersection node is taken as an edge, and a lane topology graph is constructed according to a preset lane design requirement.

[0084] It should be noted that the preset lane design requirement includes a highway route design specification and a city road intersection design standard. The preset lane design requirement is determined according to the video frame captured by the unmanned aerial vehicle. If the scene of the video frame is a highway, the preset lane design requirement is the highway route design specification. If the scene of the video frame is a city road, the preset lane design requirement is the city road intersection design standard.

[0085] In addition, an update rule is established based on the content that changes the lane, such as road construction. The lane topology graph changes the edge according to the update rule to establish a real-time updated lane topology graph. Wherein, the updated lane topology graph satisfies:

[0086]

[0087]

[0088] Wherein:

[0089] G t is the lane topology graph at t moment, G t+1 is the lane topology graph at t+1 moment, represents the updated lane topology graph, ΔG is the incremental observation data, represents the detected new lane element, is when there is an overlapping area between the topology data at t moment and t+1 moment, the conflicting lane element is eliminated by the update rule to ensure that the updated topology graph is globally consistent, is the overlap rate of the lane topology graph at t moment and the incremental observation data, is the overlapping area of the lane topology graph at t moment and the incremental observation data; is the area of the lane topology graph at t moment, is an overlap threshold.

[0090] With reference back to step S101, the target video frame is obtained by the UAV, and the pixel coordinates of the lane line are obtained in the target video frame. The pixel coordinates are clustered to obtain a preliminary structure road network. The curvature is further judged in the structure road network to determine the bending degree, and the straight intersection of the lane and the intersection is determined according to the bending degree, so that the actual lane situation is truly reflected in the lane topology graph.

[0091] Step S102, in the lane topology graph, at least two to-be-bound lanes are determined according to the driving state of the vehicle, the binding score of the vehicle and each to-be-bound lane is calculated, and the binding lane corresponding to the vehicle trajectory is determined according to the binding score.

[0092] Figure 2 It is a schematic diagram of vehicle and lane binding according to an exemplary embodiment, as shown in Figure 2 As shown, the driving state of the vehicle is obtained by the vehicle detection module of the UAV. The driving state includes the position information, the heading angle and the driving speed of the vehicle. The position information of the vehicle is sent to the binding engine. The binding engine queries the lanes near the vehicle position in the lane topology graph according to the position information, and determines at least two to-be-bound lanes. The binding score of the vehicle trajectory and each to-be-bound lane is calculated, the binding lane corresponding to the vehicle trajectory is determined according to the binding score, the binding relationship is updated, and the running trajectory of the vehicle in the binding lane is recorded.

[0093] Wherein, the binding score specifically includes:

[0094] For any one to-be-bound lane, the distance feature is determined according to the distance between the position information and the center of the to-be-bound lane, so as to determine the proximity of the vehicle to the center of the to-be-bound lane. The distance feature satisfies the following formula:

[0095]

[0096] Wherein, f dist is the distance feature, V t .pos is the world coordinate of the center point of the vehicle, e.proj is the vertical projection point coordinate of the vehicle position on the center line of the lane, δ d is the distance tolerance, and δ d The value of δ d is half of the width of the to-be-bound lane.

[0097] According to the lane direction of the to-be-bound lane and the heading angle, the angle feature is determined to ensure that the heading of the vehicle is consistent with the specified direction of the lane according to the angle feature. The angle feature satisfies the following formula:

[0098]

[0099] Wherein, f angle is the score feature, V t.θ is a heading angle of the vehicle, e.θ proj is a tangent direction angle of the lane centerline at the vehicle projection point.

[0100] According to the lane speed limit and the driving speed of the lane to be bound, a speed feature is determined to ensure the matching degree of the driving speed of the vehicle and the lane speed limit according to the speed feature. The speed feature satisfies the following formula:

[0101]

[0102] wherein f speed is the speed feature, V t .v is the driving speed of the vehicle, and e.speed is the lane speed limit of the lane.

[0103] Based on the current time, the binding score of the vehicle trajectory and the corresponding bound lane at the last time is obtained, and a history feature is determined to provide a time continuity constraint. The history feature satisfies the following formula:

[0104]

[0105] wherein f history is the history feature, S(V t-1 ,e) is the binding score of the vehicle trajectory and the same lane at t-1 time, is a history confidence decay coefficient.

[0106] Based on the distance feature, the angle feature, the speed feature and the history feature, the binding score of the vehicle trajectory and the lane to be bound is obtained. Each feature is multiplied by the respective weight, and the multiplied results are added, which specifically satisfies the following formula:

[0107]

[0108] wherein S(V t ,e) is the binding score, W1 is the first weight, W2 is the second weight, W3 is the third weight, W4 is the fourth weight, f dist is the distance feature, f angle is the angle feature, f speed is the speed feature, and f history is the history feature.

[0109] It should be noted that when the turn signal is detected, the second weight is increased by a preset value to realize dynamic adjustment of the weight.

[0110] In one embodiment, the first weight is selected as 0.4, the second weight is selected as 0.3, the third weight is selected as 0.2, and the fourth weight is selected as 0.1, and the binding score is:

[0111]

[0112] The vehicle determines a binding score of the vehicle trajectory and each of the to-be-bound lanes while satisfying traffic rules. The traffic rules include a rule of not crossing a solid line. Figure 3 is a flowchart of determining a bound lane according to another exemplary embodiment. Figure 3 As shown, determining the bound lane of the vehicle trajectory according to the binding score includes:

[0113] If the binding score of each of the first target video frames in the first preset number of continuous first target video frames is greater than the first preset threshold, an intermediate pairing set is obtained, and the intermediate pairing set is a pairing of the vehicle trajectory and the to-be-bound lane.

[0114] In the first preset number of continuous first target video frames, the binding score of the vehicle trajectory and each of the to-be-bound lanes in each frame is obtained. If the binding score of the vehicle trajectory and the to-be-bound lane in the continuous video frames is greater than the first preset threshold, the combination of the vehicle trajectory and the to-be-bound lane under the binding score that meets the condition is retained to form an intermediate pairing set. In the intermediate pairing set, there are multiple combinations of the vehicle trajectory and the to-be-bound lane.

[0115] In one embodiment, the first preset number is 3, the first preset threshold is 0.6, the to-be-bound lane includes a first lane, a second lane, and a third lane, and the binding score of the vehicle trajectory and the first lane in the continuous three frames is 0.6, 0.7, and 0.8 in sequence, the binding score of the vehicle trajectory and the second lane in the continuous three frames is 0.8, 0.8, and 0.9 in sequence, and the binding score of the vehicle trajectory and the third lane in the continuous three frames is 0.3, 0.7, and 0.8 in sequence. Then, the vehicle trajectory and the first lane, and the vehicle and the second lane are retained.

[0116] In the intermediate pairing set, if the binding score of each of the second continuous target video frames in the second preset number of continuous second target video frames is greater than the second preset threshold, the current to-be-bound lane is determined as the bound lane of the vehicle trajectory, wherein the first preset number is less than the second preset number, and the first preset threshold is less than the second preset threshold.

[0117] On the basis of the intermediate pairing set, the second preset number of continuous second target video frames are detected. If the binding score of the vehicle trajectory and each of the to-be-bound lanes in each frame is obtained in the second preset number of continuous second target video frames. If the binding score of the vehicle trajectory and the to-be-bound lane in the continuous video frames is greater than the second preset threshold, the combination of the vehicle trajectory and the to-be-bound lane under the binding score that meets the condition is retained.

[0118] In another embodiment, the second preset number is 5, the second preset threshold is 0.8, and in the intermediate pairing set, the binding scores of the vehicle trajectory in the second target video frame and the first lane are 0.6, 0.7, 0.8, 0.6, and 0.8 in sequence, and the binding scores of the vehicle trajectory in the second target video frame and the second lane are 0.8, 0.8, 0.9, 0.9, and 0.8 in sequence. It can be determined that the second lane is the binding lane of the vehicle.

[0119] If the vehicle trajectory meets the lane changing condition, the binding relationship between the vehicle trajectory and the binding lane is released, and the binding lane of the vehicle is re-bound.

[0120] The conditions for the vehicle to change lanes include that the driving speed of the vehicle is greater than a preset speed threshold, the yaw angle of the vehicle is greater than a preset yaw angle threshold, and the type of the lane line is a dashed line. For example, the vehicle lane changing condition is that the driving speed of the vehicle is greater than 0.5 m / s, the yaw angle of the vehicle is greater than 15°, and the type of the lane line is a dashed line.

[0121] When the vehicle trajectory meets the lane changing condition, it indicates that the vehicle has driven out of the bound binding lane. Therefore, when it is detected that the vehicle meets the lane changing condition, the lane binding is performed again according to the position of the vehicle to determine a new binding lane.

[0122] In another embodiment, Figure 4 is a flowchart for binding a lane according to another exemplary embodiment, in which the first preset number is 3, the second preset number is 5, the first preset threshold is 0.6, and the second preset threshold is 0.8. As shown in Figure 4 when the unmanned aerial vehicle detects a newly appearing vehicle, if the vehicle is not bound to a lane, the scores of the vehicle trajectory in the three consecutive frames and the to-be-bound lane are all greater than 0.6, it is preliminarily determined that the vehicle is possibly associated with the lane. On this basis, the scores of the vehicle trajectory in the five consecutive frames and the to-be-bound lane are all greater than 0.8, and the vehicle trajectory is bound to the to-be-bound lane. When the vehicle changes lanes, the lane is bound again to determine a new binding lane. If the unmanned aerial vehicle detects that the vehicle is suddenly blocked, the binding score continues to decrease, and if the binding score of a single frame is less than 0.4 when it is preliminarily determined that the vehicle is possibly associated with the lane, re-binding is performed. If the binding relationship has been determined, if the binding scores of two consecutive frames are less than 0.2, re-binding is performed.

[0123] Continuing to refer to step S102, the vehicle trajectory is bound to the lane on which the vehicle travels, which provides physical constraints for the vehicle. And the binding relationship between the vehicle trajectory and the bound lane provides constraints for the subsequent generated trajectory, which can provide a basis for the change of the subsequent trajectory and has higher accuracy.

[0124] Step S103, when the unmanned aerial vehicle detects that the vehicle is blocked, at least two candidate trajectories are generated according to the lane information in the bound lane and the lane topology map, and a target trajectory is determined in the candidate trajectories through a decision model.

[0125] When the unmanned aerial vehicle tracks the vehicle, it is necessary to detect whether the vehicle is blocked. The detection of being blocked specifically includes:

[0126] In the third preset number of continuous video frames, the confidence of identifying the vehicle in each frame is obtained, the average confidence is obtained, and the confidence drop rate is obtained according to the average confidence and the confidence of the last frame of continuous video frames. If the unmanned aerial vehicle does not detect the vehicle in the continuous video frames, and the confidence drop rate is greater than the preset confidence threshold, the unmanned aerial vehicle detects that the vehicle is blocked. Wherein, the confidence drop rate satisfies: the difference between the average confidence and the confidence of the last frame divided by the average confidence in the last N frames. The confidence is the probability of the unmanned aerial vehicle identifying the vehicle.

[0127] In one embodiment, the third preset number is 3, the unmanned aerial vehicle does not detect the vehicle in the continuous 3 frames of video frames, and the confidence drop rate of the continuous 3 frames of content exceeds 30%, the unmanned aerial vehicle detects the blocking event, that is, detects that the vehicle is blocked.

[0128] By detecting a plurality of continuous frames and increasing the detection index of the confidence change rate, the vehicle being truly blocked and the content not being identified due to temporary shaking can be effectively distinguished, and the accuracy of detecting that the vehicle is blocked is improved.

[0129] Continuing to refer to step S103, when the unmanned aerial vehicle detects that the vehicle is blocked, at least two candidate trajectories are generated according to the lane information in the bound lane and the lane topology map, specifically including:

[0130] Step S131, according to the lane speed limit information and the lane turning information in each lane in the lane topology map, determine the to-be-matched lane that the vehicle can drive from the bound lane.

[0131] Taking the bound lane bound by the vehicle as the starting lane, the lanes adjacent to the starting lane in the lane topology map are determined, and the to-be-matched lane that the vehicle can drive from the bound lane is determined according to the speed limit information and the lane turning information in each lane. Wherein, the to-be-matched lane represents that under the traffic rules, and meeting the lane turning and speed limit, the vehicle can change from the bound lane to the lane to ensure the passability between the lanes.

[0132] Step S132, obtain the driving distance of the vehicle within the blocked time, and generate at least two candidate trajectories according to the driving distance and the to-be-matched lane.

[0133] Obtain running parameters of the vehicle, and determine a driving distance of the vehicle in an occlusion time. According to the driving distance and a to-be-matched lane, a candidate trajectory between a bound lane and the to-be-matched lane is generated, and the candidate trajectory is at least two.

[0134] After the candidate trajectory is determined, for each candidate trajectory, a predicted speed of the vehicle in the candidate trajectory is obtained, and a kinematic score is obtained according to an actual speed of the vehicle and the predicted speed, so as to determine whether the candidate trajectory meets a kinematic constraint of the vehicle. The kinematic score specifically meets:

[0135]

[0136] wherein KinematicScore is the kinematic score, v pred is the predicted speed, v hist is the actual speed,

[0137] σ v is a historical speed standard deviation.

[0138] According to a distance between the candidate trajectory and a center of an adjacent lane, a topological consistency score is obtained, so as to determine a matching degree of the candidate trajectory and a road topological structure. The topological consistency score meets:

[0139]

[0140] wherein TopSore is the topological consistency score, d i is a minimum distance of the candidate trajectory to the adjacent lane, and σ d is a distance tolerance coefficient.

[0141] In the topological consistency score, the greater the value of the distance tolerance coefficient represents that a deviation value of the distance between the candidate trajectory and the center of the adjacent lane is greater; the smaller the value of the distance tolerance coefficient represents that the deviation value of the distance between the candidate trajectory and the center of the adjacent lane is smaller. When the value of the distance tolerance coefficient is smaller, it indicates that the candidate trajectory is closer to the lane.

[0142] When the vehicle reappears in the field of view of the unmanned aerial vehicle, feature and position information of the vehicle are obtained, a re-identification score is determined according to a similarity between a visual feature of the vehicle and a corresponding visual feature of the candidate trajectory and a norm of the position information and position information of the candidate trajectory, so as to determine a matching degree of the visual feature of the vehicle after the occlusion is removed and the visual feature before the occlusion. The re-identification score meets:

[0143] ,

[0144] wherein ReIDScore is the re-identification score, feat is the similarity, and pos is the norm of the position information.

[0145] An optimal score is obtained according to the kinematic score, the topological consistency score and the re-identification score. The optimal score satisfies:

[0146] wherein OptimalScore is the optimal score, KinematicScore is the kinematic score, TopoScore is the topological consistency score, ReIDScore is the re-identification score, w1 is the first optimization weight, w2 is the second optimization weight, and w3 is the third optimization weight.

[0147] The time of the vehicle occlusion is divided into three stages: the first preset time length after the vehicle is occluded is the initial occlusion stage; the second preset time length from the last time of the first preset time length is the middle occlusion stage; and the third preset time length from the last time of the second preset time length is the late occlusion stage. The first optimization weight, the second optimization weight and the third optimization weight are dynamically configured according to the stages of the vehicle occlusion.

[0148] Optionally, the first preset time length is 2 seconds, the second preset time length is 5 seconds, and the third preset time length is greater than 10 seconds. The weight distribution in the initial occlusion stage (0-2 seconds) is w1=0.7, w2=0.3, and w3=0; the weight distribution in the middle occlusion stage (3-8 seconds) is w1=0.4, w2=0.5, and w3=0.1; and the weight distribution in the long-term occlusion stage (greater than 8 seconds) is w1=0.2, w2=0.3, and w3=0.5.

[0149] The candidate trajectory with the highest optimal score is taken as the target trajectory. The target trajectory is smoothed to make the target trajectory smoother. Optionally, the smoothing method adopts a topologically constrained B-spline smoothing method, which satisfies the following formula:

[0150]

[0151] wherein Q(t i ) is the original trajectory point of the target trajectory, P(t i ) is the smoothed trajectory point, L j is the center line of the topological lane, λ is the constraint weight, P(u j ) is the three-dimensional coordinate of the smoothed target trajectory at the parameter u j , and u j is the parameter of the B-spline curve, which is used to locate the point to be constrained on the target trajectory.

[0152] After obtaining the optimization scores, if a difference between optimization scores of at least two candidate trajectories is less than a preset difference threshold, a straight-ahead candidate trajectory is selected as the target trajectory. Optionally, the preset difference threshold is 5%, and when the difference between optimization scores of at least two candidate trajectories is less than 5%, that is, the trajectory paths of the two candidate trajectories are very similar, the straight-ahead candidate trajectory is preferentially selected as the target trajectory.

[0153] If any two candidate trajectories intersect, a candidate trajectory in which a predicted speed of the vehicle is greater than a preset speed threshold is selected as the target trajectory. When the candidate trajectories intersect, a speed priority strategy is adopted, and a candidate trajectory in which a predicted speed of the vehicle is greater than a preset speed threshold is selected as the target trajectory.

[0154] If the target trajectory cannot be determined from the candidate trajectories within a target preset time length, a manual marking interface is started to perform manual selection.

[0155] Continuing to refer to step S103, the accuracy of detecting that the vehicle is blocked is improved through double detection of the vehicle being blocked by using multiple frames and a confidence drop rate. An accurate mapping relationship between the vehicle and a lane in a lane topology map is established, and after the vehicle being blocked is accurately detected, a candidate trajectory is formed by taking a bound lane as a starting lane and combining the lane topology map, so that the candidate trajectory meets traffic rules and geometric requirements of the lane, and each candidate trajectory can be implemented, thereby avoiding unreasonable trajectories. The target trajectory is determined in the candidate trajectory by using a kinematic score, a topology consistency score and a re-identification score, so that the target trajectory can highly meet actual conditions, and the accuracy of the target trajectory is improved.

[0156] In step S104, if there are at least two target trajectories, and the target trajectories meet a preset separation condition, the target trajectories are fused to obtain a complete trajectory.

[0157] During the period in which the vehicle is blocked, at least two target trajectories are generated. Alternatively, there are multiple vehicles on the lane, and different target trajectories are generated for different vehicles.

[0158] When it is determined that the target trajectory detected during the period in which the vehicle is blocked meets the preset separation condition, specifically, there are at least two target trajectories, and a determined target trajectory is established as a set of isolated trajectory segments. For each target trajectory in the set of isolated trajectory segments, if a difference between an end time and a start time of each target trajectory is within a preset time range, and a connected path exists between a start point and an end point of the target trajectory by verifying the start point and the end point by using traffic rules and lane turning, a vehicle type on the target trajectory is obtained, and it is determined that the vehicle types on different target trajectories are consistent.

[0159] According to the running speed, the driving time range when the vehicle is blocked can be determined. For example: if the running speed is less than 5 m / s, the maximum driving time when the vehicle is blocked is 60 seconds. If the running speed is greater than or equal to 5 m / s and less than 10 m / s, the maximum driving time when the vehicle is blocked is 30 seconds. If the running speed is greater than or equal to 10 m / s, the maximum driving time when the vehicle is blocked is 15 seconds.

[0160] When it is determined that the target trajectory meets the preset separation condition, the running speed before the vehicle is blocked and the driving time when the vehicle is blocked are obtained. According to the predicted acceleration at the end of the target trajectory and the actual acceleration of the vehicle, a motion continuity score is obtained to verify whether the target trajectory meets the running continuity. And the motion continuity score specifically meets:

[0161]

[0162] Wherein, ConsistencySore is the motion continuity score, n is the number of historical frames for verifying trajectory continuity, a k pred is the predicted acceleration at the end of the kth target trajectory, a k actual is the acceleration of the vehicle.

[0163] If the driving time is within the driving time range of the running speed, and the motion continuity score meets the continuity requirement, the fusion order of the target trajectory is determined by the driving time and the driving time, which provides guarantee for forming a complete trajectory and avoids a complete trajectory with time error.

[0164] Based on the fusion order, a complete trajectory is formed through a Markov decision model. The complete trajectory obtained through the Markov decision model specifically includes:

[0165] Based on the target trajectory, a state space is constructed, which is a set of all states of the vehicle. The state space meets:

[0166] Position state: s p ∈{lane keeping, lane changing, intersection turning}

[0167] Speed state: s v ∈{acceleration, constant speed, deceleration}

[0168] Topology state: s t ∈{main road, ramp, intersection}

[0169] The state space is taken as the input of the Markov decision model to ensure that the trajectory prediction meets the physical constraints and rule constraints of the actual driving scene.

[0170] The transition probability matrix is determined according to the actual traffic flow data of the current lane, for example, the transition probability matrix is determined with the current position state as a row of the transition probability matrix and the next position state as a column of the transition probability matrix, and each element in the transition probability matrix represents the probability of transition to the next position state after performing an action in the current position state, that is, P(s'|s,a), s' is the next position state, s is the current position state, and a is the switching action. The dimension of the transition probability matrix is determined by the state space. The switching action refers to the action of switching the lane of the vehicle, including left turn, right turn and straight driving. The transition probability matrix satisfies:

[0171] Current State \ Next State Lane Keeping Changing Lane Intersection Turning Lane Keeping 0.85 0.12 0.03 Changing Lane 0.70 0.25 0.05 Intersection Turning 0.10 0.15 0.75

[0172] According to the transition concept matrix and the state space described above, a complete trajectory is obtained through a decision function. Specifically, in the Markov decision model, the values of the current state s and the next state s' are provided by the state space, the values of P(s'|s,a) are provided by the transition probability matrix, the immediate reward of the action is evaluated by the reward function R(s,a), and the long-term reward of the action is evaluated by the value function to determine the decision function. The decision function satisfies:

[0173]

[0174] wherein π(s) is the decision function, R(s,a) is the immediate reward, γ is a discount factor, V(s) is the state value function, P(s'|s,a) is the state transition function, representing the probability of transition to s' after performing action a in state s, a is the switching action, and A is the set of switching actions of the vehicle.

[0175] The switching action of the vehicle is determined through the decision function, but the vehicle trajectory is a continuous sequence of path points. The switching action is converted into specific vehicle control instructions (such as steering wheel angle and acceleration) using a kinematic model, and then a smooth trajectory point is generated based on the geometric center line of the bound lane and the kinematic constraint. Therefore, the complete trajectory is smoothed by B-spline smoothing constraint to ensure that the acceleration change rate in the complete trajectory meets the vehicle dynamics. The B-spline smoothing constraint specifically satisfies:

[0176]

[0177] wherein T fused (t) is a complete trajectory function, representing the spatial position of the vehicle at time t, is the change rate of acceleration, and a max is the change rate threshold of acceleration.

[0178] According to the running speed of the vehicle, the timestamps of the target trajectories during the occlusion period are predicted, for example, one trajectory point every 0.1 second, and the multiple target trajectories are time-aligned according to the timestamps. If any two target trajectories overlap in time, the two target trajectories are fused by weighting, and the fused trajectory points are smoothed by topological constraints.

[0179] With reference to step S104, the fusion order is determined by using the timing information and the driving duration and motion continuity score to ensure that the connection between the target trajectories meets the spatiotemporal consistency, avoiding chaotic connection of target trajectories of different time periods, and through B-spline smoothing and target trajectory fusion, a complete trajectory with high precision can be obtained, overcoming the problem of fragmentation of different target trajectories.

[0180] In step S101, in the process of processing the target video frame by the double-branch hybrid neural network to obtain the pixel coordinates of the lane line, the following steps are specifically included:

[0181] The double-branch hybrid neural network captures the local geometric features and global context information of the lane in the target video frame. The local geometric feature information includes the lane line of the lane and the indicator arrow in the lane. The global context information includes the background information in the video frame and the association relationship between different objects. If part of the content in the lane is occluded, making the local geometric feature incomplete, the global context information is used to supplement the missing content of the local geometric feature to obtain complete lane features. Therefore, according to the local geometric feature and the global context feature, the pixel coordinates of the lane line in the target video frame are obtained. The confidence score of each pixel coordinate is calculated, and the pixel point coordinates with a confidence score greater than a preset score threshold are retained. The confidence score satisfies the following formula:

[0182]

[0183] Wherein, S i is the confidence score of the i-th pixel coordinate, f feature (u i ,v i ) is the pixel coordinate of the lane line, MLP(f feature (u i ,v i )) is the multi-layer perceptron processing of the pixel coordinate of the lane line, and δ is the activation function for mapping the MLP input to a probability value.

[0184] The retained pixel coordinates are converted from the coordinate system of the picture to the world coordinate system, so that the position of the vehicle in the lane topology graph can be directly determined by the pixel coordinates, satisfying the following formula:

[0185]

[0186] Wherein, Xi is the horizontal coordinate in the world coordinate system, Y i is the vertical coordinate in the world coordinate system, f is the focal length of the camera assembly, R(Φ, θ) is the selection matrix, z is the flight height of the UAV, u i is the horizontal pixel coordinate in the picture, v i is the vertical pixel coordinate in the picture, x t is the horizontal position coordinate of the UAV at time t, y t is the vertical position coordinate of the UAV at time t, Φ is the yaw angle of the UAV, and θ is the pitch angle of the UAV.

[0187] After determining the pixel coordinates, the pixel coordinates need to be clustered. In the clustering process, the comprehensive distance of the geometric features and semantic features of the fused lane is also calculated to distinguish lanes with different attributes, such as main roads and emergency lanes. Then the comprehensive distance satisfies:

[0188]

[0189] where d combined is the comprehensive distance, a is an empirically optimized value, d euclidean is a geometric distance item, representing the three-dimensional space Euclidean distance between the pixel coordinates of the lane lines, is the semantic feature vector of the current lane line pixel coordinates, is the semantic feature vector of the contrast lane line, is a feature difference item, representing the L2 norm distance between the two.

[0190] In step S101, after obtaining the target video frame, the target video frame also needs to be corrected for the problem of the UAV perspective distortion. The corrected target video frame specifically includes: determining a homography matrix according to the height and pitch angle of the UAV. For each target video frame, a perspective transformation is performed based on the homography matrix. To eliminate the perspective changes caused by the change of the UAV attitude, and to provide a geometric consistency guarantee for subsequent lane detection.

[0191] In one embodiment, Figure 5 is a flowchart of a vehicle trajectory recovery method in a blocked state according to another exemplary embodiment, as shown in Figure 5As shown in the figure, a video of the vehicle is obtained from a camera assembly of the unmanned aerial vehicle, target video frames are determined from the video according to SIFT feature changes, vehicle detection tracking and lane topology graph construction are performed on the vehicle in the video frames. The vehicle trajectory is bound with the lane through a trajectory-lane analysis module. When the vehicle is detected to be occluded, topology-constrained trajectory prediction is performed, so that the trajectory prediction when the vehicle is occluded is complete and consistent with the actual trajectory. When the vehicle is not detected to be occluded, the running trajectory of the vehicle is recorded. Finally, the complete vehicle driving trajectory is formed according to the unoccluded trajectory and the occluded trajectory.

[0192] In another embodiment, Figure 6 is a flow chart of a vehicle trajectory recovery method in an occlusion state according to another exemplary embodiment, as Figure 6 As shown in the figure, a video of the vehicle is obtained from a camera assembly of the unmanned aerial vehicle, target video frames are determined from the video according to SIFT feature changes, vehicle detection tracking and lane topology graph construction are performed on the vehicle in the video frames. The vehicle trajectory is bound with the lane through a trajectory-lane analysis module. When the vehicle is detected to be occluded, topology-constrained trajectory prediction is performed, so that the trajectory prediction when the vehicle is occluded is complete and consistent with the actual trajectory. When the vehicle is not detected to be occluded, the running trajectory of the vehicle is recorded. Finally, the complete vehicle driving trajectory is formed according to the unoccluded trajectory and the occluded trajectory.

[0193] In summary, the vehicle trajectory recovery method, device and computer equipment in an occlusion state provided by the embodiments of the present application, through the target video frames of the unmanned aerial vehicle, the lane topology graph is constructed, and the vehicle is bound with the lane through the lane geometry constraint and the traffic rule constraint in the lane topology graph, which provides physical constraint to ensure that the generated path conforms to the actual road conditions. During the occlusion of the vehicle, candidate trajectories are generated, and the candidate trajectories conform to the topological consistency, kinematic continuity and time continuity, and the accuracy of the generated candidate trajectories. Through the timing information and B-spline smoothing processing, the complete trajectory with global consistency is obtained, which avoids generating fragmented trajectories, so that the complete trajectory is close to the actual trajectory, thereby improving the accuracy of the recovered trajectory.

[0194] In a second aspect, the embodiments of the present application provide a vehicle trajectory recovery device in an occlusion state. Figure 7 is a block diagram of a vehicle trajectory recovery device in an occlusion state according to an exemplary embodiment. As Figure 7As shown, the vehicle trajectory recovery device in the occlusion state is applied to a UAV carrying a camera assembly, and the vehicle trajectory recovery device in the occlusion state comprises: a lane topology graph construction module configured to acquire a target video frame of vehicle driving captured by the UAV, cluster lane pixel coordinates in the target video frame to obtain an intersection, and construct a lane topology graph by taking the intersection as a node and a connection relationship between intersections as an edge;

[0195] A lane and vehicle trajectory binding module is configured to determine at least two to-be-bound lanes in the lane topology graph according to a driving state of the vehicle, calculate a binding score of the vehicle and each to-be-bound lane, and determine a binding lane corresponding to the vehicle according to the binding score.

[0196] A target trajectory determination module is configured to generate at least two candidate trajectories according to the binding lane and lane information in the lane topology graph when the UAV detects that the vehicle is occluded, and determine a target trajectory in the candidate trajectories by using a decision model.

[0197] A target trajectory fusion module is configured to fuse target trajectories to obtain a complete trajectory if there are at least two target trajectories and the target trajectories satisfy a preset separation condition.

[0198] To sum up, the vehicle trajectory recovery device in the occlusion state provided by the present application acquires a target video frame of vehicle driving captured by the UAV, and constructs a lane topology graph. The binding score between the to-be-bound lane and the vehicle is calculated, the vehicle is bound with the binding lane, the physical constraint between the vehicle and the binding lane is provided, the continuity of the vehicle running trajectory in the occlusion scene is ensured by taking the lane information in the binding lane as a constraint condition, at least two candidate trajectories are generated according to the binding lane and the lane information in the lane topology graph when the vehicle is detected to be occluded, the lane geometric constraint and the traffic rule constraint are provided for generating the candidate trajectories, the candidate trajectories are ensured to conform to the actual situation, the target trajectory is determined in the candidate trajectories, the target trajectory is ensured to be close to the actual trajectory of the vehicle, the accuracy of the target trajectory is ensured, the target trajectories satisfying the preset separation condition are fused to obtain a complete trajectory consistent in the whole scene, and the problem of trajectory fragmentation caused by frame-by-frame processing is overcome.

[0199] It should be noted that the vehicle trajectory recovery device in the occlusion state provided by the present embodiment is used to realize the above-mentioned embodiments, and will not be described again. As used above, the terms "module", "unit", "sub-unit", etc. can be a combination of software and / or hardware that realizes a predetermined function. Although the device described in the above embodiments is preferably realized in software, hardware or a combination of software and hardware is also possible and is conceived.

[0200] In a third aspect, an electronic device is provided, Figure 8 is a block diagram of an electronic device according to an exemplary embodiment. As shown, the electronic device can include a processor 81 and a memory 82 having stored computer program instructions. Figure 8

[0201] In particular, the processor 81 can include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0202] ​The memory 82 can include a mass storage for data or instructions. By way of example, and without limitation, the memory 82 can include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, a compact disk read only memory (CD-ROM), a digital versatile disk (DVD), a tape drive, a USB drive, or a combination of two or more of these. Where appropriate, the memory 82 can include removable or non-removable (or fixed) media. Where appropriate, the memory 82 can be internal or external to the data processing apparatus. In certain embodiments, the memory 82 is a nonvolatile memory. In certain embodiments, the memory 82 includes a read only memory (ROM). Where appropriate, this ROM can be mask programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory (FLASH) or a combination of two or more of these. Where appropriate, the ROM can include a combination of these without limitation. Where appropriate, the memory 82 includes a random access memory (RAM). Where appropriate, this RAM can be static random access memory (SRAM) or dynamic random access memory (DRAM), which can be Fast Page Mode Dynamic random access memory (FPMDRAM), Extended Data Output Dynamic random access memory (EDODRAM), synchronous dynamic random access memory (SDRAM), or the like.

[0203] The memory 82 can be used to store or buffer various data files required for processing and / or communication, and possible computer program instructions executed by the processor 81.

[0204] The processor 81 reads and executes the computer program instructions stored in the memory 82 to realize the vehicle trajectory recovery in any of the above-mentioned embodiments in the occlusion state.

[0205] In an embodiment, the vehicle trajectory recovery device in the occlusion state can further include a communication interface 83 and a bus 80. As shown in the figure, the processor 81, the memory 82, and the communication interface 83 are connected through the bus 80 and complete communication with each other. Figure 8

[0206] The communication interface 83 is used to realize the communication between the modules, devices, units and / or devices in the embodiments of the present application. The communication interface 83 can also realize data communication with other components, such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations, etc.

[0207] ​Bus 80 includes hardware, software, or both, to couple components of the vehicle trajectory recovery device in the occluded state to each other. Bus 80 includes, but is not limited to, at least one of a data bus, an address bus, a control bus, an expansion bus, a local bus, and the like. By way of example and not limitation, bus 80 can include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus or a combination of two or more of these. Where appropriate, bus 80 can include one or more buses. Although the present embodiments describe and show a particular bus, the present application contemplates any suitable bus or interconnect.

[0208] In a fourth aspect, the present embodiments provide a computer-readable storage medium having stored thereon a program, and the program, when executed by a processor, implements the method for recovering vehicle trajectory in an occluded state according to the first aspect.

[0209] More specifically, the computer-readable storage medium can include, but is not limited to, a portable disc, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0210] In possible implementation manners, the present application can also be implemented in the form of a program product, which comprises program codes for causing a terminal device to perform the steps of the method for recovering a vehicle trajectory in an occlusion state according to the first aspect when the program product is run on the terminal device.

[0211] The program code for executing the present application can be written in any combination of one or more programming languages, and can be executed entirely on the user device, partly on the user device, as a stand-alone software package, partly on the user device and partly on a remote device, or entirely on a remote device.

[0212] The technical features of the above-described embodiments can be combined in any manner. For the sake of brevity, not all possible combinations of the technical features in the above-described embodiments are described, however, as long as the combinations of the technical features do not result in contradictions, they shall be considered as falling within the scope of the present disclosure.

[0213] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it shall not be understood as a limitation on the scope of the patent. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these shall fall within the scope of protection of the present application. Therefore, the scope of protection of the patent of the present application shall be subject to the appended claims.

Claims

1. A method for vehicle trajectory recovery under occlusion conditions, characterized in that, The method is applied to a UAV carrying a camera assembly, and the method comprises the following steps: Obtaining a target video frame of a vehicle driving captured by the UAV, clustering lane pixel coordinates in the target video frame to obtain an intersection, taking the intersection as a node, and constructing a lane topology graph based on a connection relationship between the intersections as edges; In the lane topology graph, determining at least two to-be-bound lanes according to a driving state of the vehicle, calculating a binding score of the vehicle trajectory and each to-be-bound lane, and determining a binding lane corresponding to the vehicle trajectory according to the binding score; When the UAV detects that the vehicle is blocked, generating at least two candidate trajectories according to the binding lane and lane information in the lane topology graph, and determining a target trajectory in the candidate trajectories through a decision model; If there are at least two target trajectories, and the target trajectories meet a preset separation condition, fusing the target trajectories to obtain a complete trajectory; The driving state comprises position information, a heading angle and a driving speed of the vehicle, and the calculation of the binding score of the vehicle trajectory and each to-be-bound lane comprises the following steps: For any one to-be-bound lane, determining a distance feature according to a distance between the position information and a center of the to-be-bound lane; Determining an angle feature according to a lane direction of the to-be-bound lane and the heading angle; Determining a speed feature according to a lane speed limit of the to-be-bound lane and the driving speed; Based on a current time, obtaining a binding score of the vehicle and a corresponding binding lane at a previous time to determine a historical feature; Based on the distance feature, the angle feature, the speed feature and the historical feature, obtaining the binding score of the vehicle trajectory and the to-be-bound lane; According to the binding score, determining the binding lane corresponding to the vehicle trajectory, further comprises the following steps: If the binding score of each target video frame is greater than a first preset threshold in a first preset number of continuous first target video frames, an intermediate pairing set is obtained, and the intermediate pairing set is a pairing of the vehicle trajectory and the to-be-bound lane; In the intermediate pairing set, if the binding score of each continuous second target video frame is greater than a second preset threshold in a second preset number of continuous second target video frames, a current to-be-bound lane is determined as the binding lane of the vehicle, wherein the first preset number is less than the second preset number, and the first preset threshold is less than the second preset threshold; If the vehicle trajectory meets a lane-changing condition, the binding relationship between the vehicle trajectory and the binding lane is released, and the binding lane of the vehicle is re-determined.

2. The method of claim 1, wherein, The clustering of the lane pixel coordinates in the target video frame to obtain an intersection, taking the intersection as a node, and constructing a lane topology graph based on a connection relationship between the intersections as edges comprises the following steps: Clustering the lane pixel coordinates according to a preset lane requirement to obtain a structured road network, and determining an intersection in the structured road network according to a preset curvature requirement; According to a preset lane design requirement, constructing a lane topology graph taking the intersection as a node and the connection relationship between the intersections as edges.

3. The method of claim 1, wherein, The detection of the UAV that the vehicle is blocked comprises the following steps: In the third preset number of continuous video frames, an average confidence is obtained according to the confidence of identifying the vehicle in each frame, and a confidence drop rate is obtained according to the average confidence and the confidence of the last frame of the continuous video frames; If the unmanned aerial vehicle does not detect the vehicle in the continuous video frames, and the confidence drop rate is greater than a preset confidence threshold, the unmanned aerial vehicle detects that the vehicle is blocked.

4. The method of claim 3, wherein, The generating of the at least two candidate trajectories according to the bound lane and the lane information in the lane topology map comprises: According to the lane speed limit information and the lane turning information in each lane in the lane topology map, a to-be-matched lane to which the vehicle can drive from the bound lane is determined; A driving distance of the vehicle in the blocked time is obtained, and at least two candidate trajectories are generated according to the driving distance and the to-be-matched lane.

5. The method of claim 3, wherein, The determining of the target trajectory in the candidate trajectories by the decision model comprises: For each of the candidate trajectories, a predicted speed of the vehicle in the candidate trajectory and a distance between the candidate trajectory and an adjacent lane are obtained; A kinematics score is obtained according to an actual speed of the vehicle and the predicted speed; A topology consistency score is obtained according to a distance between the candidate trajectory and a center of the adjacent lane; When the vehicle reappears, feature and position information of the vehicle are obtained, a re-identification score is determined according to a similarity between the feature of the vehicle and a feature of the candidate trajectory, and a norm between the position information and position information of the candidate trajectory; An optimization score is obtained according to the kinematics score, the topology consistency score and the re-identification score; The candidate trajectory with the highest optimization score is taken as the target trajectory.

6. The method of claim 5, wherein, After the determining of the optimization score according to the kinematics score, the topology consistency score and the re-identification score, the method further comprises: If a difference between the optimization scores of the at least two candidate trajectories is less than a preset difference threshold, the candidate trajectory of straight driving is taken as the target trajectory; If any two of the candidate trajectories intersect, the candidate trajectory with a predicted speed of the vehicle greater than a preset speed threshold is taken as the target trajectory. 7.The method of claim 1, wherein, The preset separation condition comprises: A difference between an end time and a start time of each of the target trajectories is within a preset time range; Vehicle types driving on different target trajectories are consistent. 8.The method of claim 1, wherein, The fusing of the target trajectories to obtain a complete trajectory comprises: An operating speed of the vehicle before being blocked and a driving duration of the vehicle when being blocked are obtained; A motion continuity score is obtained according to a predicted acceleration at an end of the target trajectory and an actual acceleration of the vehicle; If the driving duration is within a driving duration range of the operating speed, and the motion continuity score meets a continuity requirement, a fusion order of the target trajectories is determined, and the target trajectories are fused according to the fusion order by the decision model to obtain a complete trajectory.

Images