Track generation method and device and electronic equipment

Abstract

The invention relates to the technical field of computers, in particular to a track generation method and device and electronic equipment, and the method comprises the steps: carrying out the target detection of an aerial image, and obtaining a target detection result which comprises the motion parameter information of a candidate ground target; determining a spatial constraint matching degree according to a connection line between a target position point in the motion parameter information and an end position point of each existing track and a relative position relationship between the connection line and a non-transregional region; determining a motion constraint matching degree according to the motion parameter information and the motion information of the ground target corresponding to each existing track; according to the spatial constraint matching degree and the motion constraint matching degree, determining a target existing trajectory from the plurality of existing trajectories; the position information and the motion information of the candidate ground target are associated and added to the existing track of the target; according to the method provided by the invention, the spatial constraint and the motion constraint in the trajectory generation process are comprehensively considered, so that the generated trajectory is more accurate.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a trajectory generation method, device and electronic device. Background Art

[0002] In the field of automotive technology, autonomous driving is an important function of intelligent driving. In the process of autonomous driving development, a large amount of high-precision trajectory data is required for algorithm research and testing. In related technologies, the methods of obtaining trajectory data mainly include real vehicle collection, roadside equipment collection, and aerial photography collection. Among them, aerial photography collection is widely used due to its flexibility, low cost, and high efficiency.

[0003] However, among the image data collected by aerial photography, there is a low accuracy of trajectory data obtained by processing the aerial images, which affects the application effect of subsequent trajectory data. Summary of the invention

[0004] In view of this, the embodiments of the present application propose a trajectory generation method, device and electronic device, which can comprehensively consider the spatial constraints and motion constraints in the trajectory generation process to make the generated trajectory more accurate.

[0005] The embodiment of the present application is implemented by adopting the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a trajectory generation method, the method comprising: performing target detection on an aerial image to obtain a target detection result, the target detection result comprising motion parameter information of an identified candidate ground target; determining a spatial constraint matching degree between the candidate ground target and each of the existing trajectories based on a line between a target position point in the motion parameter information and an end position point of each of the existing trajectories, and a relative position relationship with a non-crossable area corresponding to the candidate ground target; determining a motion constraint matching degree between the candidate ground target and each of the existing trajectories based on the motion parameter information and the motion information of the ground target corresponding to each of the existing trajectories; determining, from a plurality of the existing trajectories, a target existing trajectory in which the corresponding ground target is the same target as the candidate ground target based on the spatial constraint matching degree and the motion constraint matching degree; and associating the position information and the motion information of the candidate ground target with the target existing trajectory.

[0007] In a second aspect, an embodiment of the present application provides a trajectory generation device, which includes: a detection module, which is used to perform target detection on aerial images to obtain target detection results, wherein the target detection results include motion parameter information of the identified candidate ground targets; a first calculation module, which is used to determine the spatial constraint matching degree between the candidate ground target and each of the existing trajectories based on the relative position relationship between the line between the target position point in the motion parameter information and the end position point of each of the existing trajectories and the non-crossable area corresponding to the candidate ground target; a second calculation module, which is used to determine the motion constraint matching degree between the candidate ground target and each of the existing trajectories based on the motion parameter information and the motion information of the ground target corresponding to each of the existing trajectories; a matching module, which is used to determine, from multiple existing trajectories, the corresponding ground target and the candidate ground target as the same target according to the spatial constraint matching degree and the motion constraint matching degree; and an adding module, which is used to associate the position information and the motion information of the candidate ground target and add it to the target existing trajectory.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor; a memory, wherein computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, the above method is implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed by a processor, the above method is implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a computer program product, including computer instructions, which implement the above method when executed by a processor.

[0011] The method provided in the present application can determine whether the spatial association between the candidate ground target and each existing trajectory is reasonable through the spatial constraint matching degree, and can determine whether the kinematic association between the candidate ground target and each existing trajectory is reasonable through the motion constraint matching degree. Therefore, the existing target trajectory associated with the candidate ground target is determined through the spatial constraint matching degree and the motion constraint matching degree, so that the association between the candidate ground target and the existing target trajectory in both space and kinematics is more reasonable, thereby improving the accuracy of the trajectory.

[0012] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0014] Figure 1 A first flow chart of a trajectory generation method provided by an embodiment of the present application is shown.

[0015] Figure 2 It shows that an embodiment of the present application provides Figure 1 Schematic diagram of the process of step S130.

[0016] Figure 3 It shows that an embodiment of the present application provides Figure 1 Schematic diagram of the process of step S140.

[0017] Figure 4 A second flow chart of a trajectory generation method provided in an embodiment of the present application is shown.

[0018] Figure 5 A third flow chart of a trajectory generation method provided in an embodiment of the present application is shown.

[0019] Figure 6 A fourth flow chart of a trajectory generation method provided in an embodiment of the present application is shown.

[0020] Figure 7 A schematic diagram showing the effect of semantic segmentation of an aerial image according to an embodiment of the present application is shown.

[0021] Figure 8 An interactive interface involved in an embodiment of the present application is shown.

[0022] Fig. 9 A schematic diagram of a bicycle kinematics model according to an embodiment of the present application is shown.

[0023] Fig.10 A schematic diagram of a trajectory generation system according to an embodiment of the present application is shown.

[0024] Fig.11 It shows that an embodiment of the present application provides Fig.10 Schematic diagram of trajectory generation.

[0025] Fig.12 The embodiment of the present application provides Fig.10 Schematic diagram of the process of model training phase and data preparation phase.

[0026] Fig.13A schematic diagram of a trajectory generating device provided in an embodiment of the present application is shown.

[0027] Fig.14 A schematic diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0029] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. According to the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0030] In the following description, the terms "first\second" and the like are only used to distinguish similar objects and do not represent a specific order for the objects. It is understandable that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here. In the following description, "some embodiments or some embodiment modes" are involved, which describe a subset of all possible embodiments, but it is understandable that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0031] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0032] In the field of automotive technology, autonomous driving is an important function of intelligent driving. In the process of autonomous driving development, a large amount of high-precision trajectory data is required for algorithm research and testing. In related technologies, the methods of obtaining trajectory data mainly include real vehicle collection, roadside equipment collection, and aerial photography collection. Among them, aerial photography collection is widely used due to its flexibility, low cost, and high efficiency.

[0033] However, in the image data collected by aerial photography, ground targets in the aerial images are easily blocked, resulting in low accuracy of trajectory data obtained by processing the aerial images, thereby affecting the application effect of subsequent trajectory data.

[0034] In order to solve the above problems, the present application proposes a trajectory generation method, device and electronic device.

[0035] The embodiments provided in this application will be described below with reference to the accompanying drawings.

[0036] See also Figure 1 , Figure 1 A schematic flow chart of a trajectory generation method provided in an embodiment of the present application is given. The trajectory generation method includes steps S110-S150:

[0037] S110, performing target detection on the aerial image to obtain a target detection result, where the target detection result includes motion parameter information of the identified candidate ground targets.

[0038] The aerial image may be an overhead image obtained by an aerial photography device taking overhead photos of a target area; the target area may be, for example, an intersection area, a road intersection area, an elevated bridge area, etc.

[0039] In some implementations, an aerial video of the target area may be shot by an aerial photography device, and then captured from the aerial video to obtain aerial images at different times.

[0040] During the shooting process, the aerial photography equipment may experience position displacement and roll angle changes due to external environmental factors such as airflow, resulting in inconsistent shooting fields of view at different times. Therefore, after acquiring multiple aerial images, one aerial image can be selected as a baseline aerial image, and then the other aerial images can be aligned to the baseline aerial image.

[0041] The image alignment process is explained by taking the image feature matching algorithm as an example. After determining the reference aerial image, multiple reference feature points in the reference aerial image are extracted. Then, the alignment feature points that match each reference feature point in the aerial image to be aligned are determined respectively through the image feature matching algorithm. The transformation relationship between the reference aerial image and the aerial image to be aligned is calculated based on the matched reference feature points and the alignment feature points, so that the aerial image to be aligned can be aligned to the reference aerial image based on the change relationship.

[0042] The candidate ground target refers to the ground moving target identified from the aerial image. The motion parameter information of the candidate ground target may include the position, direction, speed, etc. of the candidate ground target; the candidate ground target may be a movable object such as a vehicle or a pedestrian.

[0043] In some embodiments, target detection for aerial images may be implemented by a target detection model; wherein the target detection model may be obtained by fine-tuning a pre-trained target detection model using sample aerial images, and the fine-tuning training process may include the following steps 1 to 3:

[0044] Step 1: Obtain sample aerial images and mark sample targets in the sample aerial images.

[0045] Among them, the sample targets that need to be labeled are determined according to the trajectory generation task. For example, if the trajectory generation task is a vehicle trajectory generation task, the sample targets that need to be labeled are vehicles; if the trajectory generation task is a pedestrian trajectory generation task, the sample targets that need to be labeled are pedestrians.

[0046] The labeling information of the sample target is the motion parameter information of the sample target.

[0047] Step 2: Use the pre-trained target detection model to perform target detection on the sample aerial image to obtain a sample detection result, wherein the sample detection result includes motion parameter information of the identified sample target.

[0048] In some embodiments, for a sample target detected in a sample aerial image, the sample target in the sample aerial image may be represented by a rotated rectangle, and the rotated rectangle may represent the position, size, and orientation of the sample target; illustratively, for a sample target j, the rotated rectangle corresponding to the sample target j may be described as:

[0049] detect ion j，pix =[u pix ,v pix ,L pix ,W pix ,θ pix ];

[0050] Among them, (u pix , v pix ) is the coordinate of the detection box surrounding the sample target j, L pix is the length of the detection box surrounding sample target j, W pix is the width of the detection box surrounding sample target j, θ pix It is the clockwise rotation angle of the detection box surrounding the sample target j relative to the u-axis.

[0051] Step 3: Calculate the model loss based on the motion parameter information of the identified sample targets and the motion parameter information of the labeled sample targets, and fine-tune the pre-trained target detection model based on the model loss to obtain the target detection model.

[0052] S120, determining the spatial constraint matching degree between the candidate ground target and each existing trajectory according to the relative position relationship between the connecting line between the target position point in the motion parameter information and the end position point of each existing trajectory and the non-crossable area corresponding to the candidate ground target.

[0053] Among them, spatial constraints refer to the constraints formed by other spatial objects in the space on the trajectory direction of the candidate ground target; for example, if the candidate ground target is a vehicle, if there is a spatial object such as a protection belt in the space, obviously, the trajectory direction of the vehicle should not pass through the protection belt, that is, the protection belt will form a spatial constraint on the trajectory direction of the candidate ground target.

[0054] Among them, the non-crossable area corresponding to the candidate ground target refers to the area that the candidate ground target cannot enter; taking the candidate ground target as a vehicle as an example, the non-crossable area can be a sidewalk, a green belt, a road construction area, etc.

[0055] It should be noted that the non-crossable area can be a plane, such as the plane where the pedestrian area is located; it can also be a line in the plane, such as the road boundary, the solid line of the road, etc.

[0056] In some implementations, step S120 may specifically include:

[0057] If the line between the target position point in the motion parameter information and the end position point of the existing trajectory intersects with the non-crossable area corresponding to the candidate ground target, the spatial constraint matching degree between the candidate ground target and each existing trajectory is determined to be the first matching degree.

[0058] If the line between the target position point in the motion parameter information and the end position point of the existing trajectory does not intersect with the non-crossable area corresponding to the candidate ground target, it is determined that the spatial constraint matching degree between the candidate ground target and each existing trajectory is the second matching degree; the second matching degree is greater than the first matching degree.

[0059] It can be understood that the line between the target position point in the motion parameter information and the end position point of the existing trajectory can represent the direction of the new trajectory obtained by associating the candidate ground target with the existing trajectory. If the line intersects with the non-crossable area corresponding to the candidate ground target, it means that the new trajectory will pass through the non-crossable area. Obviously, the new trajectory in this case is unreasonable in terms of spatial constraints. Similarly, if the line does not intersect with the non-crossable area corresponding to the candidate ground target, it means that the new trajectory will not pass through the non-crossable area. In this case, the new trajectory is reasonable in terms of spatial constraints.

[0060] In some embodiments, the position coordinates of the non-crossable area are marked in the world coordinate system, and the position coordinates of the target position point in the motion parameter information are marked in the image coordinate system. Therefore, before step S120, the target position point in the motion parameter information and the position of the non-crossable area need to be unified in the world coordinate system.

[0061] In some embodiments, the aerial photography equipment can be calibrated to determine the conversion relationship between the image coordinate system and the world coordinate system. Then, after determining the position coordinates of the target position point of the candidate ground target in the image coordinate system, the position coordinates of the target position point of the candidate ground target in the image coordinate system are converted to the position coordinates in the world coordinate system through the conversion relationship.

[0062] For example, for the identified candidate ground target j, its kinematic parameter information in the image coordinate system is described as: j，pix =[u pix ,v pix ,L pix ,W pix ,θ pix ], each kinematic parameter can be converted to the world coordinate system through the following calculation formulas 1 to 5:

[0063] Calculation formula 1: x g =(u pix -u o )·cosθ L ·S trans +(v o -v pix )·sinθ L ·S trans ;

[0064] Calculation formula 2: y g =-(u pix -u o )·sinθ L ·S trans +(v o -v pix )·cosθ L ·S trans ;

[0065] Calculation formula 3: θ g =π-θ pix -θ L ;

[0066] Calculation 4: L g =L pix ·S trans ;

[0067] Calculation formula 5: W g =W pix ·S trans .

[0068] By converting the above formulas 1 to 5, we can get the kinematic parameter information in the world coordinate system detection j,ground =[x g ,y g ,L g ,W g ,θ g ].

[0069] Among them, (u pix , v pix ) is the coordinate of the detection box surrounding the sample target j in the image coordinate system, L pix is the length of the detection box surrounding sample target j in the image coordinate system, W pix is the width of the detection box surrounding sample target j in the image coordinate system, θ pix It is the clockwise rotation angle of the detection box surrounding the sample target j relative to the u-axis in the image coordinate system.

[0070] S trans (m / pix) is the scale of the metric distance in the world coordinate system to the pixel distance in the image coordinate system; (u o ,v0) is the coordinate of the origin of the world coordinate system in the image coordinate system; θ L It is the angle that the x-axis of the world coordinate system rotates counterclockwise around the u-axis of the image coordinate system in the image coordinate system.

[0071] (x g ,y g ) is the coordinate of the detection box surrounding the sample target j in the world coordinate system, L g is the length of the detection box surrounding sample target j in the world coordinate system, W g is the width of the detection box surrounding sample target j in the world coordinate system, θ g It is the angle that the detection box surrounding sample target j rotates counterclockwise relative to the x-axis in the world coordinate system.

[0072] S130 , determining a motion constraint matching degree between the candidate ground target and each existing trajectory according to the motion parameter information and the motion information of the ground target corresponding to each existing trajectory.

[0073] Among them, motion constraints refer to the kinematic constraints on the motion of candidate ground targets. Taking the candidate ground target as a vehicle as an example, during normal driving, the speed of the vehicle will be limited to a certain speed, such as 120 kilometers per hour. If the candidate ground target is detected to have moved 10 meters within 0.1 seconds, the speed of the vehicle has reached 360 kilometers per hour, which is obviously unreasonable. For example, the change in the speed of the ground moving target within two adjacent time points with a short interval does not exceed the change threshold, and the change in the direction of motion does not exceed the specified angle.

[0074] In some embodiments, see Figure 2 , Figure 2 The embodiment of the present application provides Figure 1 The flow diagram of step S130 in FIG. 1 , the motion parameter information also includes the target motion direction; the motion information includes the position information of the last K position points of the ground target in the corresponding existing trajectory and the motion direction at the last K position points; K is a positive integer; step S130 may also include steps S131-S133:

[0075] S131. Determine the displacement change of the candidate ground target relative to each existing trajectory based on the target position point in the motion parameter information and the position information of the last K position points in each existing trajectory.

[0076] When K is 1, the distance between the target position point and the last position point can be used as the displacement change of the candidate ground target relative to the existing trajectory.

[0077] When K is greater than 1, we can first perform a weighted average based on the position coordinates of the last K position points in the existing trajectory to obtain the position information of the center position point of the last K position points. Then, we use the distance between the target position point and the center position point as the displacement change of the candidate ground target relative to the existing trajectory.

[0078] S132, determining a change in the movement direction of the candidate ground target relative to each existing trajectory according to the target movement direction and the movement directions of the ground targets corresponding to each existing trajectory at the last K position points.

[0079] When K is 1, the deviation between the target moving direction and the moving direction between the last position point can be used as the moving direction change of the candidate ground target relative to the existing trajectory.

[0080] When K is greater than 1, the motion direction of the existing trajectory can be obtained by first performing a weighted average based on the motion directions of the last K position points in the existing trajectory. Then, the deviation value between the target motion direction and the trajectory motion direction is used as the change in the motion direction of the candidate ground target relative to the existing trajectory.

[0081] S133: Determine the motion constraint matching degree between the candidate ground target and each existing trajectory according to the displacement change and the motion direction change of the candidate ground target relative to each existing trajectory.

[0082] Among them, the displacement change and the motion direction change are both negatively correlated with the motion constraint matching degree, that is, the larger the displacement change, the smaller the motion constraint matching degree; the larger the motion direction change, the smaller the motion constraint matching degree.

[0083] In some embodiments, the displacement change and the motion direction change can be weightedly summed to obtain a sum result, and the motion constraint matching degree can be determined based on the sum result; for example, multiple numerical intervals and reference motion constraint matching degrees corresponding to each numerical interval are pre-set, and after obtaining the sum result, the numerical interval to which the sum result belongs is determined, and the reference motion constraint matching degree corresponding to the numerical interval is used as the motion constraint matching degree.

[0084] In other implementations, the displacement score may be determined according to the displacement change, and the motion direction score may be determined according to the motion direction change. Then, the displacement score and the motion direction score may be weightedly summed to determine the motion constraint matching degree.

[0085] In some other embodiments, a first displacement threshold and a second displacement threshold may be pre-set, wherein the first displacement threshold is smaller than the second position threshold, and a first direction threshold and a second direction threshold may be pre-set, wherein the first direction threshold is smaller than the second direction threshold.

[0086] If the displacement change is greater than the second displacement threshold, or the motion direction change is greater than the second direction threshold, it means that the candidate ground target has at least one motion state (displacement or motion direction) that does not comply with the kinematic constraint, and the motion constraint matching degree can be determined to be the first value.

[0087] If the displacement change is less than the first displacement threshold, and the motion direction change is less than the first direction threshold, it means that the candidate ground target meets the kinematic constraints in both displacement and motion direction, and the motion constraint matching degree can be determined to be the second value.

[0088] Otherwise, the motion constraint matching degree is determined to be a third value; wherein the first value is smaller than the third value, and the third value is smaller than the second value.

[0089] S140 , determining, from a plurality of existing trajectories, an existing target trajectory in which the corresponding ground target and the candidate ground target are the same target according to the spatial constraint matching degree and the motion constraint matching degree.

[0090] In some implementations, the spatial constraint matching degree and the motion constraint matching degree may be weightedly summed to obtain a comprehensive matching degree between the candidate ground target and each existing trajectory, and the corresponding ground target and the candidate ground target may be determined from multiple existing trajectories based on the comprehensive matching degree as the existing target trajectory.

[0091] In some embodiments, an existing trajectory with the largest comprehensive matching degree may be determined from multiple existing trajectories as the existing trajectory of the target that is the same target as the candidate ground target; further, a comprehensive matching degree threshold may be pre-set, and an existing trajectory with a comprehensive matching degree greater than the comprehensive matching degree threshold and the largest comprehensive matching degree may be determined from multiple existing trajectories as the existing trajectory of the target that is the same target as the candidate ground target.

[0092] In other embodiments, after determining the comprehensive matching degree between multiple candidate ground targets involved in the target detection results and multiple existing trajectories, the Hungarian algorithm can be used to achieve the optimal matching between the multiple candidate ground targets and the multiple existing trajectories, and the existing trajectory that successfully matches the candidate ground target can be used as the existing trajectory of the target that is the same target as the candidate ground target.

[0093] S150 , associating the position information and motion information of the candidate ground target with the existing track of the target.

[0094] In some embodiments, the target position point of the candidate ground target can be smoothly connected to the end position point of the target's existing trajectory based on the position information of the candidate ground target to update the target's existing trajectory. After that, an association relationship between the motion information of the candidate ground target and the updated target's existing trajectory is established and stored to facilitate subsequent queries on the motion information of the ground target corresponding to the target's existing trajectory.

[0095] It is worth mentioning that if an existing trajectory fails to be associated with any candidate ground target involved in the target detection result, it means that the existing trajectory has ended, and the end position point of the existing trajectory is also the end point of the trajectory; similarly, if a candidate ground target involved in the target detection result fails to be associated with any existing trajectory, it means that a new trajectory needs to be constructed for the candidate ground target, and the target position point of the candidate ground target is also the starting point of the new trajectory.

[0096] The method provided in the embodiment of the present application can determine whether the spatial association between the candidate ground target and each existing trajectory is reasonable through the spatial constraint matching degree, and can determine whether the kinematic association between the candidate ground target and each existing trajectory is reasonable through the motion constraint matching degree. Therefore, the target existing trajectory associated with the candidate ground target is determined through the spatial constraint matching degree and the motion constraint matching degree, so that the association between the candidate ground target and the target existing trajectory in both space and kinematics is more reasonable, thereby improving the accuracy of the trajectory.

[0097] In some other embodiments, see Figure 3 , Figure 3 An embodiment of the present application provides Figure 1 The target detection result also includes the image features of the candidate ground target. Step S140 may also include steps S210-S230:

[0098] S210 , calculating feature similarities between the candidate ground target and each existing track according to the image features of the candidate ground target and the image features of the ground target corresponding to each existing track.

[0099] There are many ways to calculate feature similarity, such as Euclidean distance, cosine similarity, etc.

[0100] S220 , determining a comprehensive matching degree between the candidate ground target and each existing trajectory according to a feature similarity, a spatial constraint matching degree, and a motion constraint matching degree between the candidate ground target and each existing trajectory.

[0101] In some implementations, the feature similarity, the spatial constraint matching degree, and the motion constraint matching degree may be weightedly summed to obtain a comprehensive matching degree between the candidate ground target and each existing trajectory.

[0102] In other implementations, the product of the feature similarity, the spatial constraint matching degree, and the motion constraint matching degree may be used as the comprehensive matching degree between the candidate ground target and each existing track.

[0103] S230 , according to the comprehensive matching degree, determining, from among the multiple existing trajectories, an existing target trajectory in which the corresponding ground target and the candidate ground target are the same target.

[0104] In some embodiments, an existing trajectory with the largest comprehensive matching degree may be determined from multiple existing trajectories as the existing trajectory of the target that is the same target as the candidate ground target; further, a comprehensive matching degree threshold may be pre-set, and an existing trajectory with a comprehensive matching degree greater than the comprehensive matching degree threshold and the largest comprehensive matching degree may be determined from multiple existing trajectories as the existing trajectory of the target that is the same target as the candidate ground target.

[0105] In other embodiments, after determining the comprehensive matching degree between multiple candidate ground targets involved in the target detection results and multiple existing trajectories, the Hungarian algorithm can be used to achieve the optimal matching between the multiple candidate ground targets and the multiple existing trajectories, and the existing trajectory that successfully matches the candidate ground target can be used as the existing trajectory of the target that is the same target as the candidate ground target.

[0106] In the above implementation, feature similarity, spatial constraint matching and motion constraint matching are comprehensively considered to determine the target existing track associated with the candidate ground target, so that the accuracy of the association between the candidate ground target and the target existing track is higher.

[0107] In some embodiments, see Figure 4 , Figure 4 A second flow chart of the trajectory generation method provided by an embodiment of the present application is given. After step S220, the trajectory generation method further includes steps S310-S320:

[0108] S310. According to the comprehensive matching degree between the candidate ground target and each existing track, determine, from the multiple existing tracks, a reference existing track whose comprehensive matching degree with any candidate ground target in the target detection result is less than a first threshold.

[0109] It can be understood that since the comprehensive matching degree between the reference existing trajectory and any candidate ground target in the target detection result is less than the first threshold, it means that the ground target associated with the reference existing trajectory and any candidate ground target in the target detection result are not the same ground target.

[0110] S320: If the end position point in the reference existing trajectory is not in the disappearance area corresponding to the candidate ground target, generate target disappearance prompt information according to the target identifier of the ground target associated with the reference existing trajectory and the image frame number of the aerial image.

[0111] Among them, the disappearance area corresponding to the candidate ground target refers to the area where the trajectory of the ground target may disappear; for example, for a vehicle, the trajectory of the vehicle may disappear at the road exit out of the target area, and may end in the parking lot in the target area. Therefore, for the vehicle, the disappearance area can be a parking lot area, a road exit area, etc.

[0112] It is understandable that since the end position point of the reference existing trajectory is also the end point of the trajectory, and the end position point in the reference existing trajectory is not in the disappearance area corresponding to the candidate ground target, that is, the end point of the trajectory is not in the disappearance area corresponding to the candidate ground target, it is obvious that the reference existing trajectory is unreasonable in space; for example, for an intersection, the disappearance area corresponding to the vehicle is the intersection area. If the end position point of a reference existing trajectory is in the center of the road at the intersection, it means that the vehicle is stopped in the center of the road, which is obviously unreasonable.

[0113] In the above embodiment, since it is unreasonable for the reference existing trajectory to end at the aerial image, it can be reasonably inferred that there is an omission or misidentification of the ground target associated with the reference existing trajectory in the aerial image. Therefore, the target identifier of the ground target associated with the reference existing trajectory and the image frame number of the aerial image contained in the target message prompt message make it convenient for analysts to re-determine the candidate ground target that is the same ground target as the ground target associated with the reference existing trajectory from the aerial image.

[0114] In some embodiments, see Figure 5 , Figure 5 A third flow chart of a trajectory generation method provided by an embodiment of the present application is provided. After step S220, the trajectory generation method further includes steps S410-S430:

[0115] S410. According to the comprehensive matching degree between the candidate ground target and each existing track, determine a reference ground target whose comprehensive matching degree with multiple existing tracks is less than a first threshold among the candidate ground targets involved in the target detection result.

[0116] It can be understood that, since the comprehensive matching degrees between the reference ground target and the multiple existing trajectories are all less than the first threshold, it means that the reference ground target is not the same target as the ground moving target associated with any existing trajectory.

[0117] S420: According to the kinematic parameter information of the reference ground target, determine, in the reference ground target, a first ground target whose corresponding target position point is located in the starting appearance area.

[0118] S430: construct a new trajectory according to the kinematic parameter information of the first ground target; the starting position point of the new trajectory is the target position point in the kinematic parameter information of the first ground target.

[0119] Obviously, for the reference ground target whose corresponding target position point is not located in the starting appearance area, even if a new trajectory is constructed for it, the obtained new trajectory is unreasonable in terms of spatial constraints. Therefore, in some embodiments, the reference ground target whose corresponding target position point is not located in the starting appearance area can also be used as a suspicious ground target to generate suspicious ground target information. The suspicious ground target information includes the target identification of the suspicious ground target and the frame number of the corresponding aerial image, so that analysts can conduct targeted analysis of the suspicious ground target based on the suspicious ground target information.

[0120] In some embodiments, see Figure 6 , Figure 6 A fourth flow chart of a trajectory generation method provided by an embodiment of the present application is provided. After step S430, the trajectory generation method further includes steps S510-S520:

[0121] S510. Determine a trajectory combination according to a starting position point in the newly added trajectory and an ending position point in the reference existing trajectory. A trajectory combination includes a newly added trajectory and a reference existing trajectory. The starting position point of the newly added trajectory and the ending position point in the reference existing trajectory in the same trajectory combination are located in the same target occlusion area. The reference existing trajectory refers to an existing trajectory among multiple existing trajectories whose comprehensive matching degree with any candidate ground target in the target detection result is less than a first threshold.

[0122] It can be understood that, for a newly added trajectory and a reference existing trajectory in a trajectory combination, since the starting position point of the newly added trajectory and the ending position point in the reference existing trajectory are located in the same target occlusion area, it means that the newly added trajectory and the reference existing trajectory in a trajectory combination may be the trajectory of the same ground target.

[0123] S520: Generate matching prompt information according to the trajectory combination.

[0124] The matching prompt information may include the track identifier of the newly added track and the track identifier of the reference existing track, and may also include the image frame number of the aerial image involved in the newly added track and the reference existing track.

[0125] Through the above implementation, the analyst can modify the trajectory included in the trajectory combination in a targeted manner through the generated matching prompt information, thereby improving the processing efficiency of the analyst.

[0126] In some other implementations, after step S150, the trajectory generation method may further include:

[0127] If the motion constraint matching degree between the candidate ground target and the target's existing trajectory is less than a second matching degree threshold, suspicious motion prompt information is generated according to the trajectory identifier of the target's existing trajectory and the image frame number of the aerial image.

[0128] In the above embodiment, since the kinematic constraints are more dependent on the subjective settings of the analyst than the spatial constraints, on the basis that the candidate ground target and the target's existing trajectory have been associated, the motion constraint matching degree between the candidate ground target and the target's existing trajectory is less than the second matching degree threshold, which means that it is not reasonable in the currently set kinematic constraints. Therefore, based on the trajectory identifier of the target's existing trajectory and the image frame number of the aerial image, suspicious motion prompt information is generated, so that the analyst can query the corresponding target's existing trajectory and aerial image according to the suspicious motion prompt information, so as to determine whether the association between the candidate ground target and the target's existing trajectory is unreasonable, or whether the currently set kinematic constraints are unreasonable.

[0129] In some other embodiments, the trajectory generation method further includes:

[0130] Semantic segmentation is performed on the aerial images to determine the non-crossable areas, disappearing areas, starting areas, and target occluded areas in the ground area presented in the aerial images.

[0131] For easier understanding, see Figure 7 , Figure 7 An exemplary schematic diagram of the effect of semantic segmentation of an aerial image according to an embodiment of the present application is given.

[0132] exist Figure 7 In , the aerial image is an image of a crossroads, and the ground target to be detected is a vehicle; Figure 7 As shown in a, area A (i.e., the area contained in the intersection) is the key detection area for vehicles. Obviously, for the intersection shown in area A, vehicles can only enter and exit area A from each intersection, which means that the starting point and end point of each vehicle's trajectory can only be located in the intersection area, that is, Figure 7 The middle area a1-a5 is both the disappearance area and the beginning appearance area.

[0133] At the same time, for vehicles, roadside flower beds, solid lines in the middle of the road, and road boundaries are all inaccessible. Figure 7 As shown in b, the figure exemplarily shows the non-crossable areas b1, b2 and multiple non-crossable boundaries.

[0134] In addition, since aerial images are taken from a bird's-eye view, some buildings and roadside plants may block the vehicle. Figure 7 As shown in c, the target occlusion areas c1-c7 are given as an example.

[0135] In some implementations, analysts can perform correction operations on suspicious tracks and ground targets based on the generated prompt information (including target message prompt information, matching prompt information, suspicious motion prompt information, etc.), and the correction operations may be deletion, merging, adding, etc.

[0136] In some embodiments, Figure 8 As shown, Figure 8 An example of an interactive interface involved in the embodiment of the present application is given, through Figure 8 The interactive interface shown can facilitate analysts to perform correction operations based on the generated prompt information.

[0137] exist Figure 8 In the interactive interface shown, the generation status of the trajectory under each frame of the aerial image can be displayed through the preview box, so as to generate a continuous animation to realize the visualization of the trajectory; at the same time, the interactive interface provides various interactive interfaces to realize interactive operations, such as play, pause, previous frame, next frame, operation, etc.

[0138] Taking the operation interface as an example, the operation interface can provide multiple operations, such as merging, deleting, adding, regenerating, etc.; wherein merging refers to splicing multiple tracks to obtain a complete track, such as the newly added track and the reference to the existing track in a track combination in the aforementioned embodiment.

[0139] Deletion may be deletion of a certain track, such as the track involved in the suspicious motion prompt information in the aforementioned embodiment; or deletion of a certain frame of aerial image, such as deletion of a certain frame of aerial image of a specified track.

[0140] Adding refers to adding a trajectory; for example, a reference existing trajectory whose end position point is not in the disappearing area corresponding to the candidate ground target can be reasonably extended so that its end position point is in the disappearing area corresponding to the candidate ground target.

[0141] Regeneration refers to regenerating a track, for example, regenerating the track after deleting a frame of an aerial image of the specified track.

[0142] In the above implementation, the analyst can perform a visual correction operation on the generated trajectory through the interactive interface, thereby facilitating the use of the corrected trajectory for subsequent data analysis and processing.

[0143] It is worth mentioning that since analysts may delete aerial images during the trajectory correction process, the data of the generated trajectory under the aerial image will be missing, so it is necessary to predict and interpolate the data of the trajectory under the aerial image.

[0144] In some implementations, the prediction interpolation may be performed by linear interpolation, and then the interpolated trajectory data may be further smoothed by a smoothing algorithm to generate a smoothed trajectory.

[0145] For example, for track i , which contains the trajectory data: If the analyst deletes one of the aerial images during the correction process, resulting in a missing frame between t0 and tn, linear interpolation can be used to fill the missing frame data. in, To supplement the coordinates of the ground target, The yaw angle of the supplementary ground target.

[0146] After that, the trajectory can be obtained based on the interpolated data through the RTS smoothing algorithm (Rauch Tung Striebel smoother); the RTS smoothing algorithm is a fixed interval optimal smoothing algorithm, which consists of forward filtering and backward filtering. The forward filter is a classic Kalman filter, which is used to estimate the state at each moment; the backward filter is based on the forward filter and reuses part of the data to obtain a more accurate state estimate.

[0147] In the recursive process of forward filtering and backward filtering, the motion state of the ground target is predicted. In order to improve the calculation efficiency, the kinematic model of a bicycle with a fixed rear wheel can be used to predict the motion state of the ground target.

[0148] See also Fig. 9 , Fig. 9 The schematic diagram of the bicycle kinematics model is given as an example. Under the bicycle kinematics model, the motion state quantities of the ground target include: m k =[x k ,y k ,ψ k ,v k ,β k ] T .

[0149] Among them, (x k ,y k ) is the coordinate of the ground target at time k, ψ k That is, the yaw angle of the ground target at time k (the deflection angle in the yaw direction, often described in radians, with the counterclockwise direction being positive), v k That is, the speed of the ground target at time k, β k That is, the orientation angle of the center of mass of the ground target at time k.

[0150] Based on the two-wheeled vehicle model, the Jacobian matrix of the motion state at the kth time point is used as the state transfer matrix, that is, the motion state m at the k+1th time point can be obtained. k+1 =A k m k ; Among them, the Jacobian matrix at the kth time point is as follows:

[0151]

[0152] Among them, a13-a45 can be calculated by the following formula:

[0153] a 13 =-Δt·v k ·sin(ψ k +β k );

[0154] a 14 =Δt·cos(ψ k +β k );

[0155] a 15 =-Δt·v k ·sin(ψ k +β k )·Δβ k ;

[0156] a 23 =-Δt·v k ·cos(ψ k +β k )·Δβ k ;

[0157] a 24 =Δt·sin(ψ k +β k );

[0158] a 25 =Δt·v k ·cos(ψ k +β k )·Δβ k ;

[0159]

[0160] a 45 =0.

[0161] in, β k =tan -1 (tanδ k ·R Lr ),

[0162] Where L f and L r are the lengths from the center of mass of the ground target to the front and rear axes, usually fixed values, δ k is the front wheel turning angle of the ground target at the kth moment, and Δt is the time interval between the kth moment and the k+1th moment.

[0163] After determining the transfer process of the motion state quantity at adjacent moments, the relationship between the observed quantity and the motion state quantity is established, and the observation noise z is initialized. k and the state transition noise matrix H k .

[0164] Among them, z k =[x k ,y k ,ψ k ] T ;

[0165]

[0166] After that, the calculation is forward recursively calculated using the following formula:

[0167] m' k =A k m k-1 ;

[0168]

[0169] m k =m' k +K k (z k -H k m' k );

[0170] P k =(IK k H k )P' k .

[0171] And, reverse recursion calculation is performed through the following formula:

[0172]

[0173] In the above recursive process, P0 and Q0 are estimated using empirical values, while x0, y0, ψ0 in the state value m0 use the observed values ​​at the initial moment.

[0174] Finally, for the track i, starting from the initial state m0, after T times of forward recursion, the m obtained at this time is T and P T Used to initialize the reverse recursive process, that is, After T times of reverse recursion, the RTS smoothing of the trajectory is completed.

[0175] For easier understanding, see Fig.10 , Fig.10 A schematic diagram of a trajectory generation system involved in an embodiment of the present application is given, including a preparation module, a trajectory generation module and a post-processing module.

[0176] Among them, the preparation module is mainly used to perform the following preparations:

[0177] Acquire multiple aerial images; wherein the multiple aerial images are for the same target area.

[0178] Determine a reference image and calibration parameters; wherein the reference image can be any image selected from multiple aerial images; the calibration parameters are to calibrate the parameters of the aerial photography equipment, so as to determine the transformation relationship between the image coordinate system and the world coordinate system.

[0179] Aerial image alignment; specifically, align all aerial images to the reference image.

[0180] Determine the spatial constraint area; specifically, perform semantic segmentation on the reference image to obtain the spatial constraint area in the reference image; the spatial constraint area includes the non-crossable area, the starting area, the disappearing area and the target occlusion area.

[0181] The trajectory generation module is used to generate trajectories, which mainly includes detecting ground targets and generating trajectories.

[0182] Among them, detecting ground targets is to use the target detection model to perform target detection on aerial images to obtain target detection results for each aerial image; wherein the target detection results include motion parameter information of the identified candidate ground targets.

[0183] It should be noted that before using the target detection model for target detection, it is necessary to fine-tune the pre-trained target detection model through training data to improve the detection effect of the target detection model in the target detection task.

[0184] Generate trajectory: Perform trajectory prediction and data association based on the candidate ground targets detected in each aerial image to generate the trajectory of the ground target.

[0185] For the specific process of generating trajectories, please refer to Fig.11 , Fig.11 The embodiment of the present application provides Fig.10 Schematic diagram of trajectory generation.

[0186] After obtaining the target detection result, if there is no existing trajectory, a new trajectory is directly constructed for the candidate ground target; if there is an existing trajectory, the candidate ground target involved in the target detection result is data-associated with the existing trajectory, and if there is an existing trajectory of a target that is the same target as the candidate ground target in the existing trajectory, the position information and motion information of the candidate ground target are associated with the existing trajectory of the target to update the existing trajectory of the target; if there is no existing trajectory of a target that is the same target as the candidate ground target in the existing trajectory, a new trajectory is constructed for the candidate ground target, and the constructed new trajectory and the updated existing trajectory are stored in the trajectory set.

[0187] It is worth mentioning that during the trajectory generation process, prompt information will be generated, including target disappearance prompt information, matching prompt information, and suspicious motion prompt information.

[0188] The post-processing module is used to perform subsequent processing on the generated trajectory, mainly including interactive revision and smoothing.

[0189] Among them, interactive revision refers to the analyst manually revising the trajectory data of the generated trajectory, such as deleting, adding, merging, etc., according to the prompt information generated during the trajectory generation process.

[0190] Smoothing refers to further processing the revised trajectory data through interpolation and smoothing algorithms to obtain a smooth trajectory.

[0191] See also Fig.12 , Fig.12 The embodiment of the present application provides Fig.10 Schematic diagram of the process of model training phase and data preparation phase.

[0192] In the model training phase, after obtaining multiple aerial images, some of the aerial images can be used as sample aerial images to fine-tune the pre-trained target detection model. Specifically, the analyst annotates the selected sample aerial images, and then uses the annotated sample aerial images to fine-tune the pre-trained target detection model, and the fine-tuned target detection model is used for ground target detection.

[0193] For the data preparation stage, one aerial image is selected as the reference image from multiple aerial images that need to be used for ground target detection. Then, the other aerial images are aligned to the reference image. At the same time, the conversion relationship between the image coordinates and the world coordinates is obtained by calibrating the parameters of the aerial photography equipment when shooting the reference image. According to the conversion relationship, a spatial constraint area is constructed in each aligned aerial image, that is, the area that cannot be crossed, the area where the ground target begins to appear, the area where the target is blocked, etc.

[0194] After that, the ground object detection is performed on the aligned aerial images by fine-tuning the trained object detection model. The subsequent detection process is the same as Fig. 9 The processing flow of the trajectory generation module and the post-processing module in are similar and will not be repeated here.

[0195] In some embodiments, see Fig.13 , Fig.13 A schematic diagram of a trajectory generation device provided in an embodiment of the present application is given. The trajectory generation device 600 includes:

[0196] The detection module 610 is used to perform target detection on the aerial image to obtain a target detection result, which includes motion parameter information of the identified candidate ground targets.

[0197] The first calculation module 620 is used to determine the spatial constraint matching degree between the candidate ground target and each existing trajectory based on the relative position relationship between the connecting line between the target position point in the motion parameter information and the end position point of each existing trajectory and the non-crossable area corresponding to the candidate ground target.

[0198] The second calculation module 630 is used to determine the motion constraint matching degree between the candidate ground target and each existing trajectory according to the motion parameter information and the motion information of the ground target corresponding to each existing trajectory.

[0199] The matching module 640 is used to determine, from a plurality of existing trajectories, an existing target trajectory in which the corresponding ground target and the candidate ground target are the same target according to the spatial constraint matching degree and the motion constraint matching degree.

[0200] The adding module 650 is used to associate the position information and motion information of the candidate ground target with the existing track of the target.

[0201] In some embodiments, the target detection result also includes image features of the candidate ground target; the matching module 640 includes a similarity calculation unit, which is used to calculate the feature similarity between the candidate ground target and each existing trajectory based on the image features of the candidate ground target and the image features of the ground target corresponding to each existing trajectory; a comprehensive matching degree calculation unit, which is used to determine the comprehensive matching degree between the candidate ground target and each existing trajectory based on the feature similarity, spatial constraint matching degree and motion constraint matching degree between the candidate ground target and each existing trajectory; a selection unit, which is used to determine, according to the comprehensive matching degree, in multiple existing trajectories that the corresponding ground target is the same target as the candidate ground target.

[0202] In some embodiments, the motion parameter information also includes the target motion direction; the motion information includes the position information of the ground target at the last K position points in the corresponding existing trajectory and the motion direction at the last K position points; K is a positive integer; the second calculation module 630 includes a displacement change calculation unit, which is used to determine the displacement change of the candidate ground target relative to each existing trajectory based on the target position point in the motion parameter information and the position information of the last K position points in each existing trajectory; a motion direction change calculation unit, which is used to determine the motion direction change of the candidate ground target relative to each existing trajectory based on the target motion direction and the motion direction of the ground target corresponding to each existing trajectory at the last K position points; a motion constraint matching degree calculation unit, which is used to determine the motion constraint matching degree between the candidate ground target and each existing trajectory based on the displacement change and motion direction change of the candidate ground target relative to each existing trajectory.

[0203] In some embodiments, for each existing trajectory, the first calculation module 620 is specifically used to determine that the spatial constraint matching degree between the candidate ground target and each existing trajectory is a first matching degree if the line between the target position point in the motion parameter information and the end position point of the existing trajectory intersects with the non-crossable area corresponding to the candidate ground target; if the line between the target position point in the motion parameter information and the end position point of the existing trajectory does not intersect with the non-crossable area corresponding to the candidate ground target, determine that the spatial constraint matching degree between the candidate ground target and each existing trajectory is a second matching degree; the second matching degree is greater than the first matching degree.

[0204] In some embodiments, the trajectory generation device 600 also includes a first prompt module, which is used to determine, from multiple existing trajectories, a reference existing trajectory whose comprehensive matching degree with any candidate ground target in the target detection result is less than a first threshold based on the comprehensive matching degree between the candidate ground target and each existing trajectory; if the end position point in the reference existing trajectory is not in the disappearance area corresponding to the candidate ground target, generate target disappearance prompt information based on the target identifier of the ground target associated with the reference existing trajectory and the image frame number of the aerial image.

[0205] In some embodiments, the trajectory generation device 600 also includes a trajectory construction module, which is used to determine, among the candidate ground targets involved in the target detection result, a reference ground target whose comprehensive matching degree with multiple existing trajectories is less than a first threshold based on the comprehensive matching degree between the candidate ground target and each existing trajectory; determine, among the reference ground targets, a first ground target whose corresponding target position point is located in the starting appearance area based on the kinematic parameter information of the reference ground target; and construct a new trajectory based on the kinematic parameter information of the first ground target; the starting position point of the new trajectory is the target position point in the kinematic parameter information of the first ground target.

[0206] In some embodiments, the trajectory generation device 600 also includes a second prompt module, which is used to determine a trajectory combination based on the starting position point in the newly added trajectory and the ending position point in the reference existing trajectory, where a trajectory combination includes a newly added trajectory and a reference existing trajectory, and the starting position point of the newly added trajectory and the ending position point in the reference existing trajectory in the same trajectory combination are located in the same target occlusion area; the reference existing trajectory refers to an existing trajectory among multiple existing trajectories whose comprehensive matching degree with any candidate ground target in the target detection result is less than a first threshold; matching prompt information is generated according to the trajectory combination.

[0207] In some embodiments, the trajectory generation device 600 also includes a third prompt module, which is used to generate suspicious motion prompt information based on the trajectory identifier of the target's existing trajectory and the image frame number of the aerial image if the motion constraint matching degree between the candidate ground target and the target's existing trajectory is less than a second matching degree threshold.

[0208] In some embodiments, the trajectory generating device 600 further includes an image segmentation module for performing semantic segmentation on the aerial image to determine a non-crossable area, a disappearing area, a starting area, and a target occluded area in the ground area presented by the aerial image.

[0209] In some implementations, according to the trajectory generation method provided in the above embodiment, the present application also provides an electronic device, such as Fig.14 , Fig.14 A structural block diagram of an electronic device provided in an embodiment of the present application is given, where the electronic device 700 includes a processor 710; a memory 720; and computer-readable instructions are stored on the memory 720. When the computer-readable instructions are executed by the processor 710, the above method is implemented.

[0210] The electronic device 700 may be a terminal device, and the terminal device may be a vehicle-mounted terminal, a desktop computer, a laptop computer, etc.

[0211] The processor 710 may include one or more processing cores. The processor 710 uses various interfaces and lines to connect the various parts of the entire wearable device, and executes various functions of the wearable device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 720, and calling data stored in the memory 720. Optionally, the processor 710 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 710 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but may be implemented separately through a communication chip.

[0212] The memory 720 may include a random access memory (RAM) or a read-only memory (ROM). The memory 720 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 720 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the electronic device during use.

[0213] In some embodiments, the present application further provides a computer-readable storage medium having computer-readable instructions stored thereon, and when the computer-readable instructions are executed by the processor 710, the above method is implemented.

[0214] The computer-readable storage medium may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for computer-readable instructions for executing any method step of the above method. These computer-readable instructions may be read from or written to one or more computer program products. The computer-readable instructions may be compressed in an appropriate form.

[0215] In particular, according to an embodiment of the present application, the process described above can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer instruction. When the computer instruction is executed by a central processing unit (CPU), various functions defined in the system of the present application are executed.

[0216] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of the overall module or unit of the module or unit function.

[0217] The above are only preferred embodiments of the present application, and are not intended to limit the present application in any form. Although the present application has been disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technical personnel in the field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present application. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. A trajectory generation method, characterized in that: include: Performing target detection on the aerial image to obtain a target detection result, wherein the target detection result includes motion parameter information of the identified candidate ground targets; Determine the spatial constraint matching degree between the candidate ground target and each of the existing trajectories according to the relative position relationship between the connecting line between the target position point in the motion parameter information and the end position point of each of the existing trajectories and the non-crossable area corresponding to the candidate ground target; Determining a motion constraint matching degree between the candidate ground target and each of the existing trajectories according to the motion parameter information and the motion information of the ground target corresponding to each of the existing trajectories; According to the spatial constraint matching degree and the motion constraint matching degree, determining from the plurality of existing trajectories an existing target trajectory in which the corresponding ground target and the candidate ground target are the same target; The position information and the motion information of the candidate ground target are associated and added to the existing track of the target.

2. The method according to claim 1, characterized in that: The target detection result also includes image features of the candidate ground target; The step of determining, from the plurality of existing trajectories, a target existing trajectory that the corresponding ground target and the candidate ground target are the same target according to the spatial constraint matching degree and the motion constraint matching degree, comprises: Calculating the feature similarity between the candidate ground target and each of the existing trajectories according to the image features of the candidate ground target and the image features of the ground targets corresponding to each of the existing trajectories; Determining a comprehensive matching degree between the candidate ground target and each of the existing trajectories according to a feature similarity, a spatial constraint matching degree, and a motion constraint matching degree between the candidate ground target and each of the existing trajectories; According to the comprehensive matching degree, an existing target track is determined in which the corresponding ground target and the candidate ground target are the same target among the multiple existing tracks.

3. The method according to claim 1, characterized in that: The motion parameter information also includes the target motion direction; the motion information includes the position information of the last K position points of the ground target in the corresponding existing trajectory and the motion direction at the last K position points; K is a positive integer; The determining, according to the motion parameter information and the motion information of the ground target corresponding to each of the existing trajectories, the motion constraint matching degree between the candidate ground target and each of the existing trajectories comprises: Determine the displacement change of the candidate ground target relative to each existing trajectory according to the target position point in the motion parameter information and the position information of the last K position points in each of the existing trajectories; Determine the change in the movement direction of the candidate ground target relative to each existing trajectory according to the target movement direction and the movement directions of the ground targets corresponding to each existing trajectory at the last K position points; The motion constraint matching degree between the candidate ground target and each of the existing trajectories is determined according to the displacement change and the motion direction change of the candidate ground target relative to each of the existing trajectories.

4. The method according to claim 1, characterized in that: The determining of the spatial constraint matching degree between the candidate ground target and each of the existing trajectories according to the relative position relationship between the connecting line between the target position point in the motion parameter information and the end position point of each of the existing trajectories and the non-crossable area corresponding to the candidate ground target includes: For each of the existing trajectories, perform the following: If a line between the target position point in the motion parameter information and the end position point of the existing trajectory intersects with a non-crossable area corresponding to the candidate ground target, determining that the spatial constraint matching degree between the candidate ground target and each of the existing trajectories is a first matching degree; If the line between the target position point in the motion parameter information and the end position point of the existing trajectory does not intersect with the non-crossable area corresponding to the candidate ground target, it is determined that the spatial constraint matching degree between the candidate ground target and each of the existing trajectories is the second matching degree; the second matching degree is greater than the first matching degree.

5. The method according to claim 2, characterized in that: After determining the comprehensive matching degree between the candidate ground target and each of the existing trajectories according to the feature similarity, spatial constraint matching degree and motion constraint matching degree between the candidate ground target and each of the existing trajectories, the method further includes: According to the comprehensive matching degree between the candidate ground target and each of the existing trajectories, determining, from the multiple existing trajectories, a reference existing trajectory whose comprehensive matching degree with any candidate ground target in the target detection result is less than a first threshold; If the end position point in the reference existing trajectory is not in the disappearance area corresponding to the candidate ground target, target disappearance prompt information is generated according to the target identifier of the ground target associated with the reference existing trajectory and the image frame number of the aerial image.

6. The method according to claim 2, characterized in that After determining the comprehensive matching degree between the candidate ground target and each of the existing trajectories according to the feature similarity, spatial constraint matching degree and motion constraint matching degree between the candidate ground target and each of the existing trajectories, the method further includes: According to the comprehensive matching degree between the candidate ground target and each of the existing tracks, determining a reference ground target whose comprehensive matching degree with the multiple existing tracks is less than a first threshold among the candidate ground targets involved in the target detection result; According to the kinematic parameter information of the reference ground target, determining in the reference ground target a first ground target whose corresponding target position point is located in the starting appearance area; A newly added trajectory is constructed according to the kinematic parameter information of the first ground target; the starting position point of the newly added trajectory is the target position point in the kinematic parameter information of the first ground target.

7. The method according to claim 5 or 6, characterized in that: After constructing the newly added trajectory according to the kinematic parameter information of the first ground target, the method further includes: Determine a trajectory combination according to a starting position point in a newly added trajectory and an ending position point in a reference existing trajectory, wherein a trajectory combination includes a newly added trajectory and a reference existing trajectory, and a starting position point of the newly added trajectory and an ending position point in the reference existing trajectory in the same trajectory combination are located in the same target occlusion area; the reference existing trajectory refers to an existing trajectory among the multiple existing trajectories, the comprehensive matching degree between which and any candidate ground target in the target detection result is less than a first threshold; According to the trajectory combination, matching prompt information is generated.

8. The method according to claim 2, characterized in that: After associating the position information and the motion information of the candidate ground target with each other and adding them to the existing track of the target, the method further comprises: If the motion constraint matching degree between the candidate ground target and the existing trajectory of the target is less than a second matching degree threshold, suspicious motion prompt information is generated according to the trajectory identifier of the existing trajectory of the target and the image frame number of the aerial image.

9. The method according to claim 7, characterized in that: The method further comprises: Semantic segmentation is performed on the aerial image to determine the non-crossable area, the disappearing area, the starting area and the target occluded area in the ground area presented by the aerial image.

10. A trajectory generating device, characterized in that: include: A detection module is used to perform target detection on the aerial image to obtain a target detection result, wherein the target detection result includes motion parameter information of the identified candidate ground targets; A first calculation module is used to determine the spatial constraint matching degree between the candidate ground target and each of the existing trajectories according to the relative position relationship between the connecting line between the target position point in the motion parameter information and the end position point of each of the existing trajectories and the non-crossable area corresponding to the candidate ground target; A second calculation module is used to determine the motion constraint matching degree between the candidate ground target and each of the existing trajectories according to the motion parameter information and the motion information of the ground target corresponding to each of the existing trajectories; A matching module, configured to determine, from the plurality of existing trajectories, an existing target trajectory that the corresponding ground target is the same target as the candidate ground target, according to the spatial constraint matching degree and the motion constraint matching degree; The adding module is used to associate the position information and the motion information of the candidate ground target with each other and add them to the existing track of the target.

11. An electronic device, characterized in that: include: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images