A target trajectory generation method and device, electronic equipment and medium

Abstract

Embodiments of the present application provide a target trajectory generation method and device, electronic equipment and medium. The method comprises: for each image group collected by each image collection device, obtaining the image position of each target in each image frame in the image group; converting the image position into a position in a world coordinate system according to a preset conversion relationship to obtain a world position corresponding to the image position; for each image frame set, fusing the world positions of each same target in each image frame in the image frame set to obtain a fused position of the target; and for each target, associating the fused position of the target according to the collection time sequence of the image frame set corresponding to the target to generate a fused trajectory of the target. The method can generate trajectories of multiple targets and can adapt to the trajectory generation requirements of multiple complex targets in scenarios such as urban traffic.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and medium for generating target trajectories. Background Technology

[0002] In some surveillance scenarios, multiple cameras are installed to capture moving targets. For example, multiple cameras are typically installed at intersections to capture video of vehicles and other targets at the intersection. Similarly, multiple cameras can be installed in animal enclosures to capture the animals. The captured video footage can then be used to generate movement trajectories for vehicles and animals, and these trajectories can be used to determine if vehicles have committed traffic violations such as running red lights, or if animals are exhibiting any unusual behavior.

[0003] The current methods for generating target movement trajectories are mainly as follows: detect targets in surveillance videos, then build a target re-identification model to match the starting position of a specific target to be identified in other cameras, and finally obtain its movement trajectory by analyzing the target trajectory in both forward and reverse directions.

[0004] However, the above methods can only generate the trajectory of a specific target and cannot meet the needs of generating trajectories for multiple complex targets in scenarios such as urban traffic. Summary of the Invention

[0005] The purpose of this invention is to provide a method, apparatus, electronic device, and medium for generating target trajectories, so as to generate the movement trajectories of multiple targets.

[0006] In a first aspect, embodiments of the present invention provide a method for generating a target trajectory, comprising:

[0007] For each image group acquired by each image acquisition device, obtain the image location of each target in each image frame of that image group;

[0008] According to a preset transformation relationship, the image position is converted into a position in the world coordinate system to obtain the world position corresponding to the image position;

[0009] For each set of image frames, the world positions of the same target in each image frame in the set of image frames are fused to obtain the fused position of the target; wherein, the set of image frames is a set of image frames that are acquired at the same time.

[0010] For each target, the fusion position of the target is associated with the acquisition time sequence of the image frame set corresponding to the target to generate the fusion trajectory of the target.

[0011] Optionally, the number of image acquisition devices is at least 3;

[0012] The step of fusing the world positions of the same target in each image frame of the image frame set to obtain the fused position of the target includes:

[0013] For any two image frames in the image frame set, construct a similarity matrix for each target in the two image frames based on the world position of each target in the image frame set;

[0014] The similarity matrix is ​​calculated using the Hungarian algorithm. If the similarity between two targets at corresponding positions in the similarity matrix and located in different image frames is greater than a preset similarity threshold, the two targets are determined to be the same target.

[0015] The world positions of the same target in the image frame set are fused to obtain the fused position, and the fused position is used as the fused position of the target corresponding to the same target.

[0016] Optionally, the step of constructing a similarity matrix for each target in any two image frames in the image frame set, based on the world location of each target in the image frame set, includes:

[0017] Based on the world position of each target in the image frame set, determine the velocity direction of the target;

[0018] For any two image frames in the image frame set, a similarity matrix is ​​constructed for each target in the two image frames based on the world position and velocity direction of each target in the two image frames.

[0019] Optionally, after fusing the world locations of the same targets in the image frame set to obtain the fused location, and using the fused location as the fused location of the fused target corresponding to the same target, the method further includes:

[0020] For each fusion target, if there are at least two targets from the same image group among the multiple identical targets corresponding to the fusion target, remove the targets among the at least two targets whose distance from the fusion target is not the minimum distance; and update the fusion position of the fusion target.

[0021] Optionally, before generating the fusion trajectory of a target by associating its fusion position with the acquisition time sequence of the image frame set corresponding to that target for each target, the method further includes:

[0022] For each image group, a single-camera trajectory for each target is generated based on multiple world locations of each target in that image group;

[0023] For each target, the fusion position of the target is associated with the acquisition time sequence of the image frame set corresponding to the target to generate the fusion trajectory of the target, including:

[0024] For each fusion target, if the current fusion position of the fusion target and the fusion position in the existing trajectory both originate from the same single camera trajectory, it is determined that the current fusion position of the fusion target is associated with the existing trajectory; wherein, the existing trajectory is the trajectory formed by associating the fusion positions of various targets acquired before the current fusion position of the fusion target.

[0025] If the current fusion position of the target does not originate from the same single camera trajectory as any existing trajectory, and the current fusion position of the target is in a preset motion state, then the current fusion position of the target is determined to be unrelated to the existing trajectory.

[0026] If the current fusion position of the fusion target does not originate from the same single camera trajectory as any existing trajectory fusion position, and the current fusion position of the fusion target is not in a preset motion state, construct the correlation matrix between each fusion target and the existing trajectory in the image frame corresponding to the fusion target.

[0027] The correlation matrix is ​​calculated using the Hungarian algorithm. If the correlation between the fusion target and the existing trajectory at the corresponding position in the correlation matrix is ​​greater than a preset correlation threshold, the current fusion position of the fusion target is determined to be associated with the existing trajectory.

[0028] If all the fusion positions corresponding to the fusion target are associated, the fusion trajectory of the fusion target is generated based on all the associated fusion positions of the fusion target.

[0029] Optionally, after determining that the fusion target is associated with the existing trajectory, the method further includes:

[0030] The existing trajectory is updated based on the current fusion position of the fusion target.

[0031] Optionally, before fusing the world positions of the same target in each image frame of each image frame set to obtain the fused position of the target, the method further includes:

[0032] For each image group, a single-camera trajectory for each target is generated based on multiple world locations of each target in that image group;

[0033] Match the single-camera trajectories in any two image groups;

[0034] Choose any set of matching single-camera trajectories, and calculate the distance between the start and end positions of the first single-camera trajectory in the set and each position of the second single-camera trajectory; and calculate the distance between the start and end positions of the second single-camera trajectory in the set and each position of the first single-camera trajectory.

[0035] If there is a position in the second single-camera trajectory that is less than a preset distance threshold from the start or end position of the first single-camera trajectory, and there is a position in the first single-camera trajectory that is less than a preset distance threshold from the start or end position of the second single-camera trajectory, the matching single-camera trajectories in this group are retained; otherwise, the single-camera trajectories in this group are deleted.

[0036] For each set of matched single-camera trajectories retained, the shortest distance is selected from the distance between the start and end positions of the first single-camera trajectory in the set and each position of the second single-camera trajectory, and the distance between the start and end positions of the second single-camera trajectory in the set and each position of the first single-camera trajectory. The image frames where the two positions corresponding to the shortest distance are located are determined as the primary synchronization image frames.

[0037] For any two image groups, calculate the average acquisition time of each pair of primary synchronization image frames in the two image groups, and take the image frames in the two image groups whose acquisition time is the average time as the final synchronization image frames of the two image groups.

[0038] Secondly, embodiments of the present invention also provide a target trajectory generation apparatus, comprising:

[0039] The location acquisition module is used to acquire the image location of each target in each image frame of each image group acquired by each image acquisition device.

[0040] The position conversion module is used to convert the image position into a position in the world coordinate system according to a preset conversion relationship, so as to obtain the world position corresponding to the image position;

[0041] The location fusion module is used to fuse the world positions of the same target in each image frame in each image frame set to obtain the fused position of the target; wherein, the image frame set is an image set composed of image frames acquired at the same time.

[0042] The trajectory generation module is used to associate the fusion position of each target with the acquisition time sequence of the image frame set corresponding to the target, and generate the fusion trajectory of the target.

[0043] Optionally, the number of image acquisition devices is at least 3;

[0044] The location fusion module includes:

[0045] The similarity matrix determination submodule is used to construct a similarity matrix for each target in any two image frames in the image frame set, based on the world position of each target in the image frame set.

[0046] The target determination submodule is used to solve the similarity matrix according to the Hungarian algorithm. If the similarity between two targets at corresponding positions in the similarity matrix and located in different image frames is greater than a preset similarity threshold, the two targets are determined to be the same target.

[0047] The location fusion submodule is used to fuse the world locations of the same targets in the image frame set to obtain the fused location, and use the fused location as the fused location of the fused target corresponding to the same target.

[0048] Optionally, the similarity matrix determination submodule is specifically used to determine the velocity direction of each target based on the world position of each target in the image frame set; and for any two image frames in the image frame set, to construct a similarity matrix of each target in the two image frames based on the world position and velocity direction of each target in the two image frames.

[0049] Optionally, the device further includes:

[0050] The fusion position update module is used to, for each fusion target, if there are at least two targets from the same image group among the multiple identical targets corresponding to the fusion target, remove the targets among the at least two targets whose distance from the fusion target is not the minimum distance; and update the fusion position of the fusion target.

[0051] Optionally, the device further includes:

[0052] A single-camera trajectory generation module is used to generate a single-camera trajectory for each target in each image group based on multiple world positions of each target in the image group.

[0053] The trajectory generation module is specifically used for each fusion target. If the current fusion position of the fusion target and the fusion position in the existing trajectory all originate from the same single-camera trajectory, the module determines that the current fusion position of the fusion target is associated with the existing trajectory. The existing trajectory is the trajectory formed by associating the fusion positions of various targets acquired before the current fusion position of the fusion target. If the current fusion position of the fusion target and the fusion position in any existing trajectory do not originate from the same single-camera trajectory, and the current fusion position of the fusion target is in a preset motion state, the module constructs a correlation matrix between each fusion target and the existing trajectory in the image frame corresponding to the fusion target. The module solves the correlation matrix using the Hungarian algorithm. If the calculated correlation between the fusion target and the existing trajectory at the corresponding position in the correlation matrix is ​​greater than a preset correlation threshold, the module determines that the current fusion position of the fusion target is associated with the existing trajectory. If all fusion positions corresponding to the fusion target are associated, the module generates the fusion trajectory of the fusion target based on all associated fusion positions.

[0054] Optionally, the device further includes:

[0055] The trajectory update module is used to update the existing trajectory based on the current fusion location of the fusion target.

[0056] Optionally, the device further includes:

[0057] The synchronization time determination module is used to generate a single-camera trajectory for each target in each image group based on multiple world positions of each target in the image group; match the single-camera trajectories in any two image groups; select any set of matched single-camera trajectories and calculate the distance between the start and end positions of the first single-camera trajectory in the set and each position of the second single-camera trajectory; and calculate the distance between the start and end positions of the second single-camera trajectory in the set and each position of the first single-camera trajectory; if there is a position in the second single-camera trajectory whose distance to the start or end position of the first single-camera trajectory is less than a preset distance threshold, and there is a position in the first single-camera trajectory whose distance to the start or end position of the second single-camera trajectory is less than... For each set of matching single-camera tracks, retain the matching single-camera tracks at the preset distance threshold position; otherwise, delete the single-camera tracks in that set. For each set of matching single-camera tracks retained, select the shortest distance from the distance between the start and end positions of the first single-camera track in that set to each position of the second single-camera track, and from the distance between the start and end positions of the second single-camera track in that set to each position of the first single-camera track. Determine the image frames corresponding to the two positions with the shortest distance as the primary synchronization image frames. For any two image sets, calculate the average acquisition time of each pair of primary synchronization image frames in the two image sets, and use the image frames in the two image sets whose acquisition time is the average time as the final synchronization image frames of the two image sets.

[0058] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0059] Memory, used to store computer programs;

[0060] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect above.

[0061] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the steps described in the first aspect above.

[0062] Beneficial effects of the embodiments of the present invention:

[0063] The method provided in this embodiment of the invention involves obtaining the image positions of each target in each image frame of an image group acquired by each image acquisition device; converting the image positions to positions in the world coordinate system according to a preset transformation relationship to obtain the world positions corresponding to the image positions; fusing the world positions of the same target in each image frame of each image frame in the image frame set to obtain the fused position of the target; and associating the fused positions of each target with the target according to the acquisition time sequence of the image frame set corresponding to the target to generate the fused trajectory of the target. In other words, the method provided in this embodiment of the invention can generate trajectories of multiple targets, adapting to the trajectory generation needs of multiple complex targets in scenarios such as urban traffic.

[0064] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0066] Figure 1 A flowchart illustrating a method for generating a target trajectory according to an embodiment of the present invention;

[0067] Figure 2 A flowchart illustrating an automatic correction method provided in an embodiment of the present invention;

[0068] Figure 3 A flowchart of a location fusion method provided in an embodiment of the present invention;

[0069] Figure 4 A flowchart for generating a target fusion trajectory provided in an embodiment of the present invention;

[0070] Figure 5 This is a schematic flowchart of a trajectory correction method provided in an embodiment of the present invention;

[0071] Figure 6 A schematic diagram of a target trajectory generation device provided in an embodiment of the present invention;

[0072] Figure 7 Another schematic diagram of the target trajectory generation device provided in an embodiment of the present invention;

[0073] Figure 8 A schematic diagram of another structure of the target trajectory generation device provided in an embodiment of the present invention;

[0074] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0075] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0076] To generate the trajectory of multiple targets and meet the trajectory generation needs of multiple complex targets in scenarios such as urban traffic, embodiments of the present invention provide a method, apparatus, electronic device, storage medium, and computer program product for generating target trajectories.

[0077] The method for generating a target trajectory provided in the embodiments of the present invention will be described below. The method for generating a target trajectory provided in the embodiments of the present invention can be applied to any electronic device with image processing capabilities, and is not specifically limited thereto.

[0078] Figure 1 A flowchart of a target trajectory generation method provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes:

[0079] S101: For each image group acquired by the image acquisition device, obtain the image location of each target in each image frame of the image group.

[0080] S102, according to the preset transformation relationship, the image position is converted into a position in the world coordinate system to obtain the world position corresponding to the image position.

[0081] S103, for each set of image frames, fuse the world positions of the same target in each image frame in the set of image frames to obtain the fused position of the target.

[0082] The image frame set is an image set composed of various image frames acquired at the same time.

[0083] S104. For each target, based on the acquisition time sequence of the image frame set corresponding to the target, the fusion position of the target is associated to generate the fusion trajectory of the target.

[0084] The method provided in this embodiment of the invention involves obtaining the image positions of each target in each image frame of an image group acquired by each image acquisition device; converting the image positions to positions in the world coordinate system according to a preset transformation relationship to obtain the world positions corresponding to the image positions; fusing the world positions of the same target in each image frame of each image frame in the image frame set to obtain the fused position of the target; and associating the fused positions of each target with the target according to the acquisition time sequence of the image frame set corresponding to the target to generate the fused trajectory of the target. In other words, the method provided in this embodiment of the invention can generate trajectories of multiple targets, adapting to the trajectory generation needs of multiple complex targets in scenarios such as urban traffic.

[0085] This invention can be applied to scenarios such as intersections. In this embodiment, multiple image acquisition devices are deployed in the specific application scenario. Each image acquisition device has a different position and angle, and can acquire images from different angles within the application scenario. Each image acquisition device can acquire multiple frames of images, forming an image group. Each frame of image can include one or more targets in the application scenario, specifically movable targets such as animals and vehicles. The image acquisition devices can specifically be cameras and video recorders.

[0086] For each image acquisition device, due to the different angles at which the image acquisition devices are set up in actual application scenarios, the image position of the target in the image acquired by the image acquisition device is not consistent with the corresponding position of the target in the real world. Furthermore, application scenarios such as urban traffic require the target's trajectory to be displayed on a high-precision map to achieve accurate positioning of the target on the high-precision map. Therefore, it is necessary to convert the image position of each target in each image frame into the position in the world coordinate system to obtain the world position of the target.

[0087] In this embodiment of the invention, for each image acquisition device, the transformation relationship between the image position of the target acquired by the image acquisition device and its world position in the world coordinate system can be determined. Specifically, a set of world position coordinates of a specified target in the world coordinate system in the image frame acquired by the image acquisition device can be obtained. For example, a set of world position coordinates of a specified target in the world coordinate system can be obtained through actual distance measurement or from a high-precision map. The specified targets can be selected as lane line stop points, guide arrows, road fixtures, and other targets with obvious features. Then, for each image acquisition device, the coordinate transformation matrix corresponding to the image acquisition device can be calculated using the following formula based on the world position coordinates of multiple specified targets in the world coordinate system corresponding to the selected image acquisition device and the image position coordinates of multiple specified targets in the image frame:

[0088]

[0089] The above formula can also be expressed as:

[0090] or,

[0091]

[0092] in, Let a be the coordinate transformation matrix corresponding to the image acquisition device. 11 -a 33 All are transformation parameters in the coordinate transformation matrix. x′, y′, and w′ are the x-coordinate, y-coordinate, and z-coordinate of the specified target's world position coordinates, respectively, while u and v are the x-coordinate and y-coordinate of the specified target's image position coordinates, respectively. Since the plane height direction is normalized before and after the position coordinate transformation, a33 is 1 and w′ is 1.

[0093] Furthermore, the image coordinates and world position coordinates can form the following system of equations, solving for 8 parameters a11, a12, a13, a21, a22, a23, a31, a32. At least 4 pairs of image coordinates and world position coordinates are needed. By solving the following linear equations using least squares, the coordinate transformation matrix corresponding to the image acquisition device can be determined.

[0094]

[0095] In the actual calculation of the coordinate transformation matrix corresponding to the image acquisition device, due to potential distortions in the image acquisition device, it is necessary to correct the coordinate transformation matrix by incorporating the device's parameters. Specifically, the coordinate transformation matrix calculated based on a set of specified targets can be multiplied by the parameters of the image acquisition device to obtain a new coordinate transformation matrix. Furthermore, for application scenarios with complex road conditions, the coordinate transformation matrix can be calculated separately for each set of specified targets in the image frames acquired by the image acquisition device. Then, the average of these multiple sets of coordinate transformation matrices can be used as the coordinate transformation matrix corresponding to the image acquisition device. Combining the coordinate transformation matrices corresponding to multiple sets of specified targets provides more complete information and allows for a more accurate representation of the mapping relationship between image location coordinates and world location coordinates.

[0096] In this embodiment of the invention, after obtaining the coordinate transformation matrix corresponding to each image acquisition device in advance, the coordinate transformation matrix can be used as a preset transformation relationship to convert the image position coordinates of each target in the image frame acquired by the image acquisition device into world position coordinates in the world coordinate system. For example, the following formula can be used to convert the image position into a position in the world coordinate system to obtain the world position corresponding to the image position:

[0097] or,

[0098]

[0099] in, Let a be the coordinate transformation matrix corresponding to the image acquisition device. 11 -a 33 All of these are transformation parameters in the coordinate transformation matrix, where x′, y′, and w′ are the horizontal, vertical, and ordinate coordinates of the world position, respectively, and u and v are the horizontal and vertical coordinates of the image position, respectively.

[0100] In this embodiment of the invention, image frame synchronization processing is required for different image acquisition devices to ensure that the positions of the same targets in subsequent location fusion are at the same time. Specifically, time synchronization correction can be performed on each image acquisition device manually to ensure that each image acquisition device starts acquiring images simultaneously and at the same frequency. Alternatively, in this embodiment, the set of image frames acquired by different image acquisition devices at the same time can be defined as an image frame set based on the timestamps of the image frames acquired by the image acquisition devices, ensuring that the acquisition time of the image frames included in an image frame set is synchronized.

[0101] In one possible implementation, automatic calibration can be used to synchronously calibrate each image acquisition device. Specifically... Figure 2 A flowchart of an automatic correction method provided in an embodiment of the present invention is shown below. Figure 2 As shown, the correction method includes:

[0102] S201, For each image group, generate a single-camera trajectory for the target based on multiple world positions of each target in the image group.

[0103] Each image group consists of multiple image frames acquired by the same image acquisition device.

[0104] Specifically, the single-camera trajectory of the target can be generated using the following steps A1-A2:

[0105] Step A1: For each image group, each target in the first image frame of that image group can be used as a single-camera trajectory, and a single-camera trajectory identifier for each single-camera trajectory can be generated. In other words, for each image group, a single-camera trajectory corresponding to each target in the first image frame of that image group can be generated, and a single-camera trajectory identifier for each single-camera trajectory can be generated.

[0106] For example, if the first image frame in an image group includes target 1, target 2, and target 3, then corresponding single-camera trajectories can be generated for target 1, target 2, and target 3 respectively, and identifiers for the single-camera trajectories can be generated respectively, for example, trajectory 1, trajectory 2, and trajectory 3.

[0107] Step A2: For each target in the next image frame of the image group, the minimum distance between the target and each single-camera trajectory in the previous frame of the image group can be calculated based on the world position of the target. If the minimum distance is less than a preset threshold, it is determined that the target in the image frame is the same as the target in the single-camera trajectory, and the world position of the target is used as the trajectory point of the single-camera trajectory to update the single-camera trajectory.

[0108] In other words, for each target in the next image frame of the image group, the minimum distance between the target and each single-camera trajectory in the previous frame of the image group can be calculated based on the target's world position when the next image frame is acquired. If the minimum distance is less than a preset threshold, it means that the target's world position when the next image frame is acquired conforms to the target's movement pattern represented by the single-camera trajectory corresponding to the minimum distance. In this case, it can be determined that the target in the image frame is the same as the target corresponding to the single-camera trajectory, and the target's world position is used as the trajectory point of the single-camera trajectory to update the single-camera trajectory.

[0109] The preset threshold can be set according to the actual application. For example, if the target is a vehicle, the preset threshold can be determined based on the average speed of the vehicle at the intersection and the time interval between the acquisition of two adjacent video frames in the image group. No specific limitation is made here.

[0110] In this embodiment of the invention, DeepSort (a multi-target analysis algorithm) and FairMot (a multi-target analysis algorithm) can also be used to generate the single-camera trajectory of the target.

[0111] S202, Match the single-camera trajectories in any two image groups.

[0112] In this embodiment, if the target has a vehicle identifier, the single-camera trajectories in any two image groups can be matched based on the vehicle identifier of the target, and the single-camera trajectories of targets with the same vehicle identifier are considered as a set of matched single-camera trajectories. The vehicle identifier is used to uniquely identify a vehicle target.

[0113] In this embodiment, the matching degree between single-camera trajectories in any two image groups can also be calculated based on the distance between the single-camera trajectories. The pair of single-camera trajectories with the highest matching degree is then taken as a set of matched single-camera trajectories. For example, for any two image groups, image group A consisting of image frames acquired by image acquisition device 1 and image group B consisting of image frames acquired by image acquisition device 2, for each single-camera trajectory in image group A, the cosine similarity of the distance between the single-camera trajectory and each single-camera trajectory in image group B can be calculated as the matching degree. That is, the similarity between the single-camera trajectory and each single-camera trajectory in image group B, such as cosine similarity, can be calculated as the matching degree. Then, the single-camera trajectory in image group B with the highest matching degree is selected as the initial matching single-camera trajectory. When the matching degree between the initial matching single-camera trajectory and the single-camera trajectory reaches a preset matching degree threshold, the single-camera trajectory and the initial matching single-camera trajectory can be determined as a set of matched single-camera trajectories. The preset matching degree threshold can be set according to the actual application scenario, and is not specifically limited here. In this embodiment, after the initial matching degree between the single camera trajectory and the single camera trajectory reaches a preset matching degree threshold, it can be further determined whether the lane number of the lane where the single camera trajectory is located is consistent with the lane number of the lane where the initial matching single camera trajectory is located. If they are consistent, the single camera trajectory and the initial matching single camera trajectory can be determined as a set of matched single camera trajectories.

[0114] A set of matched single-camera trajectories includes a first single-camera trajectory and a second single-camera trajectory from different image groups.

[0115] S203, select any set of matching single-camera trajectories, and calculate the distance between the start position and end position of the first single-camera trajectory in the set and each position of the second single-camera trajectory; and calculate the distance between the start position and end position of the second single-camera trajectory in the set and each position of the first single-camera trajectory.

[0116] In other words, for any set of matching single-camera trajectories, calculate the distance from the starting position of the first single-camera trajectory to each position of the second single-camera trajectory, and calculate the distance from the ending position of the first single-camera trajectory to each position of the second single-camera trajectory.

[0117] Specifically, this step calculates the distance between the world position coordinates of the locations.

[0118] For example, a single-camera trajectory A in image group 1 matches a single-camera trajectory B in image group 2, where single-camera trajectory A is the first single-camera trajectory and single-camera trajectory B is the second single-camera trajectory. In this step, the distance between the starting position of single-camera trajectory A and each position of single-camera trajectory B can be calculated, as can the distance between the ending position of single-camera trajectory A and each position of single-camera trajectory B, and the distance between the starting and ending positions of single-camera trajectory B and each position of single-camera trajectory A, as well as the distance between the ending position of single-camera trajectory B and each position of single-camera trajectory A, can be calculated.

[0119] S204, if there is a position in the second single-camera trajectory where the distance between the starting position and the ending position of the first single-camera trajectory is less than a preset distance threshold, and there is a position in the first single-camera trajectory where the distance between the starting position and the ending position of the second single-camera trajectory is less than a preset distance threshold, retain the matching single-camera trajectories in this group; otherwise, delete the single-camera trajectories in this group.

[0120] In this embodiment of the invention, the preset distance threshold can be set to 1 meter or 2 meters, etc.

[0121] For example, if the preset distance threshold is set to 1 meter, for a matching single-camera trajectory A and a single-camera trajectory B, if the distance between the third position in single-camera trajectory B and the starting position of single-camera trajectory A is less than 1 meter, and the distance between the fourth position in single-camera trajectory A and the starting position of single-camera trajectory B is less than 1 meter, then the matching single-camera trajectory A and single-camera trajectory B are retained.

[0122] If the distance between any position in single-camera trajectory B and the starting position of single-camera trajectory A is not less than 1 meter, and the distance between any position in single-camera trajectory B and the ending position of single-camera trajectory A is not less than 1 meter, and the distance between any position in single-camera trajectory A and the starting position of single-camera trajectory B is not less than 1 meter, and the distance between any position in single-camera trajectory A and the ending position of single-camera trajectory B is not less than 1 meter, then delete the matching single-camera trajectory A and single-camera trajectory B.

[0123] S205, for each set of matched single-camera tracks retained, select the shortest distance from the distance between the start and end positions of the first single-camera track in the set to each position of the second single-camera track, and the distance between the start and end positions of the second single-camera track in the set to each position of the first single-camera track, and determine the image frames where the two positions corresponding to the shortest distance are respectively the primary synchronization image frames.

[0124] For example, if matching single-camera tracks A and B are preserved, for the matching single-camera tracks A and B, if the distance between the starting and ending positions of single-camera track A and each position of single-camera track B, and the distance between the starting and ending positions of single-camera track B and each position of single-camera track A, is the shortest, then the image frame containing the starting position of single-camera track A and the image frame containing the third position of single-camera track B can be designated as the primary synchronization image frame.

[0125] S206. For any two image groups, calculate the average acquisition time of each pair of primary synchronization image frames in the two image groups, and take the image frames in the two image groups whose acquisition time is the average time as the final synchronization image frames of the two image groups.

[0126] For example, considering image group 1 and image group 2, if image frame A1 in image group 1 and image frame B1 in image group 2 are primary synchronization image frames, and image frame A2 in image group 1 and image frame B2 in image group 2 are also primary synchronization image frames, where the acquisition time of image frame A1 is t... A1 The acquisition time of image frame A2 is t. A2 The acquisition time of image frame B1 is t. B1 The acquisition time for image frame B2 is t. B2 Then the average time t can be calculated. 平均 =(t A1 +t A2 +t B1 +t B2 ) / 4. Then the image group 1 with acquisition time t... 平均 The acquisition time for image frame A3 and image group 2 is t. 平均 Image frame B3 serves as the final synchronized image frame for the two image groups.

[0127] If the image group does not contain an image acquisition time of t 平均 If the image frames are such that the acquisition time and t are determined in the image group, then the relationship between the acquisition time and t can be determined. 平均 The image frame with the smallest difference between the two is taken as the final synchronized image frame of the image group.

[0128] Furthermore, the acquisition time of image frames in image group 1 that are the same number of frames after and separated from image frame A3, and the acquisition time of image frames in image group 2 that are the same number of frames after and separated from image frame B3, are synchronized. In other words, if the acquisition time of image frame A3 is synchronized with that of image frame B3, then the acquisition time of each corresponding image frame acquired after them is also synchronized.

[0129] based on Figure 2 The method described above can automatically determine the image frames whose acquisition time is synchronized between each image group.

[0130] If multiple matching single-camera trajectories exist between any two image groups, the number of time-synchronized image frames can be calculated for each matching single-camera trajectory. The average number of time-synchronized image frames determined by multiple matching single-camera trajectories is taken as the time-synchronized frame. For example, single-camera trajectory A in image group 1 matches single-camera trajectory B in image group 2, and based on single-camera trajectories A and B, the acquisition time of the first image frame in image group 1 is synchronized with the acquisition time of the third image frame in image group 2. Single-camera trajectory C in image group 1 matches single-camera trajectory D in image group 2, and based on single-camera trajectories C and D, the acquisition time of the first image frame in image group 1 is synchronized with the acquisition time of the fifth image frame in image group 2. Then, the average number of matching frames can be taken to finally obtain the acquisition time synchronization of the first image frame in image group 1 with the fourth image frame in image group 2.

[0131] The method provided in this embodiment of the invention, for each target, associates the fusion position of the target according to the acquisition time sequence of the image frame set corresponding to the target, and generates the fusion trajectory of the target. That is, the method provided in this embodiment of the invention can generate trajectories for multiple targets, and can adapt to the trajectory generation needs of multiple complex targets in scenarios such as urban traffic. Furthermore, the method provided in this embodiment of the invention provides more reliable spatiotemporal information for target trajectory fusion by synchronizing image frames acquired by multiple image acquisition devices.

[0132] In this embodiment of the invention, the number of image acquisition devices can be two, three or more, and no specific limitation is made here.

[0133] In one possible implementation, Figure 3 A flowchart of a location fusion method provided in an embodiment of the present invention is shown below. Figure 3 As shown, for application scenarios with three or more image acquisition devices, fusing the world positions of the same target in each image frame of the image frame set to obtain the fused position of the target may include:

[0134] S301, For any two image frames in the image frame set, construct a similarity matrix for each target in the two image frames based on the world position of each target in the image frame set.

[0135] In this embodiment of the invention, the step of constructing a similarity matrix for each target in any two image frames in the image frame set, based on the world position of each target in the image frame set, can specifically take the following steps B1-B2:

[0136] Step B1: Determine the velocity direction of each target based on its world position in the set of image frames.

[0137] Specifically, for each target, the position difference between the world position corresponding to the target's image position in the current image frame and the world position corresponding to the target's image position in the previous image frame can be calculated. That is, the displacement of the target during the acquisition time interval between the current image frame and the previous image frame. The ratio between the position difference and the acquisition time difference between the two image frames is determined as the velocity of the target. The sign of the velocity indicates the direction of the target's velocity.

[0138] Step B2: For any two image frames in the image frame set, construct a similarity matrix for each target in the two image frames based on the world position and velocity direction of each target in the two image frames.

[0139] Since a target may appear simultaneously in multiple image acquisition devices in urban traffic scenarios such as traffic intersections, in order to avoid missing the target's trajectory, this embodiment can first match the targets acquired by any two image acquisition devices to determine the common targets among the targets acquired by the two image acquisition devices.

[0140] Specifically, for any two image frames in the image frame set, such as image frame A and image frame B, if image frame A contains m targets and image frame B contains n targets, the cosine similarity between the velocity vectors of each target in image frame A and image frame B can be calculated based on the world position of each target in image frame A and image frame B and the velocity direction of each target in image frame A and image frame B. This cosine similarity can be used as the similarity to obtain an m×n similarity matrix.

[0141] In other words, based on the world position of each target in image frame A and image frame B, and the velocity direction of each target in image frame A and image frame B, the velocity vector of each target in image frame A and image frame B can be calculated. Then, the cosine similarity between the velocity vectors of m targets in image frame A and the velocity vectors of n targets in image frame B can be calculated. The similarity can be arranged into an m×n matrix to obtain an m×n similarity matrix.

[0142] In this embodiment of the invention, the method is not limited to that mentioned in the above steps. Based on the calculated cosine similarity between the velocity vectors of each target, the similarity between targets can also be determined by combining various features such as appearance, posture, or target type, according to the actual application scenario. The cosine similarity between the velocity vectors of each target, along with the similarity between appearance, posture, and target type, can be superimposed to obtain the corresponding m×n similarity matrix. Here, the target type can be an animal, vehicle, or signage, etc.; the appearance features of the target can be vehicle identification marks or animal appearance features, etc.; and the posture features of the target can be its world position, velocity, and velocity direction, etc.

[0143] In this embodiment of the invention, if the target is a vehicle, the vehicle identifier of the target can also be obtained, and the similarity between any two targets located in different image frames can be calculated based on the vehicle identifier.

[0144] S302, calculate the similarity matrix according to the Hungarian algorithm; if the similarity between two targets at corresponding positions in the similarity matrix and located in different image frames is greater than a preset similarity threshold, determine that the two targets are the same target.

[0145] In this embodiment of the invention, after obtaining the similarity matrix of each target in two image frames, the Hungarian algorithm can be used to solve the similarity matrix to obtain the one-to-one matching relationship between each target. For example, if image frame A1 in the image frame set includes 5 targets: a1, a2, a3, a4, and a5, and image frame B1 includes 3 targets: b1, b2, and b3, and the similarity matrix of each target in image frame A1 and image frame B1 is a 5×3 matrix, the Hungarian algorithm can be used to solve the similarity matrix to obtain the one-to-one matching relationship between each target: a1 matches b2, a2 ​​matches b1, and a4 matches b3.

[0146] In this embodiment of the invention, for each image group, a single-camera trajectory of the target can be generated based on multiple world positions of each target in the image group. Specifically, the single-camera trajectory of the target can be generated using the above steps A1-A2, which will not be elaborated here.

[0147] For any two image frames in the image frame set, after constructing a similarity matrix for each target in the two image frames based on the world position of each target in the image frame set, if there is a historical association between a pair of targets located in the two image frames, the similarity between the two targets can be directly set to 1. For example, if image frame A1 in the image frame set includes 5 targets: a1, a2, a3, a4, and a5, and image frame B1 includes 3 targets: b1, b2, and b3, then if, in the previous image frame of image frame A1, the target in the single-camera trajectory belonging to target a1 matches the target in the single-camera trajectory belonging to target b2 in the previous image frame of image frame B1, then the similarity between target a1 and target b2 can be directly set to 1.

[0148] After solving the similarity matrix, if the similarity between two targets located at corresponding positions in the similarity matrix but in different image frames is greater than a preset similarity threshold, the two targets are determined to be the same target. For example, if the similarity is calculated to match a1 and b2, a2 ​​and b1, and a4 and b3, that is, a1 and b2 are in corresponding positions in the similarity matrix, a2 and b1 are in corresponding positions in the similarity matrix, and a4 and b3 are in corresponding positions in the similarity matrix, then it can be further determined whether the similarity between a1 and b2 is greater than the preset similarity threshold, whether the similarity between a2 and b1 is greater than the preset similarity threshold, and whether the similarity between a4 and b3 is greater than the preset similarity threshold. If the similarity between a1 and b2 is greater than the preset similarity threshold, the similarity between a2 and b1 is not greater than the preset similarity threshold, and the similarity between a4 and b3 is greater than the preset similarity threshold, then a1 and b2 can be determined to be the same target, a4 and b3 can be determined to be the same target, and a2 and b1 are not the same target. The preset similarity threshold can be set according to the actual application scenario, and no specific limitation is made here.

[0149] S303, the world positions of the same targets in the image frame set are fused to obtain the fused position, and the fused position is used as the fused position of the fused target corresponding to the same target.

[0150] Specifically, the average world coordinates of the world positions of the same target in the image frame set can be calculated, and the average value can be used as the position represented by the fused position. Furthermore, the fused position can be used as the fused position of the fused target corresponding to the same target.

[0151] This step integrates the matching results of any two targets located in different image frames in the image frame set to obtain the fusion position of the fused targets corresponding to all identical targets in the image frame set.

[0152] In this embodiment, to ensure that each fusion target can only match at most one target in each image frame acquired by each image acquisition device (i.e., each target acquired by each image acquisition device can only appear in one fusion target), after fusing the world positions of the same targets in the image frame set to obtain the fused position, and using the fused position as the fusion position of the fusion target corresponding to the same target, the method further includes: for each fusion target, if there are at least two targets from the same image group among the multiple identical targets corresponding to the fusion target, removing the targets among the at least two targets whose distance from the fusion target is not the minimum distance; and updating the fusion position of the fusion target. Specifically, the distances between the at least two targets and the fusion target can be calculated, the target with the smallest distance from the fusion target can be retained, other targets can be removed, and then the average world coordinates of the remaining identical targets after removing the targets can be calculated as the updated fusion position of the fusion target.

[0153] After obtaining the fusion targets, the features of each fusion target can be updated. Specifically, the features of the fusion target can include: the world coordinates of the fusion location, appearance features, target type, and vehicle identification, etc. The world coordinates of the fusion target are the average of the world coordinates of the targets in each image acquisition device from which it originates; the target type and vehicle identification of the fusion target are the type and lane number of the target closest to it among the targets in each image acquisition device from which it originates; the appearance features of the fusion target are the average of the appearance features of the targets in each image acquisition device from which it originates.

[0154] The method provided in this invention, for each target, associates the fusion position of the target according to the acquisition time sequence of the image frame set corresponding to the target, generating the fusion trajectory of the target. That is, the method provided in this invention can generate trajectories for multiple targets, adapting to the trajectory generation needs of multiple complex targets in scenarios such as urban traffic. Furthermore, the method provided in this invention provides more reliable spatiotemporal information for target trajectory fusion by synchronizing image frames acquired by multiple image acquisition devices. Moreover, by combining world coordinates, vehicle identification, movement speed, appearance, and other features to achieve position association, the association results are more reliable and have a wider range of applications, not limited to specific types of targets.

[0155] In another possible implementation, before associating the fusion position of each target with the fusion trajectory based on the acquisition time sequence of the image frame set corresponding to that target, a single-camera trajectory for each image group can be generated based on multiple world positions of each target in that image group. Specifically, steps A1-A2 described above can be used to generate the single-camera trajectory of the target, which will not be elaborated here. Furthermore, a single-camera trajectory identifier can be generated for each single-camera trajectory.

[0156] Figure 4 A flowchart of the fusion trajectory of the generated target provided in an embodiment of the present invention, such as... Figure 4 As shown, the step of generating a fusion trajectory for each target by associating its fusion position with the acquisition time sequence of the image frame set corresponding to that target, specifically may include:

[0157] S401, for each fusion target, if the current fusion position of the fusion target and the fusion position in the existing trajectory both originate from the same single camera trajectory, determine that the current fusion position of the fusion target is associated with the existing trajectory.

[0158] The existing trajectory is the trajectory formed by associating the fusion positions of all targets whose acquisition time precedes the current fusion position of the fusion target. In other words, the existing trajectory is the trajectory formed by associating the fusion positions of all targets whose acquisition time precedes the current fusion position of the fusion target, where each target preceding the current fusion position is a target whose acquisition time precedes the acquisition time of the corresponding image frame of the fusion target.

[0159] If the current fusion position of the fusion target is obtained by fusing the earliest group of image frames in each group of synchronized image frames, then the fusion target can be directly regarded as an existing trajectory, and a unified fusion trajectory identifier can be assigned to the existing trajectory.

[0160] Furthermore, according to the acquisition time sequence, for each fusion target whose acquisition time is after the first fusion target, if the current fusion position of the fusion target and the source of the fusion position in the existing trajectory both include the position in the same single camera trajectory, it is determined that the current fusion position of the fusion target is associated with the existing trajectory. For example, if the sources of the fusion position in the existing trajectory A include: the position in the single-camera trajectory 1 of image acquisition device 1, the position in the single-camera trajectory 2 of image acquisition device 2, the position in the single-camera trajectory 3 of image acquisition device 3, and the position in the single-camera trajectory 4 of image acquisition device 4; and the sources of the current fusion position of the fusion target a include: the position in the single-camera trajectory 1 of image acquisition device 1, the position in the single-camera trajectory 5 of image acquisition device 2, the position in the single-camera trajectory 6 of image acquisition device 3, and the position in the single-camera trajectory 7 of image acquisition device 4; since the sources of both the existing trajectory A and the current fusion position of the fusion target a include the position in the single-camera trajectory 1 of image acquisition device 1, it can be directly determined that the current fusion position of the fusion target is associated with the existing trajectory. Furthermore, the existing trajectory A can be updated based on the current fusion position of the fusion target a, that is, the current fusion position of the fusion target a is taken as a new trajectory point of the existing trajectory A, and a new fusion trajectory identifier is assigned to the updated existing trajectory A.

[0161] Specifically, in this embodiment, the source of the current fusion position of the fusion target and the fusion position in the existing trajectory can be determined based on the single camera trajectory identifier and the fusion trajectory identifier of the single camera trajectory.

[0162] S402, if the current fusion position of the fusion target and the source of the fusion position in any existing trajectory do not include the position in the same single camera trajectory, and the current fusion position of the fusion target is in a preset motion state, it is determined that the current fusion position of the fusion target is not related to the existing trajectory.

[0163] The preset motion state is that the motion state of the fusion target is in the state of leaving the intersection.

[0164] In this embodiment, if the single-camera trajectory identifier of the existing trajectory source is lost and the existing trajectory is in motion leaving the intersection, then the existing trajectory is prohibited from being associated with any fusion target; if the single-camera trajectory identifier of the existing trajectory source is canceled and the existing trajectory is in motion leaving the intersection, then the existing trajectory is directly canceled, that is, the existing trajectory is deleted.

[0165] S403, if the current fusion position of the fusion target and the source of the fusion position in any existing trajectory do not include the position in the same single camera trajectory, and the current fusion position of the fusion target is not in a preset motion state, construct the correlation matrix between each fusion target and the existing trajectory in the image frame corresponding to the fusion target.

[0166] If the current fusion location of the target does not originate from any existing trajectory within the same single-camera trajectory, then the flag indicating whether to use the same single-camera trajectory can be set to 0. A flag of 0 indicates that it is not necessary to determine the existing trajectory associated with the current fusion location of the target based on the single-camera trajectory; a flag of 1 indicates that it is necessary to determine the existing trajectory associated with the current fusion location of the target based on whether its origin is the same as the origin of the single-camera trajectory.

[0167] Specifically, the similarity between the world coordinate distances of the fusion target and existing trajectories can be calculated to construct a correlation matrix between each fusion target and existing trajectories in the image frame corresponding to the fusion target. Using this correlation matrix, local and global correlation processing is performed between the fusion target and existing trajectories. Local correlation processing involves using the Hungarian algorithm to calculate the correlation result between the fusion target and existing trajectories based on the correlation matrix. Global correlation processing involves using the Hungarian algorithm to calculate the correlation result between the fusion target and existing trajectories whose source single-camera trajectory identifiers are missing, based on the correlation matrix. Based on the correlation results of these local and global correlations, a new fusion trajectory identifier can be assigned to the matched fusion target, and its trajectory features can be updated. For unmatched fusion targets, if the unmatched fusion target meets the trajectory creation condition, a new trajectory is created for that fusion target as an existing trajectory; otherwise, the existing trajectory is set to a missing state. Specifically, if the unmatched fusion target is obtained by fusing the earliest acquired set of image frames from each group of synchronized image frames, then the fusion target is determined to meet the trajectory creation condition.

[0168] S404, calculate the correlation matrix according to the Hungarian algorithm. If the correlation between the fusion target and the existing trajectory at the corresponding position in the correlation matrix is ​​greater than the preset correlation threshold, determine that the current fusion position of the fusion target is associated with the existing trajectory.

[0169] The preset correlation threshold can be set according to the actual application situation, and no specific limit is made here.

[0170] In this embodiment of the invention, the correlation matrix can be solved using the Hungarian algorithm to obtain the one-to-one matching relationship between each fusion target and the existing trajectory in the image frame corresponding to the fusion target. For example, if there are three targets in the image frame corresponding to the fusion target: c1, c2, and c3, and the existing trajectories include: existing trajectory A and existing trajectory B, and the correlation matrix is ​​a 3×2 matrix, the correlation matrix can be solved using the Hungarian algorithm to obtain the one-to-one matching relationship between each fusion target and the existing trajectory in the image frame corresponding to the fusion target: c1 matches existing trajectory B, and c2 matches existing trajectory A.

[0171] If the correlation between c1 and the existing trajectory B is greater than a preset correlation threshold, it can be determined that the current fusion position of the fusion target c1 is associated with the existing trajectory B. If the correlation between c2 and the existing trajectory A is not greater than a preset correlation threshold, it can be determined that the current fusion position of the fusion target c2 is not associated with the existing trajectory A.

[0172] After determining that the fusion target is associated with the existing trajectory, the existing trajectory can be updated based on the current fusion position of the fusion target. That is, the current fusion position of the fusion target is used as a new trajectory point of the existing trajectory, and a new fusion trajectory identifier is assigned to the updated existing trajectory.

[0173] S405, if all the fusion positions corresponding to the fusion target are associated, generate the fusion trajectory of the fusion target based on all the associated fusion positions of the fusion target.

[0174] The method provided in this invention, for each target, associates the fusion position of the target according to the acquisition time sequence of the image frame set corresponding to the target, generating the fusion trajectory of the target. That is, the method provided in this invention can generate trajectories for multiple targets, adapting to the trajectory generation needs of multiple complex targets in scenarios such as urban traffic. Furthermore, it can generate the fusion trajectory of targets in real time, making it suitable for scenarios with high real-time requirements, such as smart intersections. Moreover, the method provided in this invention provides more reliable spatiotemporal information for target trajectory fusion by synchronizing image frames acquired by multiple image acquisition devices. Furthermore, it combines world coordinates, lane number, movement speed, appearance, and other features to achieve position association, making the association results more reliable and widely applicable, not limited to specific types of targets. Furthermore, the method provided in this invention utilizes single-camera trajectories for target trajectory fusion, making the fusion trajectory more reliable and effectively reducing the phenomenon of fusion trajectory identifiers being carried back. In addition, the method provided in this invention does not require manually obtaining the relationship matrix of the areas acquired by the image acquisition devices, resulting in lower usage costs and better universality.

[0175] The method provided in this embodiment of the invention can be applied to real-time scenarios to generate the fused trajectory of the target in real time.

[0176] In one possible implementation, for scenarios that do not require full real-time performance, trajectory delay correction can be used to correct the fused trajectory, thereby reducing trajectory jumps. Figure 5 This is a schematic flowchart of a trajectory correction method provided in an embodiment of the present invention, as shown below. Figure 5 As shown: For applications using time-delay correction, trajectory delay caching can be performed, followed by trajectory smoothing, and finally, the smoothed trajectory can be output. Specifically, multiple frames of fused trajectory over a fixed time period can be cached, where n can be set to 10 frames, 20 frames, etc. Then, the average world coordinates of the fused trajectory of the previous i frames are calculated successively, where i = 0, 1, 2...n. Finally, the current smoothed fused trajectory is output frame by frame, realizing the correction processing of the fused trajectory and making the output fused trajectory smoother and more fluid. For applications without time-delay correction, the fused trajectory can be output directly.

[0177] The method provided in this invention, for each target, associates the fusion position of the target according to the acquisition time sequence of the image frame set corresponding to the target, generating the fusion trajectory of the target. That is, the method provided in this invention can generate trajectories for multiple targets, adapting to the trajectory generation needs of multiple complex targets in scenarios such as urban traffic. Furthermore, it can generate the fusion trajectory of targets in real time, making it suitable for scenarios with high real-time requirements, such as smart intersections. Moreover, the method provided in this invention provides more reliable spatiotemporal information for target trajectory fusion by synchronizing image frames acquired by multiple image acquisition devices. Furthermore, by combining world coordinates, lane number, movement speed, appearance, and other features to achieve position association, the association results are more reliable and have a wider range of applications, not limited to specific types of targets.

[0178] For application scenarios that generate trajectories for a specific target, the method provided in this embodiment of the invention generates a fused trajectory for a specific target. By synchronizing image frames acquired by multiple image acquisition devices, it provides more reliable spatiotemporal information for trajectory fusion of the target. Furthermore, by combining features such as world coordinates, lane number, movement speed, and appearance to achieve position association, the association results are more reliable and have a wider range of applications. Therefore, compared with current related technologies, the fused trajectory generated for a specific target using the method provided in this embodiment of the invention is more accurate.

[0179] Corresponding to the above-described method for generating target trajectories, embodiments of the present invention also provide a device for generating target trajectories. The device for generating target trajectories provided in these embodiments is described below. Figure 6This is a schematic diagram of a target trajectory generation device provided in an embodiment of the present invention, such as... Figure 6 As shown, the device includes:

[0180] The location acquisition module 601 is used to acquire the image location of each target in each image frame of each image group acquired by each image acquisition device.

[0181] The position conversion module 602 is used to convert the image position into a position in the world coordinate system according to a preset conversion relationship, so as to obtain the world position corresponding to the image position;

[0182] The location fusion module 603 is used to fuse the world positions of the same target in each image frame in each image frame in each image frame set to obtain the fused position of the target; wherein, the image frame set is an image set composed of each image frame acquired at the same time.

[0183] The trajectory generation module 604 is used to generate a fused trajectory for each target by associating the fusion position of the target with the acquisition time sequence of the image frame set corresponding to the target.

[0184] Using the apparatus provided in this embodiment of the invention, for each image group acquired by an image acquisition device, the image positions of each target in each image frame of the image group are obtained; according to a preset transformation relationship, the image positions are converted into positions in the world coordinate system to obtain the world positions corresponding to the image positions; for each set of image frames, the world positions of each identical target in each image frame of the set are fused to obtain the fused position of the target; for each target, according to the acquisition time sequence of the image frame set corresponding to the target, the fused positions of the target are associated to generate the fused trajectory of the target. That is, the apparatus provided in this embodiment of the invention can generate trajectories of multiple targets, and can adapt to the trajectory generation needs of multiple complex targets in scenarios such as urban traffic.

[0185] Optionally, the number of image acquisition devices is at least 3;

[0186] See Figure 7 The location fusion module 603 includes:

[0187] The similarity matrix determination submodule 701 is used to construct a similarity matrix for each target in any two image frames in the image frame set based on the world position of each target in the image frame set.

[0188] The target determination submodule 702 is used to solve the similarity matrix according to the Hungarian algorithm. If the similarity between two targets at corresponding positions in the similarity matrix and located in different image frames is greater than a preset similarity threshold, the two targets are determined to be the same target.

[0189] The location fusion submodule 703 is used to fuse the world positions of the same target in the image frame set to obtain the fused position, and use the fused position as the fused position of the fused target corresponding to the same target.

[0190] Optionally, the similarity matrix determination submodule 701 is specifically used to determine the velocity direction of each target based on the world position of each target in the image frame set; and for any two image frames in the image frame set, to construct a similarity matrix of each target in the two image frames based on the world position and velocity direction of each target in the two image frames.

[0191] Optional, see Figure 8 The device further includes:

[0192] The fusion position update module 801 is used to, for each fusion target, if there are at least two targets from the same image group among the multiple identical targets corresponding to the fusion target, remove the targets among the at least two targets whose distance from the fusion target is not the minimum distance; and update the fusion position of the fusion target.

[0193] Optional, see Figure 8 The device further includes:

[0194] The single-camera trajectory generation module 802 is used to generate a single-camera trajectory for each target in each image group based on multiple world positions of each target in the image group.

[0195] The trajectory generation module 604 is specifically configured to, for each fusion target, determine that the current fusion position of the fusion target is associated with the existing trajectory if the source of the current fusion position of the fusion target and the fusion position in the existing trajectory both include positions in the same single-camera trajectory; wherein, the existing trajectory is the trajectory formed by associating the fusion positions of various targets acquired before the current fusion position of the fusion target; if the source of the current fusion position of the fusion target and any fusion position in the existing trajectory does not include positions in the same single-camera trajectory, and the current fusion position of the fusion target is in a preset motion state, determine that the current fusion position of the fusion target is not associated with the existing trajectory; if the fusion target's current fusion position is not associated with the existing trajectory, the module determines that the current fusion position of the fusion target is not associated with the existing trajectory. The current fusion position of the target does not originate from any position in the same single-camera trajectory as any existing fusion position, and the current fusion position of the target is not in a preset motion state. A correlation matrix is ​​constructed between each fusion target and existing trajectory in the image frame corresponding to the target. The correlation matrix is ​​solved using the Hungarian algorithm. If the correlation between the target and the existing trajectory at the corresponding position in the correlation matrix is ​​greater than a preset correlation threshold, the current fusion position of the target is determined to be associated with the existing trajectory. If all fusion positions corresponding to the target are associated, the fusion trajectory of the target is generated based on all associated fusion positions.

[0196] Optional, see Figure 8 The device further includes:

[0197] The trajectory update module 803 is used to update the existing trajectory based on the current fusion position of the fusion target.

[0198] Optional, see Figure 8 The device further includes:

[0199] The synchronization time determination module 804 is used to generate a single-camera trajectory for each target in each image group based on multiple world positions of each target in the image group; match the single-camera trajectories in any two image groups; select any set of matched single-camera trajectories and calculate the distance between the start and end positions of the first single-camera trajectory in the set and each position of the second single-camera trajectory; and calculate the distance between the start and end positions of the second single-camera trajectory in the set and each position of the first single-camera trajectory; if there is a position in the second single-camera trajectory whose distance to the start or end position of the first single-camera trajectory is less than a preset distance threshold, and there is a position in the first single-camera trajectory whose distance to the start or end position of the second single-camera trajectory is less than a preset distance threshold; At a preset distance threshold, retain the matching single-camera trajectories of the group; otherwise, delete the single-camera trajectories of the group. For each group of retained matching single-camera trajectories, select the shortest distance from the distance between the start and end positions of the first single-camera trajectory in the group to each position of the second single-camera trajectory, and from the distance between the start and end positions of the second single-camera trajectory in the group to each position of the first single-camera trajectory. Determine the image frames corresponding to the two positions of the shortest distance as the primary synchronization image frames. For any two image groups, calculate the average acquisition time of each pair of primary synchronization image frames in the two image groups, and take the image frames in the two image groups whose acquisition time is the average time as the final synchronization image frames of the two image groups.

[0200] Using the apparatus provided in this embodiment of the invention, for each target, the fusion position of the target is associated according to the acquisition time sequence of the image frame set corresponding to the target, generating the fusion trajectory of the target. That is, the apparatus provided in this embodiment of the invention can generate trajectories for multiple targets, adapting to the trajectory generation needs of multiple complex targets in scenarios such as urban traffic. Furthermore, the fusion trajectory of the target can be generated in real time, meaning the apparatus provided in this embodiment of the invention is suitable for scenarios with high real-time requirements, such as smart intersections. Moreover, the apparatus provided in this embodiment of the invention provides more reliable spatiotemporal information for target trajectory fusion by synchronizing image frames acquired by multiple image acquisition devices. Furthermore, by combining world coordinates, lane number, movement speed, appearance, and other features to achieve position association, the association results are more reliable and have a wider range of applications, not limited to specific types of targets. Furthermore, the apparatus provided in this embodiment of the invention uses single-camera trajectory for target trajectory fusion, making the fused trajectory more reliable and effectively reducing the phenomenon of fused trajectory identifiers being carried back. In addition, the apparatus provided in this embodiment of the invention does not require manual acquisition of the relationship matrix of the areas acquired by the image acquisition devices, resulting in lower usage costs and better universality.

[0201] This invention also provides an electronic device, such as... Figure 9As shown, it includes a processor 901, a communication interface 902, a memory 903, and a communication bus 904. The processor 901, communication interface 902, and memory 903 communicate with each other via the communication bus 904.

[0202] Memory 903 is used to store computer programs;

[0203] When the processor 901 executes the program stored in the memory 903, it implements the steps of the target trajectory generation method described in any of the above embodiments.

[0204] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0205] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0206] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0207] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0208] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the method for generating any of the above-described target trajectories.

[0209] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the target trajectory generation methods in the above embodiments.

[0210] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0211] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0212] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0213] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method of generating a target trajectory, characterized by, The method comprises the following steps: For each image group collected by each image acquisition device, the image positions of each target in each image frame in the image group are obtained, wherein one image group comprises a plurality of image frames collected by one image acquisition device; According to a preset conversion relationship, the image positions are converted into positions in a world coordinate system to obtain world positions corresponding to the image positions, wherein the conversion relationship corresponding to each image acquisition device is used to represent the mapping relationship between the image position coordinates of the target collected by the image acquisition device and the world position coordinates; For each image frame set, the world positions of each same target in each image frame in the image frame set are fused to obtain the fusion position of the target; wherein the image frame set is an image set composed of image frames collected synchronously in time; For each target, the fusion positions of the target are associated according to the collection time sequence of the image frame set corresponding to the target to generate the fusion track of the target; The method further comprises the following steps after the step of fusing the world positions of the same target in the image frame set to obtain the fused position and taking the fused position as the fusion position of the fusion target corresponding to the same target: For each fusion target, if there are at least two targets from the same image group in the plurality of same targets corresponding to the fusion target, the target with a distance from the fusion target that is not the minimum distance is removed; and the fusion position of the fusion target is updated. The number of image acquisition devices is at least 3; The step of fusing the world positions of each same target in each image frame in the image frame set to obtain the fusion position of the target further comprises the following steps:

2. The method of claim 1, wherein, For any two image frames in the image frame set, a similarity matrix of each target in the two image frames is constructed according to the world positions of each target in the image frame set; According to the Hungarian algorithm, the similarity matrix is solved; if the similarity between two targets at the corresponding positions in the similarity matrix and located in different image frames is greater than a preset similarity threshold, the two targets are determined to be the same target. The step of constructing the similarity matrix of each target in the two image frames according to the world positions of each target in the image frame set comprises the following steps: Based on the world positions of each target in the image frame set, the speed direction of the target is determined; 3. The method of claim 2, wherein, For any two image frames in the image frame set, the similarity matrix of each target in the two image frames is constructed based on the world positions and speed directions of each target in the two image frames. ​ ​ 4. The method of claim 1, wherein, Before the step of associating the fusion positions of the target according to the acquisition time sequence of the image frame set corresponding to the target, and generating the fusion track of the target, the method further comprises: generating a single-camera track of each target in each image group based on the multiple world positions of the target in the image group; Before the step of associating the fusion positions of the target according to the acquisition time sequence of the image frame set corresponding to the target, and generating the fusion track of the target, the method further comprises: For each fusion target, if the current fusion position of the fusion target and the sources of the fusion positions in the existing track all include positions in the same single-camera track, it is determined that the current fusion position of the fusion target is associated with the existing track; wherein the existing track is a track formed by associating the fusion positions of the targets acquired before the current fusion position of the fusion target; If the current fusion position of the fusion target and the sources of the fusion positions in any one of the existing tracks do not include positions in the same single-camera track, and the current fusion position of the fusion target is in a preset motion state, it is determined that the current fusion position of the fusion target is not associated with the existing track; If the current fusion position of the fusion target and the sources of the fusion positions in any one of the existing tracks do not include positions in the same single-camera track, and the current fusion position of the fusion target is not in a preset motion state, an association degree matrix between the fusion target and the existing track in the image frame corresponding to the fusion target is constructed; According to the Hungarian algorithm, if the association degree between the fusion target and the existing track at the corresponding position in the association degree matrix is greater than a preset association degree threshold, it is determined that the current fusion position of the fusion target is associated with the existing track; If the fusion positions corresponding to the fusion target are all associated, a fusion track of the fusion target is generated based on all the associated fusion positions of the fusion target.

5. The method of claim 1, wherein, Before the step of fusing the world positions of each same target in each image frame set in the image frame set to obtain the fusion position of the target, the method further comprises: generating a single-camera track of each target in each image group based on the multiple world positions of the target in the image group; matching the single-camera tracks in any two image groups; Optionally, a group of matched single-camera tracks is selected, and the distance between the start position and the end position of the first single-camera track in the group and each position of the second single-camera track is calculated respectively; and the distance between the start position and the end position of the second single-camera track in the group and each position of the first single-camera track is calculated respectively; If there is a position in the second single-camera track that is less than a preset distance threshold from the start position or the end position of the first single-camera track, and there is a position in the first single-camera track that is less than a preset distance threshold from the start position or the end position of the second single-camera track, the group of matched single-camera tracks is retained; otherwise, the single-camera tracks in the group are deleted; For each set of matched single-camera tracks reserved, the shortest distance is selected from the distance between the start position and the end position of the first single-camera track and each position of the second single-camera track in the set, and the distance between the start position and the end position of the second single-camera track and each position of the first single-camera track in the set, and the image frames in which the two positions corresponding to the shortest distance are located are determined as the primary synchronization image frames; For any two image groups, the average time of the acquisition time of each pair of primary synchronization image frames in the two image groups is calculated, and the image frame with the average time of the acquisition time in the two image groups is taken as the final synchronization image frame of the two image groups.

6. An apparatus for generating a target trajectory, characterized by comprising: Comprise: The position acquisition module is used for acquiring the image position of each target in each image frame in each image group collected by each image acquisition device, wherein an image group comprises a plurality of image frames collected by one image acquisition device; The position conversion module is used for converting the image position into a position in a world coordinate system according to a preset conversion relationship to obtain a world position corresponding to the image position, wherein the conversion relationship corresponding to each image acquisition device is used to represent the mapping relationship between the image position coordinates and the world position coordinates of the target collected by the image acquisition device; The position fusion module is used for fusing the world positions of each same target in each image frame in each image frame set to obtain the fusion position of the target; wherein the image frame set is an image set formed by the image frames with synchronous acquisition time; The track generation module is used for associating the fusion positions of the target according to the acquisition time sequence of the image frame set corresponding to the target to generate the fusion track of the target for each target; The position fusion module comprises: The position fusion submodule is used for fusing the world positions of the same target in the image frame set to obtain the fused position, and taking the fused position as the fusion position of the fusion target corresponding to the same target; The device further comprises: The fusion position update module is used for, for each fusion target, if there are at least two targets from the same image group in the plurality of same targets corresponding to the fusion target, eliminating the target with a non-minimum distance from the fusion target among the at least two targets; and updating the fusion position of the fusion target.

7. The apparatus of claim 6, wherein, The number of image acquisition devices is at least 3; The position fusion module further comprises: The similarity matrix determination submodule is used for constructing the similarity matrix of each target in any two image frames in the image frame set according to the world positions of each target in the image frame set; The target determination submodule is used for solving the similarity matrix according to the Hungarian algorithm, and if the similarity between the two targets at the corresponding positions in the similarity matrix and located in different image frames is greater than a preset similarity threshold, the two targets are determined to be the same target.

8. An electronic device, comprising: The device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus. a memory for storing a computer program; a processor for implementing the method steps of any one of claims 1-5 when executing the program stored in the memory.

9. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and the computer program is executed by the processor to implement the method steps of any one of claims 1-5.

Images