Target tracking method and system based on multi-video source fusion

Abstract

The present invention discloses a target tracking method and system for multi-video source fusion, including: S1: time synchronization of cameras, determining camera distortion based on reference targets; S2: acquiring camera image sequences, identifying targets using pre-trained convolutional neural network models, and recording the coordinate sequence of the target in the image; S3: converting the coordinate sequence of the target in the image into the position sequence of the target in a Cartesian coordinate system in combination with camera distortion; S4: defining a node set and a node transfer set using the position sequence of the target in a Cartesian coordinate system, and establishing a directed graph model; S5: constructing a node queue based on the directed graph model, and interpolating nodes that are adjacent in position but not adjacent in time to obtain a complete trajectory of the target. The present invention identifies targets through convolutional neural networks, accurately locates targets through correction processing; based on multi-source sequences, the accuracy and continuity of target tracking are improved through directed graph processing and node interpolation.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology, and in particular to a target tracking method and system for multi-video source fusion. Background Art

[0002] Target tracking refers to paying attention to the moving target of interest in the video and tracking the target in a continuous time to obtain the target's motion trajectory. At present, with the application of machine learning methods, target tracking has achieved good results, but since most target tracking methods rely only on a single video source, the information content contained is not enough to support the purpose of long-term and accurate tracking of specific targets. During the target movement process, there will be problems such as target loss and inability to accurately locate the target due to factors such as dense targets, line of sight occlusion, clutter and noise.

[0003] Currently, the limited target tracking methods based on multiple video sources are mainly based on the perspective of trajectory fusion. First, the target trajectory is obtained based on a single video source, and then based on the characteristics of the target trajectory, clustering methods, weighted methods, and Bayesian methods are used to achieve multi-source trajectory fusion of the same target. However, this method has high requirements on the accuracy and continuity of single video source data; if the error of a single video source data is large, the target trajectory obtained by using this method for target tracking will have large deviations and more interruptions. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a target tracking method and system with multi-video source fusion, the purpose of which is to make full use of the multi-source data sequence of the target on the basis of accurately locating the target, and realize accurate and continuous tracking of the target based on a directed graph model. In order to achieve the above purpose, the technical solution proposed by the present invention is:

[0005] A target tracking method for multi-video source fusion includes the following steps:

[0006] S1: Time synchronization of the camera and determination of camera distortion based on the reference target;

[0007] S2: Collect camera image sequences, use the pre-trained convolutional neural network model to identify the target, and record the coordinate sequence of the target in the image;

[0008] S3: Combined with camera distortion, the coordinate sequence of the target in the image is converted into the position sequence of the target in the Cartesian coordinate system;

[0009] S4: Using the position sequence of the target in the Cartesian coordinate system, define the node set and the node transfer set, and establish a directed graph model;

[0010] S5: Based on the directed graph model, a node queue is constructed, and nodes that are adjacent in position but not adjacent in time are interpolated to obtain the complete trajectory of the target.

[0011] As a further improvement method of the present invention:

[0012] Optionally, the step S1 of performing time synchronization on the camera and determining the camera distortion based on the reference target includes:

[0013] S101: Define the camera model:

[0014] ;

[0015] in, Indicates the camera number. Indicates that the reference target is in the camera The position in , Indicates that the reference target is in the camera The coordinate values ​​in represents the position of the reference target in the Cartesian coordinate system, Representing the real world and the camera The rotation relationship matrix between Represents the real world and the camera The translation relationship matrix between them;

[0016] S102: Determine the distortion using the coordinate position of the reference target in the two-dimensional image:

[0017] ;

[0018] ;

[0019] in, Indicates the camera The distortion of the image relative to the real world, Indicates that the reference target is in the camera The coordinate values ​​in the generated two-dimensional image are, Indicates the camera The ratio of focal length to pixel size in horizontal and vertical directions, Indicates the camera The main point.

[0020] Optionally, in step S2, collecting a camera image sequence, using a pre-trained convolutional neural network model to identify a target, and recording a coordinate sequence of the target in the image includes:

[0021] S201: Collect camera image sequence ,in, represents the image sequence acquisition time;

[0022] S202: Use the pre-trained convolutional neural network model to train the image sequence Perform target recognition. If it is determined to be a target to be tracked, the coordinate sequence of the target in the image is recorded. .

[0023] Optionally, in step S3, combining camera distortion to convert the coordinate sequence of the target in the image into a position sequence of the target in a Cartesian coordinate system includes:

[0024] S301: Using a camera The image is distorted relative to the real world, and the target is calculated in the camera The position sequence in:

[0025] ;

[0026] in, Indicates that the target is in the camera The position sequence in ;

[0027] S302: Calculate the position sequence of the target in the Cartesian coordinate system:

[0028] ;

[0029] in, represents the position sequence of the target in the Cartesian coordinate system, Represents the rotation relationship matrix Take the inverse matrix.

[0030] Optionally, in step S4, the position sequence of the target in the Cartesian coordinate system is used to define a node set and a node transfer set, and a directed graph model is established, including:

[0031] S401: Using the position sequence of the target in the Cartesian coordinate system, define a node set:

[0032] ;

[0033] in, Represents a collection of nodes. Is a collection of nodes The elements in Camera included The sequence of locations where the identified target appears and image acquisition time ;

[0034] S402: Constructing a position adjacent discriminant:

[0035] ;

[0036] in, Indicates another camera number. It represents the distance that the target moves in the Cartesian coordinate system at adjacent times. is the distance threshold, Indicates that the positions of the targets in the Cartesian coordinate system are adjacent at adjacent times. It means that the positions of the targets in the Cartesian coordinate system at adjacent times are not adjacent;

[0037] S403: Define node transfer set:

[0038] ;

[0039] in, represents the node transfer set, is the node transfer set The elements in Contains all the Position sequence pair ;

[0040] S404: Establishing a directed graph model ,in The elements in are connected by internal adjacent edges, The elements in are connected by external adjacent edges.

[0041] Optionally, in step S5, a node queue is constructed based on a directed graph model, and nodes that are adjacent in position but not adjacent in time are interpolated to obtain a complete target trajectory, including:

[0042] S501: Determine the set The longest element in ,Will All position sequences in Put in queue , create a collection Used to record target trajectory;

[0043] S502: From the queue Take out the starting node and record it as ;

[0044] S503: In order from the queue Take the next node , until the queue If it is empty, jump to step S506;

[0045] S504: Comparison and ,in express The recorded image acquisition time, express The recorded image acquisition time;

[0046] S505: If satisfied ,express and Time is adjacent, Add to collection ;

[0047] make , jump to step S503;

[0048] If not satisfied ,express and The times are not adjacent;

[0049] Along the directed graph The external neighboring edges of Time adjacent nodes ;

[0050] Will Add to collection ,make , jump to step S504;

[0051] S506: Collection The position sequence of the target recorded in the middle node sequence , which is the complete trajectory of the target.

[0052] The present invention also provides a target tracking system for multi-video source fusion, comprising:

[0053] Camera synchronization and distortion measurement module: synchronize camera time and determine the distortion of camera images relative to the real world;

[0054] Target recognition module: collects image sequences, identifies targets, and records the coordinate sequence of targets in the images;

[0055] Target position transformation module: Combined with camera distortion, the target coordinate sequence in the image is converted into the target position sequence in the Cartesian coordinate system;

[0056] Directed graph construction module: define node sets, define node transfer sets, and establish directed graph models;

[0057] Target trajectory generation module: create a queue, compare image acquisition time, determine time-adjacent nodes, traverse external adjacent edges, perform node interpolation, and output the complete trajectory of the target.

[0058] Beneficial effects:

[0059] The present invention processes and integrates data from multiple video sources, and can effectively improve the tracking performance of targets in complex environments. The method of the present invention uses a convolutional neural network to quickly and accurately identify targets, accurately locates targets through coordinate transformation based on camera distortion, and makes full use of multi-video source data based on a directed graph model defined by a target position sequence to improve the accuracy of target tracking, and improves the continuity and integrity of the target trajectory through node interpolation.

[0060] The method of the present invention performs time synchronization on cameras, which is beneficial to the fusion and further processing of multi-video source data; the method of calculating camera distortion based on a reference target is beneficial to quickly determining camera distortion and accurately processing subsequent data; the way in which a convolutional neural network is trained layer by layer is beneficial to quickly and effectively determining model parameters and improving model recognition effects.

[0061] The method of the present invention constructs a node set based on a target position sequence, which is conducive to clear and effective data management; and constructs a node transfer set based on adjacent position judgment, which is conducive to utilizing the topological relationship of multi-video source data.

[0062] The method of the present invention selects the element with the largest length Processing can reduce the subsequent processing steps; queues and collections can be created to effectively manage data, perform adjacent time judgment and neighbor node search, which is conducive to obtaining complete target trajectory data. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The figure is a flow chart of a target tracking method of multi-video source fusion according to the present invention. DETAILED DESCRIPTION

[0064] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0065] Embodiment 1: A target tracking method for fusion of multiple video sources, such as Figure 1 As shown, the following steps are included:

[0066] S1: Time synchronize the cameras and determine the camera distortion based on the reference target.

[0067] Multiple cameras are connected to a unified network clock server, and the NTP protocol is used to achieve automatic synchronization and calibration of video time;

[0068] S101: Define the camera model:

[0069] ;

[0070] in, Indicates the camera number. Indicates that the reference target is in the camera The position in , Indicates that the reference target is in the camera The coordinate values ​​in represents the position of the reference target in the Cartesian coordinate system, Represents the real world and the camera The rotation relationship matrix between Represents the real world and the camera The translation relationship matrix between them;

[0071] S102: Determine the distortion using the coordinate position of the reference target in the two-dimensional image:

[0072] ;

[0073] ;

[0074] in, Indicates the camera The distortion of the image relative to the real world, Indicates that the reference target is in the camera The coordinate values ​​in the generated two-dimensional image are, Indicates the camera The ratio of focal length to pixel size in horizontal and vertical directions, Indicates the camera The main point.

[0075] S2: Collect camera image sequences, use the pre-trained convolutional neural network model to identify the target, and record the coordinate sequence of the target in the image.

[0076] S201: Collect camera image sequence ,in, Indicates the camera number. represents the image sequence acquisition time;

[0077] S202: Use the pre-trained convolutional neural network model to train the image sequence Perform target recognition. If it is determined to be a target to be tracked, the coordinate sequence of the target in the image is recorded. .

[0078] In this embodiment, a training data set is constructed using a set of target images to be tracked, and the training data set is used to train the first layer of hidden nodes of the convolutional neural network, including the convolutional layer, the convolutional kernel, and the channel parameters, until the training effect reaches the optimal; the first layer of the convolutional neural network is fixed, and the second layer of hidden nodes of the convolutional neural network is trained using the training data set, including the convolutional layer, the convolutional kernel, and the channel parameters, until the training effect reaches the optimal; the first and second layers of the convolutional neural network are fixed, and the training data set is used to train the third layer of hidden nodes of the convolutional neural network, including the convolutional layer, the convolutional kernel, and the channel parameters, until the training effect reaches the optimal, thereby obtaining a pre-trained convolutional neural network model.

[0079] S3: Combined with camera distortion, the coordinate sequence of the target in the image is converted into the position sequence of the target in the Cartesian coordinate system.

[0080] S301: Using a camera The image is distorted relative to the real world, and the target is calculated in the camera The position sequence in:

[0081] ;

[0082] in, Indicates that the target is in the camera The position sequence in ;

[0083] S302: Calculate the position sequence of the target in the Cartesian coordinate system:

[0084] ;

[0085] in, represents the position sequence of the target in the Cartesian coordinate system, Represents the rotation relationship matrix Take the inverse matrix.

[0086] S4: Using the position sequence of the target in the Cartesian coordinate system, define the node set and the node transfer set, and establish a directed graph model.

[0087] S401: Using the position sequence of the target in the Cartesian coordinate system, define a node set:

[0088] ;

[0089] in, Represents a collection of nodes. Is a collection of nodes The elements in Camera included The sequence of locations where the identified target appears and image acquisition time ;

[0090] S402: Constructing a position adjacent discriminant:

[0091] ;

[0092] in, Indicates another camera number. It represents the distance that the target moves in the Cartesian coordinate system at adjacent times. is the distance threshold, Indicates that the positions of the targets in the Cartesian coordinate system are adjacent at adjacent times. It means that the positions of the targets in the Cartesian coordinate system at adjacent times are not adjacent;

[0093] S403: Define node transfer set:

[0094] ;

[0095] in, represents the node transfer set, is the node transfer set The elements in Contains all the Position sequence pair ;

[0096] S404: Establishing a directed graph model ,in The elements in are connected by internal adjacent edges, The elements in are connected by external adjacent edges.

[0097] S5: Based on the directed graph model, a node queue is constructed, and nodes that are adjacent in position but not adjacent in time are interpolated to obtain the complete trajectory of the target.

[0098] S501: Determine the set The longest element in ,Will All position sequences in Put in queue , create a collection Used to record target trajectory;

[0099] S502: From the queue Take out the starting node and record it as ;

[0100] S503: In order from the queue Take the next node , until the queue If it is empty, jump to step S506;

[0101] S504: Comparison and ,in express The recorded image acquisition time, express The recorded image acquisition time;

[0102] S505: If satisfied ,express and Time is adjacent, Add to collection ;

[0103] make , jump to step S503;

[0104] If not satisfied ,express and The times are not adjacent;

[0105] Along the directed graph The external neighboring edges of Time adjacent nodes ;

[0106] Will Add to collection ,make , jump to step S504;

[0107] S506: Collection The position sequence of the target recorded in the middle node sequence , which is the complete trajectory of the target.

[0108] Embodiment 2: The present invention also discloses a target tracking system for multi-video source fusion, comprising the following five modules:

[0109] Camera synchronization and distortion measurement module: synchronize camera time and determine the distortion of camera images relative to the real world;

[0110] Target recognition module: collects image sequences, identifies targets, and records the coordinate sequence of targets in the images;

[0111] Target position transformation module: Combined with camera distortion, the target coordinate sequence in the image is converted into the target position sequence in the Cartesian coordinate system;

[0112] Directed graph construction module: define node sets, define node transfer sets, and establish directed graph models;

[0113] Target trajectory generation module: create a queue, compare image acquisition time, determine time-adjacent nodes, traverse external adjacent edges, perform node interpolation, and output the complete trajectory of the target.

[0114] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0115] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0116] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A target tracking method for multi-video source fusion, characterized in that: The following steps are involved: S1: Time synchronization of the camera and determination of camera distortion based on a reference target; including: S101: Define the camera model: ； in, Indicates the camera number. Indicates that the reference target is in the camera The position in , Indicates that the reference target is in the camera The coordinate values ​​in represents the position of the reference target in the Cartesian coordinate system, Represents the real world and the camera The rotation relationship matrix between Represents the real world and the camera The translation relationship matrix between them; S102: Determine the distortion using the coordinate position of the reference target in the two-dimensional image: ； ； in, Indicates the camera The distortion of the image relative to the real world, Indicates that the reference target is in the camera The coordinate values ​​in the generated two-dimensional image are, Indicates the camera The ratio of focal length to pixel size in horizontal and vertical directions, Indicates the camera The main point of S2: Collect camera image sequences, use the pre-trained convolutional neural network model to identify the target, and record the coordinate sequence of the target in the image; including: S201: Collecting camera image sequences ,in, represents the image sequence acquisition time; S202: Use the pre-trained convolutional neural network model to train the image sequence Perform target recognition. If it is determined to be a target to be tracked, the coordinate sequence of the target in the image is recorded. ; S3: Combined with the camera distortion, the coordinate sequence of the target in the image is converted into the position sequence of the target in the Cartesian coordinate system; including: S301: Using a camera The image is distorted relative to the real world, and the target is calculated in the camera The position sequence in: ； in, Indicates that the target is in the camera The position sequence in ; S302: Calculate the position sequence of the target in the Cartesian coordinate system: ； in, represents the position sequence of the target in the Cartesian coordinate system, Represents the rotation relationship matrix Take the inverse matrix; S4: Using the position sequence of the target in the Cartesian coordinate system, define the node set and the node transfer set, and establish a directed graph model; including: S401: Using the position sequence of the target in the Cartesian coordinate system, define a node set: ； in, Represents a collection of nodes. Is a collection of nodes The elements in Camera included The sequence of locations where the identified target appears and image acquisition time ; S402: Constructing a position adjacent discriminant: ； in, Indicates another camera number. It represents the distance that the target moves in the Cartesian coordinate system at adjacent times. is the distance threshold, Indicates that the positions of the targets in the Cartesian coordinate system are adjacent at adjacent times. It means that the positions of the targets at adjacent times in the Cartesian coordinate system are not adjacent; S403: Define node transfer set: ； in, represents the node transfer set, is the node transfer set The elements in Contains all the Position sequence pair ; S404: Establishing a directed graph model ,in The elements in are connected by internal adjacent edges, The elements in are connected by external adjacent edges; S5: Based on the directed graph model, a node queue is constructed, and nodes with adjacent positions but not adjacent times are interpolated to obtain the complete trajectory of the target; including: S501: Determine the set The longest element in ,Will All position sequences in Put in queue , create a collection Used to record target trajectory; S502: From the queue Take out the starting node and record it as ; S503: In order from the queue Take the next node , until the queue If it is empty, jump to step S506; S504: Comparison and ,in express The recorded image acquisition time, express The recorded image acquisition time; S505: If satisfied ,express and Time is adjacent, Add to collection ; make , jump to step S503; If not satisfied ,express and The times are not adjacent; Along the directed graph The external neighboring edges of Time adjacent nodes ; Will Add to collection ,make , jump to step S504; S506: Collection The position sequence of the target recorded in the middle node sequence , which is the complete trajectory of the target.

2. A target tracking system for multi-video source fusion, characterized in that: include: Camera synchronization and distortion measurement module: synchronize camera time and determine the distortion of camera images relative to the real world; Target recognition module: collects image sequences, identifies targets, and records the coordinate sequence of targets in the images; Target position transformation module: Combined with camera distortion, the target coordinate sequence in the image is converted into the target position sequence in the Cartesian coordinate system; Directed graph construction module: define node sets, define node transfer sets, and establish directed graph models; Target trajectory generation module: create a queue, compare image acquisition time, determine time-adjacent nodes, traverse external adjacent edges, perform node interpolation, and output the complete trajectory of the target; To realize the target tracking method of multi-video source fusion as described in claim 1.

Images