Video target tracking and prediction method and system based on graph neural network

Through the video target tracking method based on graph neural network, the pre-trained model is used to identify target categories and relationship categories, build a spatial graph, and use long and short-term memory network to predict trajectory, which solves the video target tracking under view angle changes and background clutter Accuracy issues improve the accuracy and efficiency of tracking and prediction.

CN118735962BActive Publication Date: 2024-11-22CHANGSHA CHAOCHUANG ELECTRONICS TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411226692.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-03
Publication Date
2024-11-22
Estimated Expiration
2044-09-03

AI Technical Summary

Technical Problem

Existing video target tracking methods are difficult to accurately identify and track under conditions such as video viewing angle changes and background clutter, resulting in difficulty in starting, increasing false tracks, high calculation costs, poor trajectory positioning accuracy, correlation errors and trajectory discontinuity .

Method used

Using a graph-based neural network method, the target category and relation category are identified through the pre-trained Faster-RCNN model, a spatial graph is constructed, and the target trajectory is predicted using the long and short-term memory network to improve the tracking accuracy and prediction accuracy.

Benefits of technology

Accurate starting tracking and trajectory prediction of video goals are achieved, the accuracy and efficiency of identification and prediction are improved, the calculation cost is reduced, and the occurrence of false trajectories is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118735962B_ABST
    Figure CN118735962B_ABST
Patent Text Reader

Abstract

The present invention discloses a video target tracking and prediction method and system based on graph neural network, the method comprising: S1: using pre-trained Faster-RCNN model 1 and Faster-RCNN model 2 respectively, identifying targets and target relationships in the video, and marking target categories and target relationship categories; S2: constructing a spatial graph based on the identified target categories and target relationship categories; S3: judging the correctness of the spatial graph, and updating the incorrect spatial graph; S4: determining the starting time of target tracking based on the confirmed spatial graph; S5: tracking the target, and using a long short-term memory network to predict the target trajectory based on the obtained target trajectory. The present invention is based on a graph neural network, uses the spatial relationship of the target, realizes the accurate start of target tracking, and uses a long short-term memory network to accurately predict the target trajectory.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and in particular to a video target tracking and prediction method and system based on graph neural network. Background Art

[0002] Object tracking belongs to the field of computer vision. It is achieved by detecting and tracking the target of interest from a sequence of video frames. It is widely used in the fields of intelligent driving, road monitoring, and smart transportation. Due to the changes in video viewing angle and uneven background clutter, classic tracking algorithms cannot be directly applied to video target tracking. Accurate identification of targets is the premise and key to tracking. Limited by the video background, clutter, and occlusion and overlap of adjacent targets, the detection and identification of targets has a low accuracy rate. This makes it difficult to start target tracking and increases false tracks.

[0003] The existing target tracking method assumes that the target has multiple starting trajectories, obtains multiple small trajectories in a short period of time, and removes false trajectories by trajectory association to obtain the accurate and long trajectory of the target. However, this method consumes a lot of computing costs and greatly delays the trajectory determination time. At the same time, the target trajectory obtained has the risk of poor positioning accuracy, association errors, and trajectory discontinuity. Summary of the invention

[0004] In view of this, the present invention provides a video target tracking and prediction method and system based on graph neural network. The present invention is based on graph neural network, utilizes the spatial relationship of the target to achieve accurate start of target tracking, and adopts long short-term memory network to accurately predict the target trajectory.

[0005] The technical solution proposed by the present invention is a video target tracking and prediction method based on graph neural network, comprising the following steps:

[0006] S1: Use the pre-trained Faster-RCNN model 1 and Faster-RCNN model 2 to identify objects and object relationships in the video, and label the object categories and object relationship categories;

[0007] S2: Construct a spatial graph based on the identified target categories and target relationship categories;

[0008] S3: judge the correctness of the spatial graph and update the incorrect spatial graph;

[0009] S4: Determine the start time of target tracking based on the confirmed spatial map;

[0010] S5: Track the target and predict the target trajectory based on the obtained target trajectory using a long short-term memory network.

[0011] Optionally, the step S1 includes:

[0012] S11: Label the target categories and target relationship categories in the video respectively and build a target category dataset and target relation category dataset ;

[0013] S12: Build Faster-RCNN network 1 and based on the target category dataset Train the Faster-RCNN network 1 to obtain the pre-trained Faster-RCNN model 1;

[0014] Construct Faster-RCNN network 2 and use it based on the target relationship category dataset Train the Faster-RCNN network 2 to obtain the pre-trained Faster-RCNN model 2;

[0015] S13: Input the video frame to be identified into the pre-trained Faster-RCNN model 1, mark the target category, and obtain the video frame marked with the target category;

[0016] The video frames with the labeled target categories are input into the pre-trained Faster-RCNN model 2 to label the target relationship categories.

[0017] Optionally, in step S12:

[0018] The training steps of the Faster-RCNN network include:

[0019] S121: Divide the data set into 5 sub-data sets of equal number;

[0020] S122: Use the 2nd to 5th sub-datasets as training sets to train the Faster-RCNN network and obtain the first model;

[0021] Use the first sub-dataset as the test set and evaluate the accuracy of the first model on the test set;

[0022] S123: Use the first sub-dataset and the third to fifth sub-datasets as training sets to train the Faster-RCNN network and obtain the second model;

[0023] Use the second sub-dataset as the test set and evaluate the accuracy of the second model on the test set;

[0024] S124: Use the first and second sub-datasets and the fourth and fifth sub-datasets as training sets to train the Faster-RCNN network and obtain the third model;

[0025] Use the third sub-dataset as the test set and evaluate the accuracy of the third model on the test set;

[0026] S125: Use the 1st to 3rd sub-datasets and the 5th sub-dataset as training sets to train the Faster-RCNN network and obtain the fourth model;

[0027] Use the fourth sub-dataset as the test set and evaluate the accuracy of the fourth model on the test set;

[0028] S126: Use the 1st to 4th sub-datasets as training sets to train the Faster-RCNN network and obtain the fifth model;

[0029] Use the fifth sub-dataset as the test set and evaluate the accuracy of the fifth model on the test set;

[0030] S127: Take the model with the highest accuracy among the five models as the pre-trained Faster-RCNN model;

[0031] Wherein, the Faster-RCNN network is Faster-RCNN network 1 or Faster-RCNN network 2;

[0032] When the Faster-RCNN network is Faster-RCNN network 1, the dataset is a target category dataset , the Faster-RCNN model is Faster-RCNN model 1;

[0033] When the Faster-RCNN network is Faster-RCNN model network 2, the dataset is a target relationship category dataset , the Faster-RCNN model is Faster-RCNN model 1.

[0034] Optionally, in step S2, the spatial graph is represented as:

[0035] ;

[0036] in, Indicates the first goals and The spatial map of the target, Respectively represent goals and The category of the target, Indicates goals and The relationship between the targets, The spatial diagram is derived from Frame video.

[0037] Optionally, step S3 includes:

[0038] S31: Using the trained support vector machine, the correctness of the spatial graph is judged;

[0039] If the spatial map is correct, then this spatial map is used as the confirmed spatial map;

[0040] S32: If the spatial map is incorrect, the original spatial map is updated with the spatial map from the next frame of video, and the spatial map from the next frame of video is executed in step S31.

[0041] Optionally, the step S31 includes:

[0042] Jordi goals and The relationship between the targets If the space diagram is fixed, Marked as positive samples;

[0043] Jordi goals and The relationship between the targets is changing, then the space map Marked as negative samples;

[0044] Positive samples and negative samples constitute the data set for support vector machine training .

[0045] Optionally, step S5 includes:

[0046] S51: determining the actual position of the target in the global coordinate system according to the relationship between the video coordinate system and the global coordinate system;

[0047] S52: extracting the acquired target trajectory features, where the target trajectory features include speed, acceleration, and direction;

[0048] The target trajectory features are input into the long short-term memory network, and the long short-term memory network predicts the target trajectory.

[0049] Optionally, the step S51 includes:

[0050] S511: Determine the reference point located in the video and clarify the actual position of the reference point in the global coordinate system ;

[0051] S512: Select an edge point in the video frame as the coordinate origin, establish a video coordinate system, and clarify the position of the reference point in the video coordinate system ;

[0052] Determine the position of the target in the video coordinate system :

[0053] S513: There is a relationship between the position of the reference point in the video coordinate system and the actual position in the global coordinate system:

[0054] ;

[0055] in, Represents the transformation matrix between the video coordinate system and the global coordinate system,

[0056] ;

[0057] According to the relationship between the video coordinate system and the global coordinate system, the actual position of the target in the global coordinate system is obtained:

[0058] .

[0059] The present invention also provides a video target tracking and prediction system based on graph neural network, comprising:

[0060] Target category and target relationship category recognition module: pre-train Faster-RCNN model 1, pre-train Faster-RCNN model 2, identify target category, identify target relationship category, label target category, label target relationship category;

[0061] Spatial graph construction module: constructs a spatial graph;

[0062] Spatial graph judgment module: judges the correctness of the spatial graph and updates the incorrect spatial graph;

[0063] Target tracking start module: obtain the target tracking start time;

[0064] Target tracking prediction module: determine the actual position of the target, extract trajectory features, and predict the trajectory; Beneficial Effects

[0065] The present invention utilizes a pre-trained model to accurately identify target categories and target relationship categories, and on this basis, utilizes the invariance of target relationships to improve the accuracy of target and target relationship recognition; based on an established spatial map, utilizes the spatial relationship of the target to achieve accurate initiation of target tracking; extracts target trajectory features, and uses a long short-term memory network, which is beneficial to providing target prediction precision and accuracy.

[0066] In the model training stage, the present invention adopts a method of dividing subsets into multiple trainings, which is conducive to making full use of the data set and improving the precision and accuracy of model training; the present invention constructs a spatial graph, which is conducive to binding the target type and the target relationship type, and mining deep information without changing the data volume; the present invention judges the correctness of the spatial graph and updates the incorrect spatial graph, which is conducive to finding the video frame with the most obvious target features; the present invention uses the reference points located in the video, which is conducive to achieving accurate positioning of the target in the global coordinate system. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A flowchart of a method for video target tracking and prediction based on a graph neural network is provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0068] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0069] Embodiment 1:

[0070] A video target tracking and prediction method based on graph neural network, such as Figure 1 As shown, the following steps are included:

[0071] S1: Use the pre-trained Faster-RCNN model 1 and Faster-RCNN model 2 to identify objects and object relationships in the video, and label the object categories and object relationship categories;

[0072] Step S1 includes:

[0073] S11: Label the target categories and target relationship categories in the video respectively and build a target category dataset and target relation category dataset ;

[0074] In the embodiment of the present invention, there are multiple vehicle targets in the video, and a frame is drawn for each vehicle, and the frame is as close to the vehicle as possible, and other interference in the frame is as little as possible; the vehicles are distinguished according to the appearance conditions such as the color, shape, and logo of the vehicle, and the same vehicle is marked as the same target; the relationship between adjacent vehicle targets in the video includes: occlusion, overlap, and synchronization. The relationship between the vehicle and the driver is synchronization, and the relationship between the vehicle and the road is occlusion. For the area containing multiple targets, the frame is drawn to include relevant targets as much as possible without omission.

[0075] S12: Build Faster-RCNN network 1 and based on the target category dataset Train the Faster-RCNN network 1 to obtain the pre-trained Faster-RCNN model 1;

[0076] Construct Faster-RCNN network 2 and use it based on the target relationship category dataset Train the Faster-RCNN network 2 to obtain the pre-trained Faster-RCNN model 2;

[0077] Among them, the training steps of the Faster-RCNN network include:

[0078] S121: Divide the data set into 5 sub-data sets of equal number;

[0079] S122: Use the 2nd to 5th sub-datasets as training sets to train the Faster-RCNN network and obtain the first model;

[0080] Use the first sub-dataset as the test set and evaluate the accuracy of the first model on the test set;

[0081] S123: Use the first sub-dataset and the third to fifth sub-datasets as training sets to train the Faster-RCNN network and obtain the second model;

[0082] Use the second sub-dataset as the test set and evaluate the accuracy of the second model on the test set;

[0083] S124: Use the first and second sub-datasets and the fourth and fifth sub-datasets as training sets to train the Faster-RCNN network and obtain the third model;

[0084] Use the third sub-dataset as the test set and evaluate the accuracy of the third model on the test set;

[0085] S125: Use the 1st to 3rd sub-datasets and the 5th sub-dataset as training sets to train the Faster-RCNN network and obtain the fourth model;

[0086] Use the fourth sub-dataset as the test set and evaluate the accuracy of the fourth model on the test set;

[0087] S126: Use the 1st to 4th sub-datasets as training sets to train the Faster-RCNN network and obtain the fifth model;

[0088] Use the fifth sub-dataset as the test set and evaluate the accuracy of the fifth model on the test set;

[0089] S127: Take the model with the highest accuracy among the five models as the pre-trained Faster-RCNN model;

[0090] Wherein, the Faster-RCNN network is Faster-RCNN network 1 or Faster-RCNN network 2;

[0091] When the Faster-RCNN network is Faster-RCNN network 1, the dataset is a target category dataset , the Faster-RCNN model is Faster-RCNN model 1;

[0092] When the Faster-RCNN network is Faster-RCNN model network 2, the dataset is a target relationship category dataset , the Faster-RCNN model is Faster-RCNN model 1.

[0093] S13: Input the video frame to be identified into the pre-trained Faster-RCNN model 1, mark the target category, and obtain the video frame marked with the target category;

[0094] The video frames with the labeled target categories are input into the pre-trained Faster-RCNN model 2 to label the target relationship categories.

[0095] S2: Construct a spatial graph based on the identified target categories and target relationship categories;

[0096] The spatial diagram is represented as:

[0097] ;

[0098] in, Indicates the first goals and The spatial map of the target, Respectively represent goals and The category of the target, Indicates goals and The relationship between the targets, The spatial diagram is derived from Frame video.

[0099] S3: judge the correctness of the spatial graph and update the incorrect spatial graph;

[0100] Step S3 includes:

[0101] S31: Using the trained support vector machine, the correctness of the spatial graph is judged;

[0102] If the spatial map is correct, then this spatial map is used as the confirmed spatial map;

[0103] Among them, if goals and The relationship between the targets If the space diagram is fixed, Marked as a positive sample; if goals and The relationship between the targets is changing, then the space map Marked as negative samples; positive samples and negative samples constitute the data set for support vector machine training .

[0104] S32: If the spatial map is incorrect, the original spatial map is updated with the spatial map from the next frame of video, and the spatial map from the next frame of video is executed in step S31.

[0105] In the embodiment of the present invention, during the movement of the car, the relationship between the driver and the car is synchronization, and the relationship between the car and the road is occlusion. The above relationship does not change during the movement of the target. The correctness of the spatial graph is judged by using the relationship invariance of special targets.

[0106] S4: Based on the confirmed spatial map, determine the start time of target tracking.

[0107] S5: Track the target and predict the target trajectory based on the obtained target trajectory using a long short-term memory network.

[0108] Step S5 includes:

[0109] S51: determining the actual position of the target in the global coordinate system according to the relationship between the video coordinate system and the global coordinate system;

[0110] S511: Determine the reference point located in the video and clarify the actual position of the reference point in the global coordinate system ;

[0111] In the embodiment of the present invention, a stationary background is selected as a reference point, such as a starting point of a road, a street lamp, or a telephone pole.

[0112] S512: Select an edge point in the video frame as the coordinate origin, establish a video coordinate system, and clarify the position of the reference point in the video coordinate system ;

[0113] Determine the position of the target in the video coordinate system ;

[0114] S513: There is a relationship between the position of the reference point in the video coordinate system and the actual position in the global coordinate system:

[0115] ;

[0116] in, Represents the transformation matrix between the video coordinate system and the global coordinate system,

[0117] ;

[0118] According to the relationship between the video coordinate system and the global coordinate system, the actual position of the target in the global coordinate system is obtained:

[0119] .

[0120] S52: extracting the acquired target trajectory features, where the target trajectory features include speed, acceleration, and direction;

[0121] The target trajectory features are input into the long short-term memory network, and the long short-term memory network predicts the target trajectory.

[0122] Embodiment 2: The present invention also provides a video target tracking and prediction system based on graph neural network, comprising the following five modules:

[0123] Target category and target relationship category recognition module: pre-train Faster-RCNN model 1, pre-train Faster-RCNN model 2, identify target category, identify target relationship category, label target category, label target relationship category;

[0124] Spatial graph construction module: constructs a spatial graph;

[0125] Spatial graph judgment module: judges the correctness of the spatial graph and updates the incorrect spatial graph;

[0126] Target tracking start module: obtain the target tracking start time;

[0127] Target tracking prediction module: determine the actual position of the target, extract trajectory features, and predict the trajectory.

[0128] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0130] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A video target tracking and prediction method based on graph neural network, characterized in that: The method comprises: S1: Use the pre-trained Faster-RCNN model 1 and Faster-RCNN model 2 to identify objects and object relationships in the video, and label the object categories and object relationship categories; S2: Construct a spatial graph based on the identified target categories and target relationship categories; The spatial diagram is represented as: ; in, Indicates the first goals and The spatial map of the target, Respectively represent goals and The category of the target, Indicates goals and The relationship between the targets, The spatial diagram is derived from Frame video; S3: judge the correctness of the spatial graph and update the incorrect spatial graph; The step S3 comprises: S31: Using the trained support vector machine, the correctness of the spatial graph is judged; If the spatial map is correct, then this spatial map is used as the confirmed spatial map; The step S31 comprises: Jordi goals and The relationship between the targets If the space diagram is fixed, Marked as positive samples; Jordi goals and The relationship between the targets is changing, then the spatial map Marked as negative samples; Positive samples and negative samples constitute the data set for support vector machine training ; S32: if the spatial map is incorrect, the original spatial map is updated with the spatial map from the next frame of video, and the spatial map from the next frame of video is executed in step S31; S4: Determine the start time of target tracking based on the confirmed spatial map; S5: Track the target and predict the target trajectory based on the obtained target trajectory using a long short-term memory network.

2. The video target tracking and prediction method based on graph neural network according to claim 1 is characterized in that: The step S1 comprises: S11: Label the target categories and target relationship categories in the video respectively and build a target category dataset and target relation category dataset ; S12: Build Faster-RCNN network 1 and based on the target category dataset Train the Faster-RCNN network 1 to obtain the pre-trained Faster-RCNN model 1; Construct Faster-RCNN network 2 and use it based on the target relationship category dataset Train the Faster-RCNN network 2 to obtain the pre-trained Faster-RCNN model 2; S13: Input the video frame to be identified into the pre-trained Faster-RCNN model 1, mark the target category, and obtain the video frame marked with the target category; The video frames with the labeled target categories are input into the pre-trained Faster-RCNN model 2 to label the target relationship categories.

3. The video target tracking and prediction method based on graph neural network according to claim 2 is characterized in that: In step S12: The training steps of the Faster-RCNN network include: S121: Divide the data set into 5 sub-data sets of equal number; S122: Use the 2nd to 5th sub-datasets as training sets to train the Faster-RCNN network and obtain the first model; Use the first sub-dataset as the test set and evaluate the accuracy of the first model on the test set; S123: Use the first sub-dataset and the third to fifth sub-datasets as training sets to train the Faster-RCNN network and obtain the second model; Use the second sub-dataset as the test set and evaluate the accuracy of the second model on the test set; S124: Use the first and second sub-datasets and the fourth and fifth sub-datasets as training sets to train the Faster-RCNN network and obtain the third model; Use the third sub-dataset as the test set and evaluate the accuracy of the third model on the test set; S125: Use the 1st to 3rd sub-datasets and the 5th sub-dataset as training sets to train the Faster-RCNN network and obtain the fourth model; Use the fourth sub-dataset as the test set and evaluate the accuracy of the fourth model on the test set; S126: Use the 1st to 4th sub-datasets as training sets to train the Faster-RCNN network and obtain the fifth model; Use the fifth sub-dataset as the test set and evaluate the accuracy of the fifth model on the test set; S127: Take the model with the highest accuracy among the five models as the pre-trained Faster-RCNN model; Wherein, the Faster-RCNN network is Faster-RCNN network 1 or Faster-RCNN network 2; When the Faster-RCNN network is Faster-RCNN network 1, the dataset is a target category dataset , the Faster-RCNN model is Faster-RCNN model 1; When the Faster-RCNN network is Faster-RCNN model network 2, the dataset is a target relationship category dataset , the Faster-RCNN model is Faster-RCNN model 2.

4. The video target tracking and prediction method based on graph neural network according to claim 1, characterized in that: The step S5 comprises: S51: determining the actual position of the target in the global coordinate system according to the relationship between the video coordinate system and the global coordinate system; S52: extracting the acquired target trajectory features, where the target trajectory features include speed, acceleration, and direction; The target trajectory features are input into the long short-term memory network, and the long short-term memory network predicts the target trajectory.

5. The video target tracking and prediction method based on graph neural network according to claim 4, characterized in that: The step S51 comprises: S511: Determine the reference point located in the video and clarify the actual position of the reference point in the global coordinate system ; S512: Select an edge point in the video frame as the coordinate origin, establish a video coordinate system, and clarify the position of the reference point in the video coordinate system ; Determine the position of the target in the video coordinate system : S513: There is a relationship between the position of the reference point in the video coordinate system and the actual position in the global coordinate system: ; in, Represents the transformation matrix between the video coordinate system and the global coordinate system, ; According to the relationship between the video coordinate system and the global coordinate system, the actual position of the target in the global coordinate system is obtained: 。 6. A video target tracking and prediction system based on graph neural network, characterized in that: include: Target category and target relationship category recognition module: pre-train Faster-RCNN model 1, pre-train Faster-RCNN model 2, identify target category, identify target relationship category, label target category, label target relationship category; Spatial graph construction module: constructs a spatial graph; Spatial graph judgment module: judges the correctness of the spatial graph and updates the incorrect spatial graph; Target tracking start module: obtain the target tracking start time; Target tracking prediction module: determine the actual position of the target, extract trajectory features, and predict the trajectory; To implement a video target tracking and prediction method based on graph neural network as described in any one of claims 1-5.