VTS radar target solution fusion method, system and equipment based on surveillance video assistance
By using surveillance video assistance methods in the ship traffic management system, the association relationship between the video target and the radar target is established, and the image feature comparison reassociation and fusing separated radar targets are solved, the problem of identity loss after radar target fusion is solved. Improved tracking stability.
Patent Information
- Application Number
- CN202210739521.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-28
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-06-28
AI Technical Summary
In the ship traffic management system, when the radar targets are fused and separated, the previous identity information cannot be associated again, resulting in false tracking or loss of tracks, affecting the system's stable tracking performance.
Through a method based on surveillance video assistance, the image sampling before fusion and the image target comparison identity after fusion are used to establish the association relationship between the video target and the radar target, predict the fusion node, and re-comparison through image feature cross-comparison Associate the separated radar target after fusion.
Maximizes the credibility of target identity confirmation after target decomposition and fusion, reduces the probability of mistracking and track loss after radar target fusion, and improves tracking stability.
Smart Images

Figure CN115201809B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of radar target fusion technology, and in particular to a VTS radar target de-fusion method, system and equipment based on monitoring video assistance. Background Art
[0002] Vessel Traffic System (VTS) is the most important maritime equipment to ensure the safety of ship navigation. It plays a vital role in reducing ship traffic accidents, improving ship navigation efficiency, and preventing water pollution. VTS is a system that uses communication facilities such as AIS base stations, radars, CCTV, wireless phones, and ship-borne terminals to monitor ships sailing in and out of ports, and provide these ships with the safety information they need during navigation. The system can monitor whether the ship is off the route, the direction of travel, the speed, and the intersection of ships, so as to quickly provide ships with the safety navigation information they need when entering and leaving the port.
[0003] In the Vessel Traffic Service System (VTS), during the navigation process, especially in inland waterways, two ships may cross or meet each other. In this case, the echoes of the two ships in the radar video will merge to form one echo. In such a phenomenon, in order to avoid collision, ships will often maneuver, and the speed and course of the ships are likely to change, which poses a great challenge to target tracking. Mistracking or track loss often occurs, affecting the continuous and stable tracking of the ship, and then affecting the stable tracking performance of the system. It has a certain impact on the water traffic supervision of VTS users. When ships may collide or other accidents, they cannot be notified to the target ship in a timely and accurate manner, which may cause major accidents.
[0004] Special processing is required for the situation where radar targets cannot be associated with their previous identity information after being separated after fusion. The classical method uses track association and track estimation to solve the target fusion, but this method is effective but still has a high probability of error. Summary of the invention
[0005] In order to solve the technical problems existing in the prior art, the present invention provides a VTS radar target defusion method, system and equipment based on surveillance video assistance. The method is based on image sampling before radar echo fusion and image target identity comparison after fusion to reduce the probability of mis-following and track loss after target fusion and improve tracking stability.
[0006] The first object of the present invention is to provide a VTS radar target de-fusion method based on monitoring video assistance.
[0007] The second object of the present invention is to provide a VTS radar target solution fusion system based on monitoring video assistance.
[0008] A third object of the present invention is to provide a computer device.
[0009] The first object of the present invention can be achieved by adopting the following technical solutions:
[0010] A VTS radar target de-fusion method based on surveillance video assistance, the method comprising:
[0011] Step 1: calibrate the monitoring area and the radar sector, select multiple video targets in the monitoring area screen, read the image position coordinates of the video targets, select radar targets corresponding to the multiple video targets in the radar sector, read the longitude and latitude coordinates of the image position of the radar target, and convert the longitude and latitude coordinates of the image position of the radar target into Mercator coordinates;
[0012] Step 2: Solve the projection relationship between the monitoring area coordinates and the radar sector coordinates, and calculate the relationship matrix H between the image position coordinates of the video target in the monitoring area and the Mercator coordinates of the radar target in the radar sector by the least squares method;
[0013] Step 3, determining the association relationship between the video target and the radar target, detecting and tracking the video target in the monitoring area and the radar target in the radar sector respectively, converting the image position coordinates of the video target in the monitoring area into Mercator coordinates through the relationship matrix H, converting the Mercator coordinates of the video target into longitude and latitude, recording the longitude and latitude tracks of the video target and the longitude and latitude tracks of the radar target, and associating the longitude and latitude tracks of the video target and the longitude and latitude tracks of the radar target;
[0014] Step 4: predict the radar target fusion occurrence node, first set the time interval, calculate the average speed of each radar target in the set time interval, and predict its position after T seconds; traverse the predicted calculated positions, and mark the current two radar targets when the distance between the two radar targets is less than the echo target size;
[0015] Step 5: Video target sampling. The video target corresponding to the marked radar target in the monitoring area is identified through the association relationship between the video target and the radar target, and the ship image corresponding to the video target in the monitoring area is extracted, and the ID of the ship image corresponding to the video target is recorded; after waiting for several seconds, when one or more radar target tracking of the marked radar target is lost, it is determined that the radar target has been fused, otherwise the marking of the marked radar target is cancelled and the process returns to step 4;
[0016] Step 6: Detect radar target separation nodes. When a new radar target appears around a single tracked radar target in the overlapping area of the radar sector and the monitoring area, and the new radar target is re-associated with the video target, it is determined that the radar target is separated after fusion.
[0017] Step seven, cross-comparison of image features between the video target judged as a radar target fused and the video target judged as a radar target separated after fusion is performed, the best correlation item is selected, and the radar target separated after fusion is re-associated with the best correlation item.
[0018] The second object of the present invention can be achieved by adopting the following technical solutions:
[0019] A VTS radar target solution and fusion system based on surveillance video assistance, the system comprises:
[0020] A calibration module is used to calibrate the monitoring area and the radar sector, select multiple video targets in the monitoring area screen, read the image position coordinates of the video targets, select radar targets corresponding to the multiple video targets in the radar sector, read the longitude and latitude coordinates of the image position of the radar target, and convert the longitude and latitude coordinates of the image position of the radar target into Mercator coordinates;
[0021] The association module is used to solve the projection relationship between the coordinates of the monitoring area and the coordinates of the radar sector, and calculate the relationship matrix H between the image position coordinates of the video target in the monitoring area and the Mercator coordinates of the radar target in the radar sector by the least square method; determine the association relationship between the video target and the radar target, respectively detect and track the video target in the monitoring area and the radar target in the radar sector, convert the image position coordinates of the video target in the monitoring area into Mercator coordinates by the relationship matrix H, convert the Mercator coordinates of the video target into longitude and latitude, record the longitude and latitude tracks of the video target and the longitude and latitude tracks of the radar target, and associate the longitude and latitude tracks of the video target with the longitude and latitude tracks of the radar target;
[0022] The prediction fusion occurrence module is used to predict the radar target fusion occurrence node. First, a time interval is set, the average speed of each radar target in the set time interval is calculated, and its position after T seconds is predicted; the predicted calculated positions are traversed, and when the distance between two radar targets is less than the echo target size, the current two radar targets are marked;
[0023] The video target image sampling module is used for video target sampling. The video target corresponding to the marked radar target in the monitoring area is identified through the association relationship between the video target and the radar target, the ship image corresponding to the video target in the monitoring area is extracted, and the ID of the ship image corresponding to the video target is recorded; after waiting for several seconds, when one or more radar target tracking of the marked radar target is lost, it is determined that the radar target has been fused, otherwise the mark of the marked radar target is cancelled and the process returns to step 4;
[0024] The predicted target separation module is used to detect radar target separation nodes. When a new radar target appears around a single tracked radar target in the overlapping area of the radar sector and the monitoring area, and the new radar target is re-associated with the video target, it is judged that the radar target is separated after fusion;
[0025] The optimal correlation item association module traverses N video targets judged as radar targets that have been fused and cross-compares the image features of video targets judged as radar targets that have been separated after fusion, selects the optimal correlation item, and re-associates the radar targets separated after fusion with the optimal correlation item.
[0026] The third object of the present invention can be achieved by adopting the following technical solutions:
[0027] A computer device comprises a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the above-mentioned VTS radar target de-fusion method based on the assistance of surveillance video is implemented.
[0028] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0029] The present invention provides a VTS radar target defusion method, system and device based on monitoring video assistance. Through a method based on image sampling before fusion of radar echoes and image target identity comparison after fusion, a monitoring video-assisted method is used to maximize the credibility of target identity confirmation after target defusion, reduce the probability of mis-following and track loss after radar target fusion, and improve tracking stability. Compared with the traditional method, the effect of confirming the target identity after defusion is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0031] Figure 1is a flow chart of a VTS radar target solution fusion method in an embodiment of the present invention;
[0032] Figure 2 is a schematic diagram of the correspondence between image point selection and radar coordinate point selection in an embodiment of the present invention;
[0033] Figure 3 is a schematic diagram of a relationship matrix in an embodiment of the present invention;
[0034] Figure 4 is a schematic diagram of probability distribution of speed difference in an embodiment of the present invention;
[0035] Figure 5 Schematic diagram of radar fusion and separation process in an embodiment of the present invention. DETAILED DESCRIPTION
[0036] The technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is obvious that the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and the implementation of the present invention is not limited to this. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0037] Example 1
[0038] like Figure 1 As shown, the VTS radar target defusion method based on monitoring video assistance of the present invention comprises the following steps:
[0039] Step 1: calibrate the monitoring area and radar sector, select multiple video targets in the monitoring area screen, read the image position coordinates of the video targets, select radar targets corresponding to the multiple video targets in the radar sector, read the longitude and latitude coordinates of the image position of the radar targets, and convert the longitude and latitude coordinates of the image position of the radar targets into Mercator coordinates.
[0040] like Figure 2 As shown in FIG. 1 , the corresponding schematic diagram of image point selection and radar coordinate point selection is shown. In this embodiment, the surveillance camera is adjusted within the radar coverage range so that the video covers the water area to be supervised. First, four video targets are selected in the video screen, and the image position coordinates of the video targets are taken. The image position coordinates of the video targets are the coordinates of the bottom midpoint of the image of the video targets. The image position coordinate set of the video targets is recorded as P camera , represented by (u,v,1) T ; At the same time, select four corresponding radar targets in the radar sector. The image position of the radar target is the image center of gravity of the radar target. Read the longitude and latitude coordinates of the image center of gravity of the radar target and convert them into Mercator coordinates. The Mercator coordinate set of the radar target is recorded as P radar, represented by (x,y,1) T , the formula for converting longitude and latitude coordinates to Mercator coordinates is as follows:
[0041]
[0042] Where R represents the radius of the earth, R is 6378.1370 kilometers, Ln represents the geographical longitude, and La represents the geographical latitude.
[0043] Step 2: Solve the projection relationship between the monitoring area coordinates and the radar sector coordinates, and calculate the relationship matrix H between the image position coordinates of the video target in the monitoring area and the Mercator coordinates of the radar target in the radar sector by the least squares method.
[0044] Solve the projection relationship between the monitoring area coordinates and the radar sector coordinates. The mapping relationship between the image position coordinates of the video target in the monitoring area and the Mercator coordinates of the radar target in the radar sector is as follows:
[0045] P radar HP camera
[0046]
[0047] Among them, P radar is the Mercator coordinate set of the radar target, expressed as (x, y, 1) T ; camera is the image position coordinate set of the video target, expressed as (u,v,1) T ; H is a 3×3 non-singular matrix.
[0048] The relationship matrix H is obtained by the least squares method, and the above mapping relationship is transformed into the following form:
[0049]
[0050] Substituting the above four corresponding points into the least squares principle, we can get:
[0051]
[0052] like Figure 3 As shown, A represents the above 8×8 matrix, h represents the middle 8×1 matrix, and b represents the 8×1 matrix on the right side of the equal sign; the elements in h and the values to be solved are obtained according to the matrix operation relationship:
[0053] h=(A T A) -1 (A T b)
[0054] Bring the elements of the h matrix back to the all-relation matrix H.
[0055] Step three, determine the association relationship between the video target and the radar target, detect and track the video target in the monitoring area and the radar target in the radar sector respectively, convert the image position coordinates of the video target in the monitoring area into Mercator coordinates through the relationship matrix H, convert the Mercator coordinates of the video target into longitude and latitude, record the longitude and latitude tracks of the video target and the longitude and latitude tracks of the radar target, and associate the longitude and latitude tracks of the video target and the longitude and latitude tracks of the radar target.
[0056] Specifically, pictures of ships in the scene are collected, annotated and trained with the YOLOV4 algorithm model, the YOLOV4 algorithm model is used to detect video targets in the monitoring area, and the KCF algorithm is used to track the video targets; the connected domain analysis algorithm is used to detect radar targets in the radar sector, and the Kalman tracking algorithm is used to track radar targets in the radar sector.
[0057] The latitude and longitude tracks of the video target are associated with the latitude and longitude tracks of the radar target. The association method is as follows:
[0058] 1. For each radar target tracked by the radar, the nearest neighbor method is used to select the video target in the monitoring area as the pre-associated object, and the distance between each radar target and the pre-associated object is recorded.
[0059] Here we assume that the earth is a spherical model, and the distance S between the two is calculated as follows:
[0060]
[0061] Among them, Lng1 Lat1 represents the longitude and latitude of the radar target, Lng2 Lat2 represents the longitude and latitude of the traversal point; α=Lat1-Lat2 is the difference between the latitudes of the two points; β=Lng1-Lng2 is the difference between the longitudes of the two points; 6378.137 is the equatorial radius of the earth, in kilometers.
[0062] 2. Calculate the ship heading and ship speed of each video target through the trajectory of the video target.
[0063] In this embodiment, the heading here is based on the true north as the reference direction, and the video track points of the ship are recorded continuously for 5 seconds at intervals of 0.1 seconds. The coordinate form of the track is the pixel coordinate (u, v). Among them, u represents the pixel coordinate with the upper left vertex of the image as the 0 point and the right direction of the image as the positive direction; and v represents the pixel coordinate with the upper left vertex of the image as the 0 point and the right direction of the image as the downward direction.
[0064] 3. Set the probability threshold, and calculate the probability value of the ship distance difference, the probability value of the ship heading difference, and the probability value of the ship speed difference according to the ship distance difference, ship heading difference, and ship speed difference between the video target and the radar target, and calculate the joint probability value.
[0065] The ship distance difference and the probability value P conform to the following formula:
[0066]
[0067] in, is the ship distance difference, which conforms to the distribution of σ=7;
[0068] The ship heading difference and the probability value P conform to the following formula:
[0069]
[0070] in, is the ship heading difference, which conforms to the distribution of σ=20;
[0071] The ship speed difference and the probability value P conform to the following formula:
[0072]
[0073] in, is the speed difference of the ships, and the speed difference of the ships conforms to the distribution of σ=1.5.
[0074] The joint probability value is the product of the three probability values: the probability value of the ship distance difference, the probability value of the ship heading difference, and the probability value of the ship speed difference.
[0075] The probability threshold is selected as any value between 0.3 and 0.45. The differences of the three parameters are: ship distance difference, ship heading difference, and ship speed difference. The distance difference is calculated in the same way as in the method 1 above; the heading difference is the absolute value of the ship heading measured in the radar system minus the target heading calculated after video conversion; the speed difference is the absolute value of the ship speed measured in the radar system minus the target speed calculated after video conversion.
[0076] 4. Cancel the pre-association relationship whose joint probability value is lower than the set probability threshold, and determine the association relationship whose joint probability value is higher than the set probability threshold.
[0077] In this embodiment, the threshold is preferably set to 0.4. Assuming that the ship distance difference between the target converted from the video and the radar target is 6, the ship heading difference is 12°, and the ship speed difference is 1.0, the calculated probabilities are: 0.693, 0.835, and 0.801 respectively; the joint probability value is the product of the three probabilities 0.463; if the joint probability value is higher than the set probability threshold 0.4, the association relationship between the two targets is determined.
[0078] like Figure 4The diagram shows the probability distribution of the speed difference. The curve in the figure is the probability distribution curve when σ=1.5. The mark point in the figure indicates that when the speed difference between the two is 1, the probability of the association between the two is 0.801 according to the distribution curve.
[0079] Step 4: Predict the radar target fusion occurrence node. First, set the time interval, calculate the average speed of each radar target within the set time interval, and predict its position T seconds later. Traverse the predicted positions and mark the current two radar targets when the distance between the two radar targets is smaller than the echo target size.
[0080] Specifically, a time interval is set, the average speed of the radar target within the interval is calculated, and its position is predicted 3 seconds later; the average speed is calculated by taking the total displacement of the ship's track points in the past 3 seconds and dividing it by time. The predicted positions are traversed, and when the distance between two radar targets is less than the maximum distance from the center of gravity of their echoes to the edge, they are marked. Figure 5 As shown, the schematic diagram of radar fusion and separation process.
[0081] Step 5: Video target sampling. The video target corresponding to the marked radar target in the monitoring area is identified through the association relationship between the video target and the radar target, the ship image corresponding to the video target in the monitoring area is extracted, and the ID (Identity document) of the ship image corresponding to the video target is recorded; after waiting for several seconds (preferably 5 seconds), when one or more of the two or more radar targets marked above loses tracking, it is determined that the radar targets have been fused, otherwise the marking of the marked radar target is cancelled and the process returns to step 4.
[0082] Step six, detect the radar target separation node. When a new radar target appears around a single tracked radar target in the overlapping area of the radar sector and the monitoring area, and the new radar target is re-associated with the video target, it is judged as a radar target separation after fusion.
[0083] Step seven, traverse N video targets judged as radar target fusion situations and re-associate video targets judged as radar target fusion and separation situations to perform image feature cross-comparison, select the best correlation item, and re-associate the radar target separated after fusion with the best correlation item.
[0084] Specifically, traversing N video targets judged to be radar target fused and video targets judged to be radar target fused and separated to perform image feature cross-comparison, including:
[0085] (1) Whenever the new radar target in step 6 is reassociated with the video target, the ship image of the video target in the video monitoring area is extracted and recorded as Figure 1, extract the HOG feature of the ship image and record it as feature 1. HOG feature is the Histogram of Oriented Gradient feature, which is a feature descriptor used for object detection in computer vision and image processing.
[0086] (2) Unify the ship images of the video targets extracted in step 5 into Figure 1 The HOG features of the ship image of each video target are extracted respectively, recorded as feature group 2, and the HOG features of the ship image of each video target in feature group 2 are compared with feature 1 in turn to calculate the hanmming distance between the features.
[0087] (3) The video target with the closest hamming distance that is judged to be a radar target that has been fused is selected as the optimal associated item, and the ID of the video target currently associated with the radar target separated after fusion with the ID selected as the optimal associated item in feature group 2 is marked as the ID before fusion, so as to achieve the purpose of separating and re-associating the optimal associated item after the radar target is fused.
[0088] Embodiment 2:
[0089] This embodiment provides a VTS radar target solution fusion system based on surveillance video assistance, which includes a calibration module, an association module, a prediction fusion generation module, a prediction fusion generation module, a video target image sampling module, a prediction target separation module and an optimal correlation item association module. The specific functions of each module are as follows:
[0090] A calibration module is used to calibrate the monitoring area and the radar sector, select multiple video targets in the monitoring area screen, read the image position coordinates of the video targets, select radar targets corresponding to the multiple video targets in the radar sector, read the longitude and latitude coordinates of the image position of the radar target, and convert the longitude and latitude coordinates of the image position of the radar target into Mercator coordinates;
[0091] The association module is used to solve the projection relationship between the coordinates of the monitoring area and the coordinates of the radar sector, calculate the relationship matrix H between the image position coordinates of the video target in the monitoring area and the Mercator coordinates of the radar target in the radar sector by the least square method; determine the association relationship between the video target and the radar target, respectively detect and track the video target in the monitoring area and the radar target in the radar sector, convert the image position coordinates of the video target in the monitoring area into Mercator coordinates by the relationship matrix H, convert the Mercator coordinates of the video target into longitude and latitude, record the longitude and latitude tracks of the video target and the longitude and latitude tracks of the radar target, and associate the longitude and latitude tracks of the video target and the longitude and latitude tracks of the radar target.
[0092] The prediction fusion occurrence module is used to predict the radar target fusion occurrence node. First, a time interval is set, the average speed of each radar target within the set time interval is calculated, and its position after T seconds is predicted; the predicted calculated positions are traversed, and when the distance between two radar targets is less than the echo target size, the current two radar targets are marked.
[0093] The video target image sampling module is used for video target sampling. The video target corresponding to the marked radar target in the monitoring area is identified through the association relationship between the video target and the radar target, the ship image corresponding to the video target in the monitoring area is extracted, and the ID of the ship image corresponding to the video target is recorded; after waiting for several seconds, when one or more radar target tracking of the marked radar target is lost, it is determined that the radar target has been fused, otherwise the marking of the marked radar target is cancelled and the process returns to step 4.
[0094] The predicted target separation module is used to detect radar target separation nodes. When a new radar target appears around a single tracked radar target in the overlapping area of the radar sector and the monitoring area, and the new radar target is re-associated with the video target, it is judged that the radar target is separated after fusion.
[0095] The optimal correlation item association module traverses N video targets judged as radar targets that have been fused and cross-compares the image features of video targets judged as radar targets that have been separated after fusion, selects the optimal correlation item, and re-associates the radar targets separated after fusion with the optimal correlation item.
[0096] Embodiment 3:
[0097] This embodiment provides a computer device, which may be a server, a computer, etc., and includes a processor, a memory, an input device, a display, and a network interface connected through a system bus. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the processor executes the computer program stored in the memory, the VTS radar target solution fusion method based on surveillance video assistance of the above embodiment 1 is implemented as follows:
[0098] Step 1: calibrate the monitoring area and the radar sector, select multiple video targets in the monitoring area screen, read the image position coordinates of the video targets, select radar targets corresponding to the multiple video targets in the radar sector, read the longitude and latitude coordinates of the image position of the radar target, and convert the longitude and latitude coordinates of the image position of the radar target into Mercator coordinates;
[0099] Step 2: Solve the projection relationship between the monitoring area coordinates and the radar sector coordinates, and calculate the relationship matrix H between the image position coordinates of the video target in the monitoring area and the Mercator coordinates of the radar target in the radar sector by the least squares method;
[0100] Step 3, determining the association relationship between the video target and the radar target, detecting and tracking the video target in the monitoring area and the radar target in the radar sector respectively, converting the image position coordinates of the video target in the monitoring area into Mercator coordinates through the relationship matrix H, converting the Mercator coordinates of the video target into longitude and latitude, recording the longitude and latitude tracks of the video target and the longitude and latitude tracks of the radar target, and associating the longitude and latitude tracks of the video target and the longitude and latitude tracks of the radar target;
[0101] Step 4: predict the radar target fusion occurrence node, first set the time interval, calculate the average speed of each radar target in the set time interval, and predict its position after T seconds; traverse the predicted calculated positions, and mark the current two radar targets when the distance between the two radar targets is less than the echo target size;
[0102] Step 5: Video target sampling. The video target corresponding to the marked radar target in the monitoring area is identified through the association relationship between the video target and the radar target, and the ship image corresponding to the video target in the monitoring area is extracted, and the ID of the ship image corresponding to the video target is recorded; after waiting for several seconds, when one or more radar target tracking of the marked radar target is lost, it is determined that the radar target has been fused, otherwise the marking of the marked radar target is cancelled and the process returns to step 4;
[0103] Step six, detect the radar target separation node. When a new radar target appears around a single tracked radar target in the overlapping area of the radar sector and the monitoring area, and the new radar target is re-associated with the video target, it is judged that the radar target is separated after fusion.
[0104] Step seven, traverse N video targets judged as radar target fusion situations and video targets judged as radar target fusion and separation situations to perform cross-comparison of image features, select the best correlation item, and re-associate the radar target separated after fusion with the best correlation item.
[0105] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.
Claims
1. A VTS radar target solution fusion method based on surveillance video assistance, characterized in that: The following steps are involved: Step 1: calibrate the monitoring area and the radar sector, select multiple video targets in the monitoring area screen, read the image position coordinates of the video targets, select radar targets corresponding to the multiple video targets in the radar sector, read the longitude and latitude coordinates of the image position of the radar target, and convert the longitude and latitude coordinates of the image position of the radar target into Mercator coordinates; Step 2: Solve the projection relationship between the monitoring area coordinates and the radar sector coordinates, and calculate the relationship matrix H between the image position coordinates of the video target in the monitoring area and the Mercator coordinates of the radar target in the radar sector by the least squares method; Step 3, determining the association relationship between the video target and the radar target, detecting and tracking the video target in the monitoring area and the radar target in the radar sector respectively, converting the image position coordinates of the video target in the monitoring area into Mercator coordinates through the relationship matrix H, converting the Mercator coordinates of the video target into longitude and latitude, recording the longitude and latitude tracks of the video target and the longitude and latitude tracks of the radar target, and associating the longitude and latitude tracks of the video target and the longitude and latitude tracks of the radar target; Step 4: predict the radar target fusion occurrence node, first set the time interval, calculate the average speed of each radar target in the set time interval, and predict its position after T seconds; traverse the predicted calculated positions, and mark the current two radar targets when the distance between the two radar targets is less than the echo target size; Step 5: video target sampling: identifying the video target in the monitoring area corresponding to the marked radar target through the association relationship between the video target and the radar target, extracting the ship image corresponding to the video target in the monitoring area, and recording the ID of the ship image corresponding to the video target; After waiting for several seconds, when one or more of the above-marked radar targets loses tracking, it is determined that the radar targets have been merged, otherwise the marking of the marked radar targets is cancelled and the process returns to step 4; Step 6: Detect radar target separation nodes. When a new radar target appears around a single tracked radar target in the overlapping area of the radar sector and the monitoring area, and the new radar target is re-associated with the video target, it is determined that the radar target is separated after fusion. Step seven, traverse N video targets judged as radar target fusion situations and video targets judged as radar target fusion and separation situations to perform cross-comparison of image features, select the best correlation item, and re-associate the radar target separated after fusion with the best correlation item.
2. The VTS radar target solution fusion method based on monitoring video assistance according to claim 1 is characterized in that: The image position coordinates of the video target are the coordinates of the midpoint of the bottom of the image of the video target, and the image position coordinate set of the video target is recorded as P camera , represented by (u,v,1) T The longitude and latitude coordinates of the image position of the radar target are the longitude and latitude coordinates of the image center of gravity of the radar target, and the Mercator coordinate set of the radar target is recorded as P radar , represented by (x,y,1) T .
3. The VTS radar target solution fusion method based on monitoring video assistance according to claim 2 is characterized in that: The mapping relationship between the image position coordinates of the video target in the monitoring area and the Mercator coordinates of the radar target in the radar sector is as follows: P radar =HP camera Where H is a 3×3 non-singular matrix.
4. The VTS radar target solution fusion method based on monitoring video assistance according to claim 1 is characterized in that: The method detects and tracks the video targets in the monitoring area and the radar targets in the radar sector respectively, including: collecting pictures of ships in the scene, marking and training the YOLOV4 algorithm model, using the YOLOV4 algorithm model to detect the video targets in the monitoring area, and using the KCF algorithm to track the video targets; using the connected domain analysis algorithm to detect the radar targets in the radar sector, and using the Kalman tracking algorithm to track the radar targets in the radar sector.
5. The VTS radar target solution fusion method based on monitoring video assistance according to claim 4 is characterized in that: The associating the longitude and latitude tracks of the video target with the longitude and latitude tracks of the radar target includes: For each radar target tracked by the radar, the nearest neighbor method is used to select the video target in the monitoring area as the pre-associated object, and the distance between each radar target and the pre-associated object is recorded; Calculate the ship heading and ship speed of each video target through the trajectory of the video target; Set the probability threshold, and calculate the ship distance difference probability value, ship heading difference probability value, ship speed difference probability value, and calculate the joint probability value according to the ship distance difference, ship heading difference, and ship speed difference between the video target and the radar target. The pre-association relationship whose joint probability value is lower than the set probability threshold is cancelled, and the association relationship whose joint probability value is higher than the set probability threshold is determined.
6. The VTS radar target solution fusion method based on monitoring video assistance according to claim 5 is characterized in that: The ship distance difference and the probability value P conform to the following formula: in, is the ship distance difference, which conforms to the distribution of σ=7; The ship heading difference and the probability value P conform to the following formula: in, is the ship heading difference, which conforms to the distribution of σ=20; The ship speed difference and the probability value P conform to the following formula: in, is the speed difference of the ships, and the speed difference of the ships conforms to the distribution of σ=1.
5.
7. The VTS radar target solution fusion method based on monitoring video assistance according to claim 6 is characterized in that: The joint probability value is the product of three probability values: the probability value of ship distance difference, the probability value of ship heading difference, and the probability value of ship speed difference; the probability threshold is any value between 0.3 and 0.
45.
8. The VTS radar target solution fusion method based on monitoring video assistance according to claim 1 is characterized in that: The traversing of N video targets judged to be radar target fused and the video targets judged to be radar target fused and separated to perform image feature cross-comparison, comprises: Whenever the new radar target in step 6 is re-associated with the video target, the ship image of the video target in the monitoring area is extracted and recorded as Figure 1, and the HOG feature of the ship image is extracted and recorded as feature 1; The ship images of the video targets judged as radar targets that have been fused are unified into the size of Figure 1, the HOG features of the ship images of each video target are extracted respectively, the HOG features of the ship images of each video target are compared with feature 1 in turn, and the hanmming distance between the features is calculated; The video target with the closest hamming distance that is judged to be a radar target that has been fused is selected as the optimal associated item, and the ID of the video target currently associated with the radar target separated after fusion is replaced by the ID of the optimal associated item.
9. The VTS radar target solution fusion system based on surveillance video assistance is characterized by: The system comprises: A calibration module is used to calibrate the monitoring area and the radar sector, select multiple video targets in the monitoring area screen, read the image position coordinates of the video targets, select radar targets corresponding to the multiple video targets in the radar sector, read the longitude and latitude coordinates of the image position of the radar target, and convert the longitude and latitude coordinates of the image position of the radar target into Mercator coordinates; The association module is used to solve the projection relationship between the coordinates of the monitoring area and the coordinates of the radar sector, and calculate the relationship matrix H between the image position coordinates of the video target in the monitoring area and the Mercator coordinates of the radar target in the radar sector by the least square method; determine the association relationship between the video target and the radar target, respectively detect and track the video target in the monitoring area and the radar target in the radar sector, convert the image position coordinates of the video target in the monitoring area into Mercator coordinates by the relationship matrix H, convert the Mercator coordinates of the video target into longitude and latitude, record the longitude and latitude tracks of the video target and the longitude and latitude tracks of the radar target, and associate the longitude and latitude tracks of the video target with the longitude and latitude tracks of the radar target; The prediction fusion occurrence module is used to predict the radar target fusion occurrence node. First, a time interval is set, the average speed of each radar target in the set time interval is calculated, and its position after T seconds is predicted; the predicted calculated positions are traversed, and when the distance between two radar targets is less than the echo target size, the current two radar targets are marked; The video target image sampling module is used for video target sampling. The video target corresponding to the marked radar target in the monitoring area is identified through the association relationship between the video target and the radar target, the ship image corresponding to the video target in the monitoring area is extracted, and the ID of the ship image corresponding to the video target is recorded; after waiting for several seconds, when one or more radar target tracking of the marked radar target is lost, it is determined that the radar target has been fused, otherwise the mark of the marked radar target is cancelled and the process returns to step 4; The predicted target separation module is used to detect radar target separation nodes. When a new radar target appears around a single tracked radar target in the overlapping area of the radar sector and the monitoring area, and the new radar target is re-associated with the video target, it is judged that the radar target is separated after fusion; The optimal correlation item association module traverses N video targets judged as radar targets that have been fused and cross-compares the image features of video targets judged as radar targets that have been separated after fusion, selects the optimal correlation item, and re-associates the radar targets separated after fusion with the optimal correlation item.
10. A computer device comprising a processor and a memory for storing a program executable by the processor, characterized in that: When the processor executes the program stored in the memory, the VTS radar target de-fusion method based on surveillance video assistance described in any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
Target tracking method for marine navigation radar ARPA (Auto Radar Plotting Aids)
CN102890274A
Echo fused target track association method based on track state estimation
CN105807280A
