A method and device for associating drone video with radar tracks of marine targets

Abstract

The present application belongs to the field of radar track technology, and specifically discloses a method and device for associating drone video with radar track of marine target, the method comprising: obtaining drone video and radar track of marine target to be associated, identifying video target to be detected in drone video; generating video information sequence and radar track sequence within a time period; judging whether the marine target is within the camera field of view according to relative position information; solving the spatial position of the pixel coordinates of the video target according to the video depth; determining the association result of radar track point according to the association similarity and the association threshold; determining the association success rate of video information sequence and radar track sequence according to the association results of all radar track points, and determining the association information of video target and radar track according to the association success rate and the association similarity of the successfully associated radar track points. Through the present application, the accuracy and reliability of target identification can be improved.

Description

Technical Field

[0001] The present application belongs to the field of radar track technology, and more specifically, to a method and device for associating drone videos with radar tracks of marine targets. Background Art

[0002] Among various detection methods, radar can detect and track targets at a long distance and grasp their position and movement information, but it cannot grasp the identity information of the target; imaging sensors can directly obtain images or videos of targets and accurately grasp their identity information, but the efficiency of searching and tracking targets in a large range is low. Therefore, drones can be used to shoot videos of targets detected by radar, and the videos and radar tracks can be automatically associated to quickly identify the target.

[0003] Currently, in the field of multimodal data fusion, the association between image and video data and sequence data is mainly concentrated on the association between image and video and text or voice, which is quite different from the scene of the target's spatiotemporal position sequence. If the real three-dimensional position of the target is extracted from the video first and then associated with the radar track, it is necessary to rely on additional means such as binocular cameras or depth cameras to achieve a certain degree of accuracy and reliability, which is costly. In summary, it is difficult for existing technologies to achieve the association between video and radar tracks, which in turn results in insufficient accuracy and reliability in target identification. Summary of the invention

[0004] In view of the defects of the prior art, the purpose of this application is to provide a method and device for associating drone video with radar tracks of maritime targets, aiming to solve the problem that it is difficult to associate video and radar tracks using the prior art, resulting in insufficient accuracy and reliability in target identity recognition.

[0005] In a first aspect, the present application provides a method for associating a drone video with a radar track of a marine target, comprising:

[0006] Acquire the drone video and the radar track of the marine target to be associated, and identify the video target to be detected in the drone video;

[0007] Intercepting a time period in which the video target and the radar track exist simultaneously, and generating a video information sequence and a radar track sequence in the time period;

[0008] Obtaining relative position information of the radar track point position and the drone camera position, and judging whether the marine target is within the camera field of view according to the relative position information;

[0009] The video depth and the projection position of the radar track point on the video plane are estimated by combining the relative distance between the radar track point to be associated and the drone camera, and the spatial position of the video target pixel coordinates is solved according to the video depth;

[0010] Calculating the correlation similarity between the projection position and the spatial position according to the error coefficient, and determining the correlation result of the radar track point according to the correlation similarity and the correlation threshold;

[0011] The association success rate of the video information sequence and the radar track sequence is determined according to the association results of all radar track points, and the association information of the video target and the radar track is determined according to the association success rate and the association similarity of the successfully associated radar track points.

[0012] Optionally, capturing a time period in which the video target and the radar track to be associated exist simultaneously, and generating a video information sequence and a radar track sequence in the time period, comprises:

[0013] Intercepting a time period in which the video target and the radar track to be associated exist simultaneously, and determining whether the time period is greater than a time threshold;

[0014] When the time period is greater than the time threshold, key frames in the drone video are extracted according to the target period, the current time of each key frame, the video target pixel position, and the camera position and posture information closest to the key frame time are determined, and the video information sequence is obtained;

[0015] For the current moment of each key frame in the video sequence, the position of the radar track point corresponding to the current moment is calculated by using the interpolation method, and the radar track sequence is composed according to the positions of the radar track points.

[0016] The embodiment of the present application can ensure that the associated data is synchronized in time by capturing the time period when the video target and the radar track exist at the same time, thereby reducing association errors caused by time mismatch.

[0017] Extracting key frames from drone videos can focus on information at important moments, reduce redundant data, and improve the efficiency and accuracy of subsequent video and trajectory associations.

[0018] Optionally, obtaining relative position information of the radar track point position and the drone camera position, and judging the correlation between the video information sequence and the radar track sequence according to the relative position information, includes:

[0019] Calculate the horizontal distance, azimuth and elevation of the camera position at the time corresponding to the radar track point position; the azimuth is measured clockwise with due north as the reference point, and the elevation is measured downward with the horizontal direction as the reference point;

[0020] Obtaining relative position information of the radar track point position and the drone camera position according to the horizontal distance, azimuth and elevation angle;

[0021] The horizontal orientation and pitch angle of the camera at the corresponding moment of the radar track point position are obtained, and the relative position information and the camera field of view angle are combined to determine whether the sea target is within the field of view of the drone camera, and obtain the initial correlation result between the video information sequence and the radar track sequence.

[0022] This application calculates the horizontal distance, azimuth and elevation between the radar track point and the camera position, which helps to accurately understand the relative position of the target in three-dimensional space. By determining the relative position information between the radar track point and the camera position, the radar data and the video data are effectively associated, and the fusion of multi-source data is strengthened. By judging whether the maritime target is within the field of view, the targets that are not within the monitoring range can be effectively filtered out, reducing the redundancy of subsequent processing data, thereby improving the accuracy of target recognition.

[0023] Optionally, estimating the video depth and the projection position of the radar track point on the video plane in combination with the relative distance between the radar track point to be associated and the drone camera, and solving the spatial position of the video target pixel coordinates according to the video depth, including:

[0024] Obtaining the video horizontal depth of the radar track and the X-coordinate projection of the radar track point on the video plane according to the horizontal distance, the camera horizontal orientation and the azimuth;

[0025] The video horizontal depth is combined with the camera position and the camera pitch angle to obtain the video depth and the Y coordinate projection of the radar track point on the video plane;

[0026] Determine the projection position according to the X-coordinate projection and the Y-coordinate projection;

[0027] The pixel coordinates of the video target are determined, and the spatial position of the video target is calculated according to the video depth, the pixel coordinates, and the camera intrinsic parameters.

[0028] Optionally, it also includes:

[0029] Obtaining the internal parameter information and sensor size of the drone camera; the internal parameter information includes focal length, principal point coordinates and pixel size, and the sensor size includes sensor width and sensor height;

[0030] The pixel focal length and field of view angle of the drone camera are determined based on the intrinsic reference information and the sensor size.

[0031] Optionally, determining association information between the video target and the radar track according to association similarity of successfully associated radar track points includes:

[0032] Determine the average correlation similarity according to the correlation similarity of each successfully correlated radar track point;

[0033] Determine the number of radar tracks having an association success rate greater than a success rate threshold;

[0034] If the number is 0, it is determined that there is no radar track successfully associated with the current video target;

[0035] If the number is 1, it is determined that the current radar track is successfully associated with the current video target;

[0036] If the number is greater than 1, it is determined that the radar track with the largest average correlation similarity is successfully associated with the current video target.

[0037] Optionally, it also includes:

[0038] If there is only one video target in the drone video, all radar tracks to be associated are traversed to associate with the video target;

[0039] If there are at least two video targets in the drone video, radar track association is performed on each of the video targets.

[0040] In a second aspect, the present application also provides a device for associating drone video with a radar track of a marine target, comprising:

[0041] An identification module, used to obtain the radar track of the drone video and the marine target to be associated, and identify the video target to be detected in the drone video;

[0042] A sequence generation module, used to capture a time period in which the video target and the radar track exist simultaneously, and generate a video information sequence and a radar track sequence within the time period;

[0043] A preliminary judgment module is used to obtain the relative position information of the radar track point position and the drone camera position, and judge whether the marine target is within the camera field of view according to the relative position information;

[0044] A position calculation module, used to estimate the video depth and the projection position of the radar track point on the video plane in combination with the relative distance between the radar track point to be associated and the drone camera, and calculate the spatial position of the video target pixel coordinates according to the video depth;

[0045] A similarity module, used for calculating the correlation similarity between the projection position and the spatial position according to the error coefficient, and determining the correlation result of the radar track point according to the correlation similarity and the correlation threshold;

[0046] The association decision module is used to determine the association success rate of the video information sequence and the radar track sequence according to the association results of all radar track points, and determine the association information of the video target and the radar track according to the association success rate and the association similarity of the successfully associated radar track points.

[0047] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0048] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0049] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0050] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0051] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:

[0052] (1) This application combines drone video and radar data and uses technical means such as spatial position, depth information and error compensation to achieve the association between video information sequence and radar track sequence, thereby efficiently realizing the identity of the target and improving the accuracy and reliability of target detection.

[0053] (2) This application can ensure that the same target is analyzed by capturing the time period when the video target and the radar track exist at the same time, reducing the error caused by time difference. By obtaining the relative position information of the radar track point position and the drone camera position, it can be determined whether the target is within the camera's field of view. The accuracy of target recognition can be effectively improved through spatial information fusion.

[0054] (3) The video depth is estimated based on the relative distance between the radar track point and the drone camera, and the spatial position of the target is solved. There is no need to rely on additional means such as binocular cameras or depth cameras, thus reducing costs. The correlation similarity is calculated based on the projection position, spatial position and error coefficient of the target. The accuracy of target detection and tracking can be improved through refined calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is one of the flow diagrams of a method for associating a drone video with a radar track of a marine target provided in an embodiment of the present application;

[0056] Figure 2It is a top view of the video target position solution provided by an embodiment of the present application;

[0057] Figure 3 It is a side view of the video target position solution provided by an embodiment of the present application;

[0058] Figure 4 This is a second flow chart of a method for associating a drone video with a radar track of a marine target provided in an embodiment of the present application;

[0059] Figure 5 It is a structural schematic diagram of a device for associating drone video with radar tracks of marine targets provided in an embodiment of the present application;

[0060] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0063] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0064] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0065] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0066] Next, the technical solutions provided in the embodiments of the present application are introduced.

[0067] Reference Figure 1 The present application provides a method for associating drone video with radar tracks of marine targets, comprising:

[0068] S101. Obtain the radar track of the drone video and the marine target to be associated, and identify the video target to be detected in the drone video;

[0069] S102. Intercept the time period in which the video target and the radar track exist at the same time, and generate a video information sequence and a radar track sequence within the time period;

[0070] S103. Obtain the relative position information of the radar track point position and the drone camera position, and determine whether the sea target is within the camera field of view based on the relative position information;

[0071] S104. Estimate the video depth and the projection position of the radar track point on the video plane in combination with the relative distance between the radar track point to be associated and the drone camera, and calculate the spatial position of the video target pixel coordinates according to the video depth;

[0072] S105. Calculate the correlation similarity between the projection position and the spatial position according to the error coefficient, and determine the correlation result of the radar track point according to the correlation similarity and the correlation threshold;

[0073] S106. Determine the association success rate of the video information sequence and the radar track sequence according to the association results of all radar track points, and determine the association information of the video target and the radar track according to the association success rate and the association similarity of the successfully associated radar track points.

[0074] First, this embodiment collects video data taken by the drone and the corresponding radar tracks of the marine targets. These data usually include the timestamp, longitude and latitude, track points and other information recorded by the radar and the video frames.

[0075] In the drone video, computer vision technology (such as target detection algorithm) is used to identify the marine targets to be detected, and deep learning models (such as YOLO, R-CNN, etc.) can be used to extract the targets in the video frames.

[0076] According to the recognition result of the drone video, if there is only one video target in the drone video, all radar tracks to be associated are traversed and associated with the video target; if there are at least two video targets in the drone video, radar track association is performed for each of the video targets.

[0077] By comparing the timestamps, the time period in which the target appears in the video is captured to ensure that there is corresponding radar track data in the time period, and the video information sequence and radar track sequence in the time period are generated.

[0078] According to the longitude and latitude of the radar track point and the longitude and latitude of the camera position, the relative position information (including distance, azimuth and elevation) of the two is calculated. In this embodiment, the Haversine formula and spherical trigonometry are used to calculate.

[0079] Furthermore, based on the calculated relative position information, it is determined whether the maritime target is within the field of view of the camera, and the field of view angle of the camera is used for judgment, that is, a preliminary judgment on the correlation between the video information sequence and the radar track sequence is achieved.

[0080] The video depth is calculated based on the relative distance between the radar track point and the drone camera. This step can use the camera internal parameters and triangulation to determine the projection position of the target on the video plane. Based on the projection position and video depth, the spatial position of the pixel coordinates of the video target is solved. This can be achieved by converting the pixel coordinates into the position in the real world coordinate system.

[0081] Furthermore, the correlation similarity between the video target and the radar track point is calculated based on the projection position, spatial position and error coefficient. The similarity calculation can use methods such as Euclidean distance and cosine similarity. According to the correlation similarity and the preset correlation threshold, the correlation result of the radar track point and the video target is judged. If the similarity is higher than the threshold, it is considered that the two are successfully associated.

[0082] Then, according to the association results of all radar track points, the association success rate of the video information sequence and the radar track sequence is calculated. At the same time, the similarity of the successfully associated radar track points is recorded, and finally, the association information of the video target and the radar track is finally determined by comprehensively considering the association success rate and the association similarity of the successfully associated radar track points.

[0083] This application can more accurately detect targets in videos by associating drone videos and radar tracks, using radar tracks to provide target location and trajectory information, and drone videos to provide visual features of targets. By combining drone videos and radar data, the association of video information sequences and radar track sequences is achieved using technical means such as spatial position, depth information, and error compensation, and then the identity of targets can be quickly determined, improving the accuracy and reliability of target detection.

[0084] Optionally, capturing a time period in which the video target and the radar track to be associated exist simultaneously, and generating a video information sequence and a radar track sequence in the time period, comprises:

[0085] Intercepting a time period in which the video target and the radar track to be associated exist simultaneously, and determining whether the time period is greater than a time threshold;

[0086] When the time period is greater than the time threshold, key frames in the drone video are extracted according to the target period, the current time of each key frame, the video target pixel position, and the camera position and posture information closest to the key frame time are determined, and the video information sequence is obtained;

[0087] For the current moment of each key frame in the video sequence, the position of the radar track point corresponding to the current moment is calculated by using the interpolation method, and the radar track sequence is composed according to the positions of the radar track points.

[0088] Specifically, this embodiment captures the time period in which the video target and the radar track to be associated exist at the same time, and if the time period is less than a certain time threshold T dur , then the radar track is removed from the radar track set to be associated with the current video target. In this embodiment, T is preferably dur Take 1 minute.

[0089] If the time period is greater than the time threshold T dur , then extract the key frame sequence in the intercepted video according to a certain period T, and in this embodiment, T is preferably 5 seconds. Select the current time of each frame, the video target pixel position, and the camera position and posture information closest to the frame time to form a video information sequence:

[0090] S video = {V 1 , V 2 , …, V i , …, V n}

[0091] Among them, V i is the video information point, including: the current time t i 、The pixel position t_x of the video target in the image i , t_y i , the camera's position (longitude, latitude and altitude) lon_c i , lat_c i , alt_c i and the horizontal orientation of the camera h_dir i and the pitch angle v_dir i ; as shown in the following formula:

[0092] V i ={t i , t_x i , t_y i , lon_c i , lat_c i, alt_c i , h_dir i , v_dir i}, i = 1, 2, ..., n

[0093] Corresponding to each moment t in the video information sequence i , use the interpolation method to calculate the position of the radar track point to be associated at the corresponding time. In this embodiment, the uniformly variable speed straight line model is preferably used for interpolation to form a track information sequence:

[0094] S radar ={T 1 , T 2 ,…,T i ,…,T n}

[0095] Among them, T i Radar track information point, including: radar data at the corresponding time in the video and the longitude and latitude of the radar point lon_t i , lat_t i ; as shown in the following formula:

[0096] T i ={t i , lon_t i , lat_t i}, i = 1, 2, ..., n

[0097] By capturing the time period when the video target and the radar track exist at the same time, the embodiment of the present application can ensure that the associated data is synchronized in time, reducing the association errors caused by time mismatch. Extracting key frames from drone videos can focus on the information of important moments, reduce redundant data, and improve the efficiency and accuracy of subsequent video and track association.

[0098] Furthermore, the embodiments of the present application utilize the interpolation method to generate a smooth path between known radar track points, thereby providing more accurate track data and reducing track missing or discontinuity problems caused by low radar data sampling frequency.

[0099] Optionally, obtaining relative position information of the radar track point position and the drone camera position, and judging the correlation between the video information sequence and the radar track sequence according to the relative position information, includes:

[0100] Calculate the horizontal distance, azimuth and elevation of the camera position at the time corresponding to the radar track point position; the azimuth is measured clockwise with due north as the reference point, and the elevation is measured downward with the horizontal direction as the reference point;

[0101] Obtaining relative position information of the radar track point position and the drone camera position according to the horizontal distance, azimuth and elevation angle;

[0102] The horizontal orientation and pitch angle of the camera at the corresponding moment of the radar track point position are obtained, and the relative position information and the camera field of view angle are combined to determine whether the sea target is within the field of view of the drone camera, and obtain the initial correlation result between the video information sequence and the radar track sequence.

[0103] Specifically, in this embodiment, the radar track position (lon_t i , lat_t i ,0) relative to the camera position at that moment (lon_c i , lat_c i , alt_c i ) i 、Bear i and elevation angle elev i , where the distance is the horizontal distance between the two points, the azimuth is measured clockwise with the north direction as the reference point, and the elevation is measured downward with the horizontal direction as the reference point. The specific calculation process is shown in the following formula:

[0104]

[0105] elev i =arctan(alt_c i / dis i )

[0106] Among them, a i is the intermediate parameter; lon_c i , lat_c i , alt_c i are the longitude, latitude and altitude of the camera respectively; lon_t i , lat_t i are the longitude and latitude of the radar point respectively; R is the radius of the earth, which is 6371 km in this embodiment. i <0, then bear i =bear i +360°.

[0107] Determine whether the target is within the camera's field of view based on the camera's posture, that is, whether the following conditions are met:

[0108]

[0109] Among them, h_dir i is the horizontal direction of the camera, v_diri is the camera pitch angle, fov h is the horizontal field of view, fov v is the vertical field of view.

[0110] If it is satisfied, continue to execute the subsequent steps. If it is not satisfied, it is determined that the current point association fails.

[0111] This application calculates the horizontal distance, azimuth and elevation between the radar track point and the camera position, which helps to accurately understand the relative position of the target in three-dimensional space. By determining the relative position information between the radar track point and the camera position, the radar data and the video data are effectively associated, and the fusion of multi-source data is strengthened. By judging whether the maritime target is within the field of view, the targets that are not within the monitoring range can be effectively filtered out, reducing the redundancy of subsequent processing data, thereby improving the accuracy of target recognition.

[0112] Optionally, estimating the video depth and the projection position of the radar track point on the video plane in combination with the relative distance between the radar track point to be associated and the drone camera, and solving the spatial position of the video target pixel coordinates according to the video depth, including:

[0113] Obtaining the video horizontal depth of the radar track and the X-coordinate projection of the radar track point on the video plane according to the horizontal distance, the camera horizontal orientation and the azimuth;

[0114] The video horizontal depth is combined with the camera position and the camera pitch angle to obtain the video depth and the Y coordinate projection of the radar track point on the video plane;

[0115] Determine the projection position according to the X-coordinate projection and the Y-coordinate projection;

[0116] The pixel coordinates of the video target are determined, and the spatial position of the video target is calculated according to the video depth, the pixel coordinates, and the camera intrinsic parameters.

[0117] Specifically, refer to Figure 2 and Figure 3 , Figure 2 It is the top view of the video target position solution. Figure 3 is the side view of the video target position solution. In this embodiment, the video depth Z of the current point is estimated by using the distance of the radar track to be associated with the camera. i , and calculate the projection position of the radar track point on the video plane (X_t i , Y_t i ).

[0118] Z_w i =dis i ·cos(bear i -h_dir i )

[0119] Z i = Z_w i / cos(v_dir i )+(alt_c i -Z_w i tan(v_dir i ))·sin(v_dir i )

[0120] X_t i =dis i ·sin(bear i -h_dir i )

[0121] Y_t i =(alt_c i -Z_w i tan(v_dir i ))·cos(v_dir i )

[0122] Among them, Z i represents the video depth, Z_w i Indicates the horizontal projection of the video depth, X_t i Indicates the X-coordinate projection of the radar track point on the video plane, Y_t i Indicates the Y coordinate projection of the radar track point on the video plane.

[0123] According to the video depth estimation result, the spatial position (X_c i , Y_c i ).

[0124] X_c i =(t_x i -c x )·Z i / f x

[0125] Y_c i =(t_y i -c y )·Z i / f y

[0126] Among them, X_c i Indicates the X coordinate of the spatial position, Y_c i Indicates the Y coordinate of the spatial position, (c x , c y ) is the principal point coordinate of the drone camera internal parameter, (f x , f y) is the pixel focal length;

[0127] It should be noted that the drone camera internal parameters and sensor size are obtained through product specifications or manual calibration methods, including focal length, principal point coordinates, pixel size, sensor width and sensor height. The pixel focal length and field of view of the drone camera are determined based on the sensor size.

[0128] Then the pixel focal length (f x , f y ) and field of view (fov h , fov v ), where the horizontal pixel focal length f x =f / p x , vertical pixel focal length f y =f / p y , horizontal field of view fov h =2*arctan(w / 2 / f), vertical field of view fov v =2*arctan(h / 2 / f).

[0129] Furthermore, this embodiment also needs to obtain relevant parameters of the drone and relevant parameters of the radar track in advance. Specifically, during the drone video shooting process, the geographic coordinate position of the drone needs to be periodically recorded at the same time, including the current time, the drone's longitude lon_c, latitude lat_c and altitude alt_c, the camera's horizontal orientation h_dir and the camera's pitch angle v_dir. In this embodiment, the recording period is preferably no more than 1 second, the drone's geographic coordinates are taken as approximate camera positions, the camera's horizontal orientation is measured clockwise with the north direction as the reference point, and the camera's pitch angle is measured downward with the horizontal direction as the reference point.

[0130] The detection error of the radar, including the distance error, is obtained through the product specification or manual calibration method. dis and the azimuth error e bear When the radar tracks a target at sea, it is necessary to periodically record the position and movement information of the target, including the current time, longitude lon_t, latitude lat_t, speed v_t and heading course_t. In the method of this embodiment, the height of the radar track of the target at sea is taken as 0.

[0131] Furthermore, in this embodiment, after obtaining the projection position of the radar track point on the video plane and the spatial position of the video target, the correlation similarity s is calculated according to the radar detection error. i , as shown in the following formula:

[0132]

[0133] Where σ=max(e dis,c_dis·sin(e bear )), c_dis is a constant set by the algorithm, and in this embodiment, the value is 20km.

[0134] The correlation similarity exceeds a certain threshold T s The current point is determined to be successfully associated, otherwise the association fails. In this embodiment, the preferred similarity threshold is 0.65. After the current radar track point is executed, the next point in the sequence is selected to continue to cyclically execute the steps of preliminary association judgment and video target position solution.

[0135] Optionally, determining association information between the video target and the radar track according to association similarity of successfully associated radar track points includes:

[0136] Determine the average correlation similarity according to the correlation similarity of each successfully correlated radar track point;

[0137] Determine the number of radar tracks having an association success rate greater than a success rate threshold;

[0138] If the number is 0, it is determined that there is no radar track successfully associated with the current video target;

[0139] If the number is 1, it is determined that the current radar track is successfully associated with the current video target;

[0140] If the number is greater than 1, it is determined that the radar track with the largest average correlation similarity is successfully associated with the current video target.

[0141] Specifically, in this embodiment, the percentage rate of the points successfully associated in the video information sequence and the track information sequence to the total number of points and the average association similarity s of the associated points are counted. avg .

[0142] rate=cnt c / cnt all

[0143]

[0144] Where i = 1, 2, ..., n and satisfies s i >T s ,cnt all is the number of points in the sequence, cnt c is the number of points successfully associated in the sequence.

[0145] The percentage rate of successfully associated points in the overall similarity between the current video target and all radar tracks to be associated exceeds a certain threshold T. rateIf it is 0, it is determined that there is no radar track associated with the current video target; if it is 1, it is determined that the radar track is successfully associated with the current video target; if it is greater than 1, the average association similarity s is determined. avg The largest radar track is successfully associated with the current video target. rate The value is 0.8.

[0146] Reference Figure 4 The complete process of the embodiment of the present application includes the following steps:

[0147] S1. Data preparation

[0148] Obtain the drone camera's intrinsic parameters and camera sensor dimensions, including focal length, principal point coordinates, pixel size, sensor width, and sensor height, through product specifications or manual calibration methods, and calculate the camera's pixel focal length and field of view angle.

[0149] During the video shooting process of a drone, it is necessary to periodically record the geographic coordinate position of the drone, the horizontal orientation of the camera, and the pitch angle of the camera. The method of the present invention takes the geographic coordinates of the drone as an approximation of the camera position.

[0150] The detection error of the radar is obtained through the product specification or manual calibration method. When the radar tracks a target at sea, it is necessary to periodically record the position and motion information of the target.

[0151] S2. Video object detection

[0152] Use YOLO, R-CNN and other methods to realize target detection in drone videos. If there is only one target in the video, traverse all radar tracks to be associated, perform association calculations from S3 to S6 on the video target and each radar track, and finally obtain the association calculation result through S7; if there are multiple targets in the video, perform the above operations for each video target respectively.

[0153] S3. Generate association calculation sequence

[0154] The time period when the video target and the radar track to be associated coexist is intercepted. If the time period is less than a certain threshold, the radar track is removed from the radar track set to be associated with the current video target; otherwise, a key frame sequence is extracted from the intercepted video at a certain period, and the central pixel coordinates of the rectangular frame of the video target of the key frame and the camera position and posture information closest to the key frame moment are selected to form a video information sequence. For each moment in the video information sequence, the position of the radar track point to be associated at the corresponding moment is calculated using the interpolation method to form a track information sequence.

[0155] The data pairs at corresponding moments in the video information sequence and the track information sequence are processed from S4 to S5 respectively.

[0156] S4. Rough judgment of association

[0157] According to the Haversine formula and spherical trigonometry, the distance, azimuth and elevation of the radar track position relative to the camera position at that moment are calculated. According to the camera posture, it is determined whether the target is within the camera field of view. If it is, S5 is executed. If not, it is determined that the current point association fails, and the next point in the sequence is selected to execute S4.

[0158] S5. Video target position calculation

[0159] The video depth of the current point is estimated by using the distance of the radar track to be associated with the camera, and the projection position of the radar track point on the video plane is calculated. The spatial position of the video target pixel coordinates is solved according to the video depth estimation result. And the association similarity is calculated according to the radar detection error. If the similarity exceeds a certain threshold, the current point is determined to be successfully associated, otherwise the association fails. Then the next point in the sequence is selected to execute S4.

[0160] S6. Overall similarity calculation

[0161] The percentage of successfully associated points in the video information sequence and the track information sequence as well as the average association similarity of the associated points are counted.

[0162] S7. Relevance Decision

[0163] The percentage of successfully associated points in the overall similarity between the current video target and all radar tracks to be associated is counted, and the number of points that exceed a certain threshold is counted. If the percentage is 0, it is determined that there is no radar track associated with the current video target; if the percentage is 1, it is determined that the radar track is successfully associated with the current video target; if the percentage is greater than 1, it is determined that the radar track with the largest average association similarity is successfully associated with the current video target.

[0164] Reference Figure 5 The present application also provides a device for associating drone video with marine target radar tracks, including:

[0165] The identification module 510 is used to obtain the radar track of the drone video and the marine target to be associated, and identify the video target to be detected in the drone video;

[0166] A sequence generation module 520, for capturing a time period in which the video target and the radar track exist simultaneously, and generating a video information sequence and a radar track sequence within the time period;

[0167] A preliminary determination module 530 is used to obtain relative position information between the radar track point position and the drone camera position, and determine whether the marine target is within the camera field of view according to the relative position information;

[0168] A position calculation module 540 is used to estimate the video depth and the projection position of the radar track point on the video plane in combination with the relative distance between the radar track point to be associated and the drone camera, and calculate the spatial position of the video target pixel coordinates according to the video depth;

[0169] A similarity module 550 is used to calculate the correlation similarity between the projection position and the spatial position according to the error coefficient, and determine the correlation result of the radar track point according to the correlation similarity and the correlation threshold;

[0170] The association decision module 560 is used to determine the association success rate of the video information sequence and the radar track sequence according to the association results of all radar track points, and determine the association information of the video target and the radar track according to the association success rate and the association similarity of the successfully associated radar track points.

[0171] It can be understood that the detailed functional implementation of each of the above-mentioned units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0172] Optionally, capturing a time period in which the video target and the radar track to be associated exist simultaneously, and generating a video information sequence and a radar track sequence in the time period, comprises:

[0173] Intercepting a time period in which the video target and the radar track to be associated exist simultaneously, and determining whether the time period is greater than a time threshold;

[0174] When the time period is greater than the time threshold, key frames in the drone video are extracted according to the target period, the current time of each key frame, the video target pixel position, and the camera position and posture information closest to the key frame time are determined, and the video information sequence is obtained;

[0175] For the current moment of each key frame in the video sequence, the position of the radar track point corresponding to the current moment is calculated by using the interpolation method, and the radar track sequence is composed according to the positions of the radar track points.

[0176] Optionally, obtaining relative position information of the radar track point position and the drone camera position, and judging the correlation between the video information sequence and the radar track sequence according to the relative position information, includes:

[0177] Calculate the horizontal distance, azimuth and elevation of the camera position at the time corresponding to the radar track point position; the azimuth is measured clockwise with due north as the reference point, and the elevation is measured downward with the horizontal direction as the reference point;

[0178] Obtaining relative position information of the radar track point position and the drone camera position according to the horizontal distance, azimuth and elevation angle;

[0179] The horizontal orientation and pitch angle of the camera at the corresponding moment of the radar track point position are obtained, and the relative position information and the camera field of view angle are combined to determine whether the sea target is within the field of view of the drone camera, and obtain the initial correlation result between the video information sequence and the radar track sequence.

[0180] Optionally, estimating the video depth and the projection position of the radar track point on the video plane in combination with the relative distance between the radar track point to be associated and the drone camera, and solving the spatial position of the video target pixel coordinates according to the video depth, including:

[0181] Obtaining the video horizontal depth of the radar track and the X-coordinate projection of the radar track point on the video plane according to the horizontal distance, the camera horizontal orientation and the azimuth;

[0182] The video horizontal depth is combined with the camera position and the camera pitch angle to obtain the video depth and the Y coordinate projection of the radar track point on the video plane;

[0183] Determine the projection position according to the X-coordinate projection and the Y-coordinate projection;

[0184] The pixel coordinates of the video target are determined, and the spatial position of the video target is calculated according to the video depth, the pixel coordinates, and the camera intrinsic parameters.

[0185] Optionally, a data preparation module is also included, for:

[0186] Obtaining the internal parameter information and sensor size of the drone camera; the internal parameter information includes focal length, principal point coordinates and pixel size, and the sensor size includes sensor width and sensor height;

[0187] The pixel focal length and field of view angle of the drone camera are determined based on the intrinsic reference information and the sensor size.

[0188] Optionally, determining association information between the video target and the radar track according to association similarity of successfully associated radar track points includes:

[0189] Determine the average correlation similarity according to the correlation similarity of each successfully correlated radar track point;

[0190] Determine the number of radar tracks having an association success rate greater than a success rate threshold;

[0191] If the number is 0, it is determined that there is no radar track successfully associated with the current video target;

[0192] If the number is 1, it is determined that the current radar track is successfully associated with the current video target;

[0193] If the number is greater than 1, it is determined that the radar track with the largest average correlation similarity is successfully associated with the current video target.

[0194] Optionally, a video target determination module is also included, which is used to:

[0195] If there is only one video target in the drone video, all radar tracks to be associated are traversed to associate with the video target;

[0196] If there are at least two video targets in the drone video, radar track association is performed on each of the video targets.

[0197] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0198] Reference Figure 6 Based on the method in the above embodiment, the embodiment of the present application provides an electronic device, which may include: a processor (processor) 610, a communication interface (CommunicationsInterface) 620, a memory (memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 630 to execute the method in the above embodiment.

[0199] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0200] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0201] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0202] It is understandable that the processor in the embodiment of the present application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0203] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0204] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0205] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0206] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for associating drone video with radar tracks of marine targets, characterized in that: include: Acquire the drone video and the radar track of the marine target to be associated, and identify the video target to be detected in the drone video; Intercepting a time period in which the video target and the radar track exist simultaneously, and generating a video information sequence and a radar track sequence in the time period; Obtaining relative position information of the radar track point position and the drone camera position, and judging whether the marine target is within the camera field of view according to the relative position information; The video depth and the projection position of the radar track point on the video plane are estimated by combining the relative distance between the radar track point to be associated and the drone camera, and the spatial position of the video target pixel coordinates is solved according to the video depth; Calculating the correlation similarity between the projection position and the spatial position according to the error coefficient, and determining the correlation result of the radar track point according to the correlation similarity and the correlation threshold; The association success rate of the video information sequence and the radar track sequence is determined according to the association results of all radar track points, and the association information of the video target and the radar track is determined according to the association success rate and the association similarity of the successfully associated radar track points.

2. The association method according to claim 1, characterized in that: Intercepting a time period in which the video target and the radar track to be associated exist simultaneously, and generating a video information sequence and a radar track sequence in the time period, including: Intercepting a time period in which the video target and the radar track to be associated exist simultaneously, and determining whether the time period is greater than a time threshold; When the time period is greater than the time threshold, key frames in the drone video are extracted according to the target period, the current time of each key frame, the video target pixel position, and the camera position and posture information closest to the key frame time are determined, and the video information sequence is obtained; For the current moment of each key frame in the video information sequence, the position of the radar track point corresponding to the current moment is calculated by using the interpolation method, and the radar track sequence is composed according to the positions of the radar track points.

3. The association method according to claim 1, characterized in that: Obtaining relative position information of the radar track point position and the drone camera position, and judging the correlation between the video information sequence and the radar track sequence according to the relative position information, including: Calculate the horizontal distance, azimuth and elevation of the camera position at the time corresponding to the radar track point position; the azimuth is measured clockwise with due north as the reference point, and the elevation is measured downward with the horizontal direction as the reference point; Obtaining relative position information of the radar track point position and the drone camera position according to the horizontal distance, azimuth and elevation angle; The horizontal orientation and pitch angle of the camera at the corresponding moment of the radar track point position are obtained, and the relative position information and the camera field of view angle are combined to determine whether the sea target is within the field of view of the drone camera, and obtain the initial correlation result between the video information sequence and the radar track sequence.

4. The association method according to claim 3, characterized in that: The video depth and the projection position of the radar track point on the video plane are estimated by combining the relative distance between the radar track point to be associated and the drone camera, and the spatial position of the video target pixel coordinates is solved according to the video depth, including: Obtaining the video horizontal depth of the radar track and the X-coordinate projection of the radar track point on the video plane according to the horizontal distance, the camera horizontal orientation and the azimuth; The video horizontal depth is combined with the camera position and the camera pitch angle to obtain the video depth and the Y coordinate projection of the radar track point on the video plane; Determine the projection position according to the X-coordinate projection and the Y-coordinate projection; The pixel coordinates of the video target are determined, and the spatial position of the video target is calculated according to the video depth, the pixel coordinates, and the camera intrinsic parameters.

5. The association method according to claim 1, characterized in that: Also includes: Obtaining the internal parameter information and sensor size of the drone camera; the internal parameter information includes focal length, principal point coordinates and pixel size, and the sensor size includes sensor width and sensor height; The pixel focal length and field of view angle of the drone camera are determined based on the intrinsic reference information and the sensor size.

6. The association method according to claim 1, characterized in that: The association information between the video target and the radar track is determined based on the association similarity of the successfully associated radar track points, including: Determine the average correlation similarity according to the correlation similarity of each successfully correlated radar track point; Determine the number of radar tracks having an association success rate greater than a success rate threshold; If the number is 0, it is determined that there is no radar track successfully associated with the current video target; If the number is 1, it is determined that the current radar track is successfully associated with the current video target; If the number is greater than 1, it is determined that the radar track with the largest average correlation similarity is successfully associated with the current video target.

7. The association method according to claim 1, characterized in that: Also includes: If there is only one video target in the drone video, all radar tracks to be associated are traversed to associate with the video target; If there are at least two video targets in the drone video, radar track association is performed on each of the video targets.

8. A device for associating drone video with radar tracks of marine targets, characterized in that: include: An identification module, used to obtain the radar track of the drone video and the marine target to be associated, and identify the video target to be detected in the drone video; A sequence generation module, used to capture a time period in which the video target and the radar track exist simultaneously, and generate a video information sequence and a radar track sequence within the time period; A preliminary judgment module is used to obtain the relative position information of the radar track point position and the drone camera position, and judge whether the marine target is within the camera field of view according to the relative position information; A position calculation module, used to estimate the video depth and the projection position of the radar track point on the video plane in combination with the relative distance between the radar track point to be associated and the drone camera, and calculate the spatial position of the video target pixel coordinates according to the video depth; A similarity module, used for calculating the correlation similarity between the projection position and the spatial position according to the error coefficient, and determining the correlation result of the radar track point according to the correlation similarity and the correlation threshold; The association decision module is used to determine the association success rate of the video information sequence and the radar track sequence according to the association results of all radar track points, and determine the association information of the video target and the radar track according to the association success rate and the association similarity of the successfully associated radar track points.

9. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 7.

Images