Radar-AIS ship tracking method and device based on visual feature optimization

Abstract

The present invention provides a radar-AIS ship tracking method and device based on visual feature optimization, relating to the field of target tracking technology. The method includes: guiding an optoelectronic turntable to the target ship's direction based on target information acquired by the radar-AIS; identifying a current image captured by the turntable to obtain a number of current ships; tracking the current ships and acquiring visual feature information and track information for each current ship; and determining a target ship from the current ships for tracking based on the target information, visual feature information, and track information. This solution achieves accurate identification and tracking of ship targets, providing stable and reliable tracking results.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology, in particular to the field of ship track tracking technology, and more particularly to a radar-AIS ship tracking method and device based on visual feature optimization. Background Art

[0002] The fundamental function of a vessel traffic management system is ship tracking, which relies heavily on maritime radar and AIS base stations. The transmission frequency of ship-borne AIS transceivers is variable, subject to delays and errors. Relying solely on radar-AIS information to guide the optoelectronic turntable to track a vessel, it can be difficult to identify a target among multiple vessels in dense traffic. Even if a target vessel is successfully located, the limited amount of radar-AIS information available makes it difficult to conduct further analysis and real-time tracking. Therefore, there is an urgent need for a radar-AIS ship tracking method and device based on visual feature optimization. Summary of the Invention

[0003] The present invention provides a radar-AIS ship tracking method and device based on visual feature optimization. The method realizes accurate identification and tracking of ship targets based on shipborne radar, providing a stable and reliable tracking effect.

[0004] In a first aspect, the present invention provides a radar-AIS ship tracking method based on visual feature optimization, comprising:

[0005] According to the target information of the target ship acquired by the radar-AIS, the optoelectronic turntable is guided to the direction of the target ship;

[0006] Identifying the current image collected by the photoelectric turntable to obtain several current ships;

[0007] Tracking the current ships and obtaining visual feature information and track information of each current ship;

[0008] The target ship is determined from the current ships for tracking based on the target information, the visual feature information and the track information.

[0009] Optionally, guiding the optoelectronic turntable to turn toward the direction of the target ship according to the target information of the target ship acquired by the radar-AIS includes:

[0010] Calculating the azimuth and pitch angles of the target ship based on the target longitude and latitude, target distance, and the longitude and latitude, and installation height of the photoelectric turntable included in the target information;

[0011] The photoelectric turntable is rotated according to the azimuth angle and the pitch angle so that the picture center of the camera on the photoelectric turntable coincides with the guidance azimuth center of the radar-AIS.

[0012] Optionally, the tracking of the current ships and obtaining visual feature information and track information of each current ship includes:

[0013] Using a multi-target tracking algorithm to correlate the current ships in the current images at different times, and determine the track information of each current ship; wherein the track information includes the position change direction and the position change rate;

[0014] Feature extraction is performed on the current image to obtain visual feature information of each current ship; wherein the visual feature information includes ship type, color composition, ship building position, number of ship floors, cargo type and direction.

[0015] Optionally, the identifying the current image collected by the photoelectric turntable to obtain several current ships includes:

[0016] Inputting the current image into an object detection model and outputting a predicted box marked with a ship;

[0017] The prediction frames are screened according to the size, confidence and overlap of the prediction frames to obtain a target prediction frame marked with the current ship.

[0018] Optionally, the filtering of the prediction frames according to the size, confidence, and overlap of the prediction frames to obtain a target prediction frame marked with the current ship includes:

[0019] After eliminating the prediction boxes corresponding to the confidence levels lower than a preset confidence threshold, the prediction box with the highest current confidence level is taken as the optimal box, and the intersection-over-union ratio between the optimal box and the first prediction box retained after the elimination is calculated;

[0020] Eliminating the first prediction frames corresponding to the IoU ratios greater than a first preset threshold, performing pairing to obtain matching pairs, and calculating similarity scores of the matching pairs, so as to filter the first prediction frames based on the similarity scores to obtain second prediction frames;

[0021] The second prediction frame is filtered according to the relative position relationship of the bow of the ship to obtain the target prediction frame.

[0022] Optionally, the target information includes target position change direction, target change rate, target ship type, target color composition, target ship building position, target ship floor number, target cargo type and target direction;

[0023] The visual feature information includes ship type, color composition, ship deck position, number of ship decks, cargo type and direction.

[0024] Optionally, the determining the target ship from the current ships for tracking based on the target information, the visual feature information, and the track information includes:

[0025] screening target track information having the same target position change direction and target position change rate from the track information including the position change direction and the position change rate, and determining the current ship corresponding to the target track information as the preliminarily screened ship;

[0026] Determining a comprehensive matching score for each of the initially screened ships by comparing the visual feature information with the target information;

[0027] The initially screened ship corresponding to the highest comprehensive matching score is determined as the target ship.

[0028] Optionally, determining a comprehensive matching score for each of the preliminarily screened ships by comparing the visual feature information with the target information includes:

[0029] Determining a first matching score between the target ship type and the ship type by comparison;

[0030] determining a second matching score between the target cargo category and the cargo category by comparison;

[0031] determining a third matching score between the target orientation and the orientation by comparison;

[0032] determining a fourth matching score between the target color composition and the color composition, the target building position and the building position, and the target number of ship floors and the number of ship floors by comparison;

[0033] The comprehensive matching score is determined according to the first matching score, the second matching score, the third matching score, and the fourth matching score.

[0034] In a second aspect, the present invention further provides a radar-AIS ship tracking device based on visual feature optimization, comprising:

[0035] A monitoring and guidance module is used to guide the optoelectronic turntable to the direction of the target ship based on the target information of the target ship obtained by the radar-AIS;

[0036] an identification module, configured to identify the current image collected by the photoelectric turntable to obtain a number of current ships;

[0037] The tracking module is used to track the current ships and obtain visual feature information and track information of each current ship; and determine the target ship from the current ships for tracking based on the target information, the visual feature information and the track information.

[0038] In a third aspect, the present invention further provides a computing device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements any of the above-mentioned radar-AIS ship tracking methods based on visual feature optimization.

[0039] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute any of the above-mentioned radar-AIS ship tracking methods based on visual feature optimization.

[0040] In a fifth aspect, an embodiment of the present invention further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in any first aspect of this specification.

[0041] The present invention provides a radar-AIS ship tracking method and device based on visual feature optimization. The method first uses the target information detected by the radar-AIS to preliminarily guide the optoelectronic turntable to the approximate direction of the target ship, and then calls the ship detection algorithm to locate and track the ships in the current image, obtain the position changes of each current ship in the picture, and send the image of the current ship that can be detected into the ship feature extraction network to obtain its visual characteristic information and track information. Then, the target information returned by the radar-AIS is matched with the visual feature information and track information to determine the target ship for continuous positioning and tracking. In this way, based on the radar-AIS guided optoelectronic turntable tracking, the present invention uses visual features to assist in finding the target ship to be tracked, thereby improving the effect of the fusion perception of radar, AIS and optoelectronics, realizing the accurate identification and tracking of ship targets, providing a stable and reliable tracking effect, and further strengthening the intelligent duty capability of basic coastal defense equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1This is a flow chart of a radar-AIS ship tracking method based on visual feature optimization provided by one embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of a method for detecting a target ship provided by one embodiment of the present invention;

[0045] Figure 3 This is a hardware architecture diagram of a computing device provided by one embodiment of the present invention;

[0046] Figure 4 The figure is a structural diagram of a radar-AIS ship tracking device based on visual feature optimization provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0048] Due to limitations in radar and AIS transmission intervals and the precision of the optoelectronic turntable, relying solely on radar-AIS to guide the optoelectronic turntable to track a vessel target cannot guarantee that the target vessel is precisely centered in the image. Therefore, in this scenario, if multiple vessels are simultaneously in the image, identifying the target vessel becomes difficult. Even if the target vessel is successfully identified, the limited amount of radar-AIS information makes it difficult for controllers to conduct further analysis of the target in a timely manner. Therefore, the present invention proposes a radar-AIS and optoelectronic fusion method to address this issue.

[0049] The concept of the present invention is described below. Figure 1 The embodiment of the present invention provides a radar-AIS ship tracking method based on visual feature optimization, comprising:

[0050] Step 100, guiding the optoelectronic turntable to the direction of the target ship according to the target information of the target ship acquired by the radar-AIS;

[0051] Step 102, identifying the current image collected by the photoelectric turntable to obtain several current ships;

[0052] Step 104: Track the current ships and obtain visual feature information and track information of each current ship;

[0053] Step 106: Determine a target ship from the current ships for tracking based on the target information, visual feature information, and track information.

[0054] In the present invention, the target information detected by the radar-AIS is first used to preliminarily guide the optoelectronic turntable to the approximate direction of the target ship, and then the ship detection algorithm is called to locate and track the ships in the current image, obtain the position changes of each current ship in the picture, and send the image of the current ship that can be detected into the ship feature extraction network to obtain its visual characteristic information and track information. Then, the target information returned by the radar-AIS is matched with the visual characteristic information and track information to determine the target ship for continuous positioning and tracking. In this way, based on the radar-AIS guided optoelectronic turntable tracking, the present invention uses visual features to assist in finding the target ship to be tracked, thereby improving the effect of the fusion perception of radar, AIS and optoelectronics, realizing the accurate identification and tracking of ship targets from multiple ships, providing a stable and reliable tracking effect, and further strengthening the intelligent duty capability of basic coastal defense equipment.

[0055] Described below Figure 1 How to perform the steps shown.

[0056] First, in step 100, according to the target information of the target ship acquired by the radar-AIS, the optoelectronic turntable is guided to the direction of the target ship, including:

[0057] The azimuth and pitch angles of the target ship are calculated based on the target information including the target longitude and latitude, target distance, and the longitude and latitude and installation height of the optoelectronic turntable;

[0058] The optoelectronic turntable is rotated according to the azimuth and elevation angles so that the image center of the camera on the optoelectronic turntable coincides with the guidance azimuth center of the radar-AIS.

[0059] It should be noted that AIS, short for Automatic Identification System, is a new aid-to-navigation system used for maritime safety and communication between ships and shore, and between ships. Radar-AIS is an integrated monitoring system that combines radar and Automatic Identification System (AIS) technologies. It primarily consists of radar, AIS equipment, and a data processing center. Radar is used to detect and locate targets at sea, while AIS exchanges information with surrounding vessels via the VHF band, providing detailed information about the target, such as the ship's name, location, and heading. A camera is installed on the optoelectronic turntable.

[0060] Specifically, for the selected target ship, the target longitude and latitude detected by the radar-AIS system, the target distance and the longitude and latitude of the optoelectronic turntable itself, and the installation height are used to calculate the approximate azimuth and pitch angle of the target ship; then the optoelectronic turntable is rotated according to this azimuth and pitch angle, keeping the center of the camera's picture coincident with the guidance azimuth center of the radar, and at the same time, the field of view angle of the camera is coarsely adjusted according to the target length included in the target information.

[0061] Figure 2 The schematic diagram of radar-AIS guiding the optoelectronic turntable to track the target ship is shown. In step 100, the target ship detected by the radar-AIS P Target latitude and longitude[ lat 1, lon 1] Target ship to detection equipment O (Photoelectric turntable) straight-line distance d , target reflection width information w 1. The above information combined with the detection equipment O’ Its own latitude and longitude [ lat 0, lon 0], Installation height h 0, the target ship can be calculated P Horizontal distance from the observation point d’ , relative azimuth θ and relative pitch angle β , the detection process is as follows Figure 2 As shown. The horizontal distance d’ It can be calculated using the following haversine formula:

[0062]

[0063] in, R is the radius of the Earth; a 、 c All are intermediate amounts;

[0064] Relative azimuth θ Determined by the following formula:

[0065]

[0066] Since the observation target is a ship, its altitude is set to 0 by default, so the relative pitch angle β for:

[0067]

[0068] In this way, the optoelectronic turntable follows the relative azimuth θ and relative pitch angle βRotate to keep the center of the camera image aligned with the center of the radar guidance azimuth.

[0069] In a preferred embodiment, the field of view angle of the camera is roughly adjusted according to the target length included in the target information, so that the target length detected by the radar-AIS occupies 1 / 4 of the camera's picture field of view.

[0070] Specifically, based on the target reflection width information of the target ship detected by radar w 1. Combined with straight-line distance d To roughly adjust the field of view of the photoelectric turntable camera. Assuming that the target ship is located in the center of the picture, the horizontal field of view corresponding to the horizontal axis of the picture where the target ship is located reflects the actual distance MN It can be estimated by the following formula:

[0071]

[0072] If the target occupies about 1 / 4 of the length and width of the screen, MN= 4 w 1, then the camera field of view γ Should be adjusted to:

[0073]

[0074] In this way, not only can the target ship be clearly observed in the picture, but it also has a certain field of view, which is convenient for subsequent processing of guidance errors.

[0075] In step 102, the current image collected by the photoelectric turntable is identified to obtain several current ships, including:

[0076] Input the current image into the object detection model and output the predicted box marked with the ship;

[0077] The prediction boxes are filtered according to their size, confidence, and overlap to obtain the target prediction box marked with the current ship.

[0078] Specifically, the current image with the coarsely adjusted optoelectronic turntable orientation and camera field of view is continuously input into the target detection model, and the position, size, and confidence information of all ships in the picture are obtained through real-time reasoning and processing.

[0079] In the present invention, a target detection model is obtained by training a ship dataset composed of a large number of real-world ship images based on the YOLOv5 network architecture. The algorithm inputs a 640*640 image, and outputs a prediction box after model inference and result post-processing. The prediction box is used to frame the position, size, and confidence information of all ships in the image.

[0080] In a preferred embodiment, the prediction frames are screened according to their size, confidence, and overlap to obtain a target prediction frame marked with the current ship, including:

[0081] S1: After removing the prediction boxes corresponding to the confidence scores below the preset confidence threshold, the prediction box with the highest current confidence score is taken as the optimal box, and the intersection-over-union ratio between the optimal box and the first prediction box retained after the removal is calculated;

[0082] S2, eliminating the first prediction frames corresponding to the intersection-over-union ratios greater than a first preset threshold, then pairing them up to obtain matching pairs, and calculating similarity scores for the matching pairs. The first prediction frames are then filtered based on the similarity scores to obtain second prediction frames.

[0083] S3: Filter the second prediction frame according to the relative position relationship of the bow of the ship to obtain the target prediction frame.

[0084] In a preferred embodiment, after removing the first prediction frames corresponding to the intersection-over-union ratios greater than the first preset threshold, step S2 includes:

[0085] S21, pairing the remaining first prediction frames to obtain matching pairs, and calculating the maximum intersection area of ​​the matching pairs;

[0086] S22, determining the matching pairs corresponding to the maximum intersection area greater than a second preset threshold as overlapping matching pairs;

[0087] S23, determining an aspect ratio according to the size of the first prediction frame;

[0088] S24, for each overlapping matching pair, executing: calculating the ratio of the minimum aspect ratio to the maximum aspect ratio of the first prediction box included in the overlapping matching pair, and multiplying the ratio by the maximum intersection area of ​​the first prediction box included in the overlapping matching pair as a similarity score;

[0089] S25 , screening the overlapping matching pairs and the first prediction box according to the similarity score and the preset score threshold to obtain screened matching pairs and the second prediction box.

[0090] It should be noted that, for the first prediction frame a and the first prediction box b The matching pairs, a The intersection area IOA a =( S a ∩ S b ) / S a ; b The intersection area IOAb =( S a ∩ S b ) / S b ;in, S a 、 S b They are a area, b The area, S a ∩ S b for a and b The area of ​​the intersection of the matching pairs. The maximum intersection area of ​​the matching pairs = max( IOA a , IOA b ). For example, if IOA a < IOA b , then the maximum intersection area of ​​the matching pairs is IOA b .

[0091] Specifically, for the first prediction box retained after the first screening, the maximum intersection area between the matching pairs is calculated and determined using a pair-by-pair approach ( IOA ), if the maximum IOA If the intersection area is greater than a second preset threshold (i.e., the preset maximum intersection area threshold), the pair is considered a suspicious overlapping match pair. Each matching pair is then traversed to obtain a list of overlapping match pairs. A preset scoring threshold is then applied to further filter out ship prediction frames with high overlapping similarity, resulting in the remaining filtered matching pairs and the second prediction frame. For example, in actual implementation, setting the second preset threshold to 0.85 effectively filters out prediction frames with high overlapping areas.

[0092] It should be noted that, for the matching pairs corresponding to the maximum intersection area that is not greater than the second preset threshold, they are directly retained, that is, the prediction boxes in the matching pairs are all determined to be the second prediction boxes.

[0093] Specifically, for each overlapping matching pair in the overlapping matching pair list, the similarity score of each overlapping matching pair is determined by the following formula:

[0094]

[0095] in, sim ab To include the first prediction box a With the first prediction box bSimilarity scores of overlapping matching pairs; IOA a 、 IOA b Overlapping matching pairs a 、 b The maximum intersection area of γ a 、 γ b They are a 、 b aspect ratio; For γ a and γ b Take the minimum value; For γ a and γ b Take the maximum value.

[0096] Ship detection models typically include multiple detection heads, each focused on different scales, to detect objects at multiple scales. Consequently, for the same ship, different heads may output different detection results due to their different observation scales. Since there is only one ship target, redundant detection frames need to be filtered out. This method considers the aspect ratio to determine the shape similarity between two predicted frames, more accurately eliminating overlapping frames with similar aspect ratios while retaining overlapping frames with dissimilar shapes. Furthermore, by combining the maximum intersection area, adaptive ship target recognition is improved.

[0097] In a preferred embodiment, step S25 includes:

[0098] Determine whether the similarity score of the overlapping matching pairs is less than a preset score threshold;

[0099] If yes, the first prediction boxes in the overlapping matching pairs are all used as the second prediction boxes, and the overlapping matching pairs are used as the screening matching pairs;

[0100] If not, the first prediction box with the smaller maximum intersection area in the overlapping matching pair is used as the second prediction box.

[0101] Specifically, if the similarity score of an overlapping match pair is less than a preset score threshold, the two first prediction frames within the overlapping match pair are retained. Otherwise, the first prediction frame with the largest IOA in the overlapping match pair is deleted from the initial screening prediction results, and the overlapping match pair is removed from the list of suspicious overlapping match pairs. The retained overlapping match pair becomes the screened match pair, and the retained first prediction frame becomes the second prediction frame. For example, in actual implementation, setting the preset score threshold to 0.7 can more accurately filter out duplicate prediction frames of the same ship target.

[0102] In a preferred embodiment, step S3 includes:

[0103] The screening matching pair includes the second prediction box A and the second prediction box B, and the size of the second prediction box A is larger than the size of the second prediction box B;

[0104] For each filter matching pair, execute:

[0105] Inside the second prediction box A, using each vertex of the second prediction box A as an anchor point, construct a virtual box of the same size as the second prediction box B; wherein the number of virtual boxes is the same as the number of anchor points;

[0106] Calculate the intersection-over-union ratio between the second predicted box B and the virtual box;

[0107] When the intersection-over-union ratio is not less than a third preset threshold, the second prediction box B is deleted from the second prediction box to obtain the target prediction box.

[0108] Specifically, each screening matching pair includes a second prediction box A with a larger size and a second prediction box B with a smaller size (i.e., the size of the second prediction box A is larger than the second prediction box B). For each screening matching pair, let the four vertices of the second prediction box A be ( x A1 , y A1 )、( x A2 , y A2 )、( x A3 , y A3 )、( x A4 , y A4 ), the width of the second prediction box B is w B Gao Wei h B . Take the four vertices of the second prediction box A as anchor points, and construct a virtual box with the same width and height as the second prediction box B inside the second prediction box A. xA1 , y A1 ) point as the anchor point, the coordinates of the four vertices of the constructed virtual frame are ( x A1 , y A1 )、( x A1 , y A1 + h B )、( x A1 + w B , y A1 + h B )、( x A1 + w B , y A1 );by( x A2 , y A2 ) point as the anchor point, the coordinates of the four vertices of the constructed virtual frame are ( x A2 , y A2 - h B )、( x A2 , y A2 )、( x A2 + w B , y A2 )、( x A2 + w B , y A2 - h B );by( x A3 , y A3 ) point as the anchor point, the coordinates of the four vertices of the constructed virtual frame are ( x A3 - w B , y A3 - h B )、(x A3 - w B , y A3 )、( x A3 , y A3 )、( x A3 , y A3 - h B );by( x A4 , y A4 ) point as the anchor point, the coordinates of the four vertices of the constructed virtual frame are ( x A4 - w B , y A4 )、( x A4 - w B , y A4 + h B )、( x A4 , y A4 + h B )、( x A4 , y A4 After the four virtual frames are constructed, the intersection-and-union (IoU) values ​​are calculated with the second prediction frame B. If the calculated IoU values ​​are all less than the third preset threshold (i.e., the preset bow IoU threshold), the second prediction frame B is retained. Otherwise, the second prediction frame B is deleted. The remaining second prediction frame is then the final target prediction frame.

[0109] In this method, if a ship's bow is repeatedly detected, the prediction box for the bow is always located at the corner of the ship's prediction box. Based on this relative positional relationship between the bow and the ship target, the second prediction box retained after the second screening is further filtered by the prediction box with the larger IOA in the screened matching pairs, removing any smaller prediction boxes that may be bows. Ultimately, the target prediction box for the ship target is obtained. This solves the problems of target box overlap and repeated bow position detection that occur in existing detection methods, effectively improving the accuracy and robustness of ship target detection.

[0110] In step 104, the current ships are tracked and visual feature information and track information of each current ship are obtained, including:

[0111] A multi-target tracking algorithm is used to associate the current ships in the current images at different times to determine the track information of each current ship; the track information includes the direction and rate of position change;

[0112] The current image is subjected to feature extraction to obtain visual feature information of each current ship; wherein the visual feature information includes ship type, color composition, ship building position, number of ship floors, cargo type and direction.

[0113] In a preferred embodiment, a multi-target tracking algorithm is used to associate the current ship in the current image at different times, including:

[0114] At the current moment, the current ships in the current image at the current moment are numbered respectively using a multi-target tracking algorithm to determine the ID of each current ship;

[0115] In a preset time period after the current moment, the current ship to which the ID that can be obtained at different moments belongs is determined to be a stable ship target, so as to record and obtain the track information of the stable ship target based on the ID.

[0116] Specifically, a multi-target tracking algorithm is used to associate ships between different detection frames (i.e., current images at different times), assigning each ship a unique ID. Based on this ID, the track information (including the direction and rate of position change) of each ship in the image over a preset time period is recorded. For ship targets that can be stably detected, the predicted frames output by the target detection model are used to crop the image, generating a small image of each current ship. This small image is then preprocessed and input into a ship feature extraction network for feature extraction, thereby obtaining visual feature information such as the ship's type, color composition, ship's building location, number of floors, cargo type, and orientation. It should be noted that the direction of position change refers to the direction in which the predicted frame of the ship changes during the radar guidance interval; the rate of position change refers to the rate at which the ship's target frame changes from the horizontal and vertical mid-axis of the image over a period of time.

[0117] In the present invention, the multi-target tracking algorithm includes, but is not limited to, the ByteTrack algorithm. The ByteTrack algorithm establishes a Kalman tracker for each detected target and uses positional correspondences to match and associate identical targets. Furthermore, when faced with low-confidence target vessel detection results due to occlusion or other factors, the algorithm dynamically adjusts the target vessel matching rate to avoid missing targets, thereby improving the stability of target vessel presence.

[0118] It should be noted that, preferably, the ship feature extraction network is based on the MobileNetV4 network architecture, and uses 12 types of ship type datasets, ship color datasets, building datasets, and ship direction datasets to train ship type, color composition, building position, number of ship floors, cargo type, and orientation classifiers. Therefore, a small image of a ship with a stable ID is input into the ship feature extraction network, and the network will output ship visual feature information composed of the above information.

[0119] In step 106, the target information includes the target position change direction, target change rate, target ship type, target color composition, target ship building position, target ship floor number, target cargo type and target direction;

[0120] Visual feature information includes ship type, color composition, ship deck location, number of decks, cargo type, and orientation;

[0121] Based on target information, visual feature information, and track information, the target vessel is identified from the current vessels for tracking, including:

[0122] Selecting target track information having the same position change direction and rate as the target position change direction from the track information including the position change direction and rate, and determining the current ship corresponding to the target track information as the preliminarily selected ship;

[0123] By comparing visual feature information and target information, the comprehensive matching score of each initially screened ship is determined;

[0124] The initially screened ship corresponding to the highest comprehensive matching score is determined as the target ship.

[0125] Specifically, after acquiring visual feature information, the system uses the target information returned by the radar-AIS to screen and match each current ship, confirming the target ship. The target ship's position is then framed in the image and feature information from the radar-AIS and visual algorithms is superimposed on it, including:

[0126] The current ship is screened and matched by integrating visual feature information, track information and target information. Specifically, the position change direction and rate in the track information are matched with the target position change direction and target change rate in the target information from the radar-AIS. If the matching results are identical or substantially identical, the initial screening ship is determined. Then, the visual feature information and target information are continued to be matched to further determine the matching target ship.

[0127] In the present invention, based on the traditional radar-AIS guided optoelectronic tracking of ships, the problem that the original method was difficult to identify the ship to be tracked among multiple ships within the field of view was solved. Visual feature information was introduced to optimize the identification process of the target ship to be tracked. This not only improved the success rate of identifying the selected ship target, but also enriched the target feature information of the ship, providing convenience for the subsequent management and control work; at the same time, it further improved the linkage effect between basic coastal defense equipment such as radar, AIS, and optoelectronic turntable, and further strengthened the intelligent duty capability of the coastal defense perception terminal.

[0128] In a preferred embodiment, the comprehensive matching score of each initially screened vessel is determined by comparing the visual feature information and the target information, including:

[0129] Determining a first matching score between the target ship type and the ship type by comparison;

[0130] determining a second matching score between the target cargo category and the cargo category by comparison;

[0131] Determine the third matching score of the target orientation and the orientation by comparison;

[0132] Determining a fourth matching score between the target color composition and the color composition, the target ship building position and the ship building position, and the target ship floor number and the ship floor number by comparing;

[0133] A comprehensive matching score is determined based on the first matching score, the second matching score, the third matching score, and the fourth matching score.

[0134] Specifically, the comprehensive matching score is determined by the following formula:

[0135]

[0136] in, P Score the overall match; α 1. α 2. α 3. α 4 are weight coefficients respectively; c is the confidence level of the ship type; b 1 is the first matching score; b 2 is the second matching score; Score the third match, f i For the i The similarity between the target direction and the direction; the directions include the bow direction, the right front direction, the starboard direction, the right rear direction, the stern direction, the left rear direction, the port direction and the left front direction; Score the fourth match, d 1 is the similarity between the target color composition and the color composition,d 2 is the similarity between the target building position and the building position, d 3 is the similarity between the number of floors of the target ship and the number of floors of the ship; α 1+ α 2+ α 3+ α 4=1.

[0137] It should be noted that the first matching score is the similarity between the target ship type and the ship type; the second matching score is the similarity between the target cargo type and the cargo type.

[0138] In the present invention, the influence of confidence on ship type characteristics is taken into account; the cargo types are strictly controlled through the second matching score, the influence of cargo type differences on the comprehensive matching score is enhanced, and the accurate identification of cargo types is improved; and by comprehensively considering various orientations and suppressing the influence of orientations to a large extent, the excessive interference of orientations on the comprehensive matching score is reduced; and by suppressing the degree of color composition, ship's tower position, number of ship floors and other features, the strict limitations of these factors on positioning ships are reduced, and finally the comprehensive matching score for distinguishing the target ship is determined, and the various factors complement each other to ensure the accurate identification of the target ship.

[0139] Specifically, in the present invention, after the target ship is determined from the current ships, the radar-AIS and visual feature information are superimposed on the picture, thereby completing the precise positioning of the target ship during the tracking process. At the same time, after the target ship is determined from the current ships, a single target tracking algorithm is used to calculate the target miss amount based on the difference between the predicted position of the target ship and the center of the picture, and the photoelectric turntable control is changed from radar-AIS guidance to tracking miss amount control, thereby stabilizing the prediction frame of the target ship and ensuring that the tracked target is located in the center of the picture for a long time. It should be noted that the single target tracking algorithm can be a SiamRPN++ algorithm based on a twin network architecture, which has certain anti-occlusion and anti-scale change capabilities and can achieve relatively stable tracking.

[0140] The present invention uses visual feature information to optimize the radar-AIS guided target identification process, making it easier to identify the designated ship target to be tracked from multiple ships in the video image, and realizes the fusion and enrichment of multi-source target features, effectively improving the effect of radar, AIS, and optoelectronic fusion perception in the field of coastal defense, and strengthening the intelligent on-duty capability of the equipment.

[0141] like Figure 3 、 Figure 4 As shown, the embodiment of the present invention provides a radar-AIS ship tracking device based on visual feature optimization. The device embodiment can be implemented by software, hardware, or a combination of software and hardware. From the hardware level, Figure 3As shown in FIG, a hardware architecture diagram of a computing device of a radar-AIS ship tracking device based on visual feature optimization provided by an embodiment of the present invention is provided. Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, the computing device in the embodiment may also include other hardware, such as a forwarding chip responsible for processing messages, etc. Taking software implementation as an example, Figure 4 As shown, as a logical device, the CPU of the computing device in which it is located reads the corresponding computer program in the non-volatile memory into the internal memory and runs it. This embodiment provides a radar-AIS ship tracking device based on visual feature optimization, including:

[0142] The monitoring and guidance module 400 is used to guide the optoelectronic turntable to the direction of the target ship based on the target information of the target ship obtained by the radar-AIS;

[0143] The recognition module 402 is used to recognize the current image collected by the photoelectric turntable to obtain a number of current ships;

[0144] The tracking module 404 is used to track the current ships and obtain the visual feature information and track information of each current ship; and determine the target ship from the current ships for tracking based on the target information, visual feature information and track information.

[0145] In some specific implementations, the monitoring and guidance module 400 may be used to execute the above step 100 , the identification module 402 may be used to execute the above step 102 , and the tracking module 404 may be used to execute the above steps 104 and 106 .

[0146] In some specific implementations, the monitoring and guidance module 400 is further configured to perform the following operations:

[0147] The azimuth and pitch angles of the target ship are calculated based on the target information including the target longitude and latitude, target distance, and the longitude and latitude and installation height of the optoelectronic turntable;

[0148] The optoelectronic turntable is rotated according to the azimuth and elevation angles so that the image center of the camera on the optoelectronic turntable coincides with the guidance azimuth center of the radar-AIS.

[0149] In some specific implementations, the identification module 402 is further configured to perform the following operations:

[0150] Input the current image into the object detection model and output the predicted box marked with the ship;

[0151] The prediction boxes are filtered according to their size, confidence, and overlap to obtain the target prediction box marked with the current ship.

[0152] In some specific implementations, the identification module 402 is further configured to perform the following operations:

[0153] S1: After removing the prediction boxes corresponding to the confidence scores below the preset confidence threshold, the prediction box with the highest current confidence score is taken as the optimal box, and the intersection-over-union ratio between the optimal box and the first prediction box retained after the removal is calculated;

[0154] S2, eliminating the first prediction frames corresponding to the intersection-over-union ratios greater than a first preset threshold, then pairing them up to obtain matching pairs, and calculating similarity scores for the matching pairs. The first prediction frames are then filtered based on the similarity scores to obtain second prediction frames.

[0155] S3: Filter the second prediction frame according to the relative position relationship of the bow of the ship to obtain the target prediction frame.

[0156] In some specific implementations, the tracking module 404 is further configured to perform the following operations:

[0157] A multi-target tracking algorithm is used to associate the current ships in the current images at different times to determine the track information of each current ship; the track information includes the direction and rate of position change;

[0158] The current image is subjected to feature extraction to obtain visual feature information of each current ship; wherein the visual feature information includes ship type, color composition, ship building position, number of ship floors, cargo type and direction.

[0159] In some specific embodiments, the target information includes the target position change direction, the target change rate, the target ship type, the target color composition, the target ship building position, the target ship floor number, the target cargo type, and the target direction;

[0160] Visual feature information includes ship type, color composition, ship deck location, number of decks, cargo type and orientation.

[0161] In some specific implementations, the tracking module 404 is further configured to perform the following operations:

[0162] Selecting target track information having the same position change direction and rate as the target position change direction from the track information including the position change direction and rate, and determining the current ship corresponding to the target track information as the preliminarily selected ship;

[0163] By comparing visual feature information and target information, the comprehensive matching score of each initially screened ship is determined;

[0164] The initially screened ship corresponding to the highest comprehensive matching score is determined as the target ship.

[0165] In some specific implementations, the tracking module 404 is further configured to perform the following operations:

[0166] Determining a first matching score between the target ship type and the ship type by comparison;

[0167] determining a second matching score between the target cargo category and the cargo category by comparison;

[0168] Determine the third matching score of the target orientation and the orientation by comparison;

[0169] Determining a fourth matching score between the target color composition and the color composition, the target ship building position and the ship building position, and the target ship floor number and the ship floor number by comparing;

[0170] A comprehensive matching score is determined based on the first matching score, the second matching score, the third matching score, and the fourth matching score.

[0171] It should be understood that the illustrated structure of the embodiments of the present invention does not constitute a specific limitation on a radar-AIS ship tracking device based on visual feature optimization. In other embodiments of the present invention, a radar-AIS ship tracking device based on visual feature optimization may include more or fewer components than illustrated, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0172] The information interaction, execution process, etc. between the modules in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention. For specific contents, please refer to the description in the embodiment of the method of the present invention and will not be repeated here.

[0173] An embodiment of the present invention further provides a computing device including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, a radar-AIS ship tracking method based on visual feature optimization according to any embodiment of the present invention is implemented.

[0174] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the processor is caused to execute a radar-AIS ship tracking method based on visual feature optimization according to any embodiment of the present invention.

[0175] An embodiment of the present application also provides a computer program product, which includes a computer program. A processor of a computer device reads the computer program from a computer-readable storage medium, and the processor executes the computer program, so that the computer device executes a radar-AIS ship tracking method based on visual feature optimization as described in any of the above embodiments.

[0176] Specifically, a system or device equipped with a storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program codes stored in the storage medium.

[0177] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0178] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, and DVD+RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer via a communications network.

[0179] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0180] In addition, it can be understood that the program code read from the storage medium is written into a memory provided in an expansion board inserted into the computer or into a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0181] It should be noted that, in this article, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical factors in the process, method, article or device comprising the elements.

[0182] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A radar-AIS ship tracking method based on visual feature optimization, characterized in that: include: According to the target information of the target ship acquired by the radar-AIS, the optoelectronic turntable is guided to the direction of the target ship; The target information includes the target position change direction, target change rate, target ship type, target color composition, target ship building position, target ship floor number, target cargo type and target direction; Identifying the current image collected by the photoelectric turntable to obtain several current ships; Tracking the current ships and obtaining visual feature information and track information of each current ship; the visual feature information includes ship type, color composition, ship building position, number of ship floors, cargo type and direction; screening target track information having the same target position change direction and target position change rate from the track information including the position change direction and the position change rate, and determining the current ship corresponding to the target track information as the preliminarily screened ship; Determining a first matching score between the target ship type and the ship type by comparison; determining a second matching score between the target cargo category and the cargo category by comparison; determining a third matching score between the target orientation and the orientation by comparison; determining a fourth matching score between the target color composition and the color composition, the target building position and the building position, and the target number of ship floors and the number of ship floors by comparison; A comprehensive matching score is determined based on the first matching score, the second matching score, the third matching score, and the fourth matching score. The comprehensive matching score is determined by the following formula: Where P is the comprehensive matching score; α1, α2, α3, and α4 are weight coefficients respectively; c is the confidence level of the ship type; b1 is the first matching score; b2 is the second matching score; Score the third match, f i is the similarity between the orientation of the i-th target and the orientation; the orientation includes the bow direction, the right front direction, the starboard direction, the right rear direction, the stern direction, the left rear direction, the port direction and the left front direction; is the fourth matching score, d1 is the similarity between the target color composition and the color composition, d2 is the similarity between the target building position and the building position, and d3 is the similarity between the target ship floor number and the ship floor number; α1+α2+α3+α4=1; The initially screened ship corresponding to the highest comprehensive matching score is determined as the target ship.

2. The method according to claim 1, characterized in that The step of guiding the optoelectronic turntable to the direction of the target ship according to the target information of the target ship acquired by the radar-AIS includes: Calculating the azimuth and pitch angles of the target ship based on the target longitude and latitude, target distance, and the longitude and latitude, and installation height of the photoelectric turntable included in the target information; The photoelectric turntable is rotated according to the azimuth angle and the pitch angle so that the picture center of the camera on the photoelectric turntable coincides with the guidance azimuth center of the radar-AIS.

3. The method according to claim 1, characterized in that Tracking the current ships and obtaining visual feature information and track information of each current ship includes: Using a multi-target tracking algorithm to correlate the current ships in the current images at different times, and determine the track information of each current ship; wherein the track information includes the position change direction and the position change rate; Feature extraction is performed on the current image to obtain visual feature information of each current ship; wherein the visual feature information includes ship type, color composition, ship building position, number of ship floors, cargo type and direction.

4. The method according to claim 1, wherein The current image collected by the photoelectric turntable is identified to obtain several current ships, including: Inputting the current image into an object detection model and outputting a predicted box marked with a ship; The prediction frames are screened according to the size, confidence and overlap of the prediction frames to obtain a target prediction frame marked with the current ship.

5. The method according to claim 4, characterized in that The step of screening the prediction frame according to the size, confidence, and overlap of the prediction frame to obtain a target prediction frame marked with the current ship includes: After eliminating the prediction boxes corresponding to the confidence levels lower than a preset confidence threshold, the prediction box with the highest current confidence level is taken as the optimal box, and the intersection-over-union ratio between the optimal box and the first prediction box retained after the elimination is calculated; Eliminating the first prediction frames corresponding to the IoU ratios greater than a first preset threshold, performing pairing to obtain matching pairs, and calculating similarity scores of the matching pairs, so as to filter the first prediction frames based on the similarity scores to obtain second prediction frames; The second prediction frame is filtered according to the relative position relationship of the bow of the ship to obtain the target prediction frame.

6. A radar-AIS ship tracking device based on visual feature optimization, characterized in that: Used to implement the method according to any one of claims 1 to 5, comprising: A monitoring and guidance module is used to guide the optoelectronic turntable to the direction of the target ship based on the target information of the target ship obtained by the radar-AIS; an identification module, configured to identify the current image collected by the photoelectric turntable to obtain a number of current ships; The tracking module is used to track the current ships and obtain visual feature information and track information of each current ship; and determine the target ship from the current ships for tracking based on the target information, the visual feature information and the track information.

7. A computing device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 5.

9. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 5.

Images