Radar-AIS ship tracking method and device based on visual feature optimization

Abstract

The invention provides a radar-AIS ship tracking method and device based on visual feature optimization, and relates to the technical field of target tracking. The method comprises the following steps: guiding a photoelectric turntable to turn to the direction of a target ship according to target information of the target ship acquired by a radar-AIS (Automatic Identification System); identifying the current image acquired by the photoelectric turntable to obtain a plurality of current ships; current ships are tracked, and visual feature information and track information of the current ships are obtained; and according to the target information, the visual feature information and the track information, determining a target ship from the current ships for tracking. According to the scheme, accurate identification and tracking of the ship target are realized, and a stable and reliable tracking effect is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, in particular to the technical field of ship track tracking, and particularly to a ship tracking method and device based on radar-AIS optimized by visual features. Background Art

[0002] The basic function of the ship traffic management system is ship tracking, and the ship tracking function mainly depends on maritime radars and AIS base stations. The transmission frequency of the on-board AIS transceiver is not fixed, and there are delays and errors. If only the radar-AIS information is relied on to guide the optoelectronic turret to track the ship target, it will be difficult to identify the target among multiple ships when the ships are dense. Even if the target ship to be tracked is successfully found, it will be limited by the amount of radar-AIS information, making it difficult to conduct further analysis and real-time tracking of the target in a timely manner. Therefore, there is an urgent need to provide a ship tracking method and device based on radar-AIS optimized by visual features. Summary of the Invention

[0003] The present invention provides a ship tracking method and device based on radar-AIS optimized by visual features. This method realizes the accurate identification and tracking of ship targets based on on-board radars, providing a stable and reliable tracking effect.

[0004] In a first aspect, the present invention provides a ship tracking method based on radar-AIS optimized by visual features, including: Guiding the optoelectronic turret to turn to the azimuth of the target ship according to the target information of the target ship obtained by radar-AIS; Identifying the current image collected by the optoelectronic turret to obtain a number of current ships; Tracking the current ships and obtaining the visual feature information and track information of each current ship; Determining the target ship from the current ships for tracking according to the target information, the visual feature information, and the track information.

[0005] Optionally, the guiding the optoelectronic turret to turn to the azimuth of the target ship according to the target information of the target ship obtained by radar-AIS includes: Calculating the azimuth angle and elevation angle of the target ship according to the target longitude and latitude, target distance included in the target information, and the longitude and latitude, installation height of the optoelectronic turret; Rotating the optoelectronic turret following the azimuth angle and the elevation angle so that the center of the camera screen on the optoelectronic turret coincides with the guiding azimuth center of the radar-AIS.

[0006] Optionally, tracking the current ship and obtaining the visual feature information and trajectory information of each current ship, including: Using a multi-object tracking algorithm to associate the current ships in the current image at different times, and determining the trajectory information of each current ship; wherein, the trajectory information includes the position change direction and the position change rate; Performing feature extraction on the current image to obtain the visual feature information of each current ship; wherein, the visual feature information includes the ship type, color composition, position of the superstructure, number of superstructure floors, type of goods, and orientation.

[0007] Optionally, recognizing the current image collected by the optoelectronic turret to obtain several current ships, including: Inputting the current image into a target detection model to output prediction frames marked with ships; Screening the prediction frames according to the size, confidence level, and overlap situation of the prediction frames to obtain target prediction frames marked with the current ships.

[0008] Optionally, the screening the prediction frames according to the size, confidence level, and overlap situation of the prediction frames to obtain target prediction frames marked with the current ships includes: After removing the prediction frames corresponding to the confidence levels lower than the preset confidence threshold, taking the prediction frame with the highest current confidence level as the optimal frame, and calculating the intersection over union between the optimal frame and the first prediction frames retained after removal; Removing the first prediction frames corresponding to the intersection over union greater than the first preset threshold, then pairing them two by two to obtain matching pairs, and calculating the similarity scores of the matching pairs to screen the first prediction frames based on the similarity scores to obtain second prediction frames; Screening the second prediction frames according to the relative position relationship of the bows in the ships to obtain the target prediction frames.

[0009] Optionally, the target information includes the target position change direction, target change rate, target ship type, target color composition, target superstructure position, target superstructure floor number, target type of goods, and target orientation; The visual feature information includes the ship type, color composition, superstructure position, superstructure floor number, type of goods, and orientation.

[0010] Optionally, determining the target ship from the current ships for tracking according to the target information, the visual feature information, and the trajectory information, including: Filter out target track information that is the same as the target position change direction and the target position change rate from the track information including the position change direction and the position change rate, and determine the current ship corresponding to the target track information as the preliminarily screened ship; Determine the comprehensive matching score of each of the preliminarily screened ships by comparing the visual feature information and the target information; Determine the preliminarily screened ship corresponding to the highest comprehensive matching score as the target ship.

[0011] Optionally, the determining the comprehensive matching score of each of the preliminarily screened ships by comparing the visual feature information and the target information includes: Determine the first matching score between the target ship type and the ship type by comparison; Determine the second matching score between the target cargo type and the cargo type by comparison; Determine the third matching score between the target orientation and the orientation by comparison; Determine the fourth matching score between the target color composition and the color composition, the target shiphouse position and the shiphouse position, and the target number of ship floors and the number of ship floors by comparison; Determine the comprehensive matching score according to the first matching score, the second matching score, the third matching score, and the fourth matching score.

[0012] In a second aspect, the present invention further provides a ship tracking device based on radar-AIS optimized by visual features, including: A monitoring and guiding module, configured to guide the optoelectronic turntable to turn to the azimuth of the target ship according to the target information of the target ship obtained by radar-AIS; An identification module, configured to identify the current image collected by the optoelectronic turntable to obtain a number of current ships; A tracking module, configured to track the current ships, and obtain the visual feature information and track information of each current ship; and determine the target ship from the current ships for tracking according to the target information, the visual feature information, and the track information.

[0013] In a third aspect, the present invention further provides a computing device, including a memory and a processor, where a computer program is stored in the memory, and when the processor executes the computer program, the ship tracking method based on radar-AIS optimized by visual features described in any one of the above is implemented.

[0014] Fourthly, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed on a computer, the computer is made to execute the ship tracking method based on vision feature optimization described in any one of the above.

[0015] Fifthly, an embodiment of the present invention further provides a computer program product, including computer instructions, and when the computer instructions are executed by a processor, the steps of the method described in any first aspect of this specification are implemented.

[0016] The present invention provides a ship tracking method and device based on vision feature optimization of radar-AIS. The method first preliminarily guides an optoelectronic turntable to turn to the approximate azimuth of a target ship with the target information detected by radar-AIS, then calls a ship detection algorithm to locate and track the ships in the current image, obtains the position changes of each current ship in the picture, sends the images of the current ships that can be detected into a ship feature extraction network to obtain their visual characteristic information and track information, and then comprehensively matches the target information transmitted back by radar-AIS with the visual feature information and track information to determine the target ship for continuous positioning and tracking. In this way, on the basis of the radar-AIS guiding the optoelectronic turntable to track, the present invention uses vision features to assist in finding the target ship to be tracked, improves the effect of the fusion perception of radar, AIS and optoelectronics, realizes the accurate identification and tracking of ship targets, provides a stable and reliable tracking effect, and further strengthens the intelligent duty ability of basic coastal defense equipment. Description of the Drawings

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to these drawings without creative efforts.

[0018] Figure 1 is a flowchart of a ship tracking method based on vision feature optimization of radar-AIS provided by an embodiment of the present invention; Figure 2 is a schematic diagram of detecting a target ship provided by an embodiment of the present invention; Figure 3 is a hardware architecture diagram of a computing device provided by an embodiment of the present invention; Figure 4 is a structural diagram of a ship tracking device based on vision feature optimization of radar-AIS provided by an embodiment of the present invention. Detailed Embodiments

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0020] Due to the limitations of the radar, AIS transmission interval, and the accuracy of the optoelectronic turret, when relying solely on radar-AIS to guide the optoelectronic turret to track ship targets, it is impossible to ensure that the ship target is exactly located at the center of the screen. Therefore, in this case, if there are multiple ships in the screen, it will be difficult to identify the target ship. Even if the target ship is successfully identified, due to the limited amount of information of radar-AIS, it is difficult for the control party to further analyze the target in a timely manner. Therefore, the present invention proposes a method for fusing radar-AIS and optoelectronics to solve the above problems.

[0021] The following describes the concept of the present invention. Please refer to Figure 1 , the embodiments of the present invention provide a ship tracking method based on radar-AIS optimized by visual features, including: Step 100: According to the target information of the target ship obtained by radar-AIS, guide the optoelectronic turret to turn to the azimuth of the target ship; Step 102: Identify the current image collected by the optoelectronic turret to obtain several current ships; Step 104: Track the current ships and obtain the visual feature information and trajectory information of each current ship; Step 106: Determine the target ship from the current ships according to the target information, visual feature information, and trajectory information for tracking.

[0022] In the present invention, first, the target information detected by radar-AIS is used to initially guide the optoelectronic turret to turn to the approximate azimuth of the target ship. Then, the ship detection algorithm is called to locate and track the ships in the current image, obtain the position changes of each current ship in the screen, send the images of the current ships that can be detected into the ship feature extraction network to obtain their visual characteristic information and trajectory information, and then comprehensively match the target information transmitted back by radar-AIS with the visual feature information and trajectory information to determine the target ship for continuous positioning and tracking. In this way, based on the radar-AIS guidance for the optoelectronic turret to track, the present invention uses visual features to assist in finding the target ship to be tracked, improves the effect of the fusion perception of radar, AIS, and optoelectronics, realizes the accurate identification and tracking of ship targets from multiple ships, provides a stable and reliable tracking effect, and further strengthens the intelligent duty ability of basic coastal defense equipment.

[0023] The following describes Figure 1 the execution manner of each step shown.

[0024] First, in step 100, according to the target information of the target ship obtained by radar-AIS, the optoelectronic turret is guided to turn to the azimuth of the target ship, including: Based on the target longitude and latitude, target distance included in the target information, and the longitude and latitude, installation height of the optoelectronic turret, the azimuth angle and elevation angle of the target ship are calculated; The optoelectronic turret is rotated following the azimuth angle and elevation angle, so that the center of the camera image on the optoelectronic turret coincides with the guiding azimuth center of the radar-AIS.

[0025] It should be noted that AIS is the Automatic Identification System for Ships, which refers to a new type of navigation aid system applied to maritime safety and communication between ships and the shore, and between ships. Radar-AIS is an integrated monitoring system combining radar and Automatic Identification System (AIS) technologies, mainly composed of radar, AIS equipment, and a data processing center; the radar is used to detect and locate targets at sea, and the AIS exchanges information with surrounding ships through the VHF band to provide detailed information about the targets such as ship name, position, course, etc. A camera is installed on the optoelectronic turret.

[0026] Specifically, for the selected target ship, the approximate azimuth angle and elevation angle of the target ship are calculated using the target longitude and latitude, target distance detected by the radar-AIS system, and the longitude and latitude, installation height of the optoelectronic turret itself; then the optoelectronic turret is rotated following this azimuth angle and elevation angle to keep the center of the camera image coinciding with the guiding azimuth center of the radar, and at the same time, the field of view angle of the camera is roughly adjusted according to the target length included in the target information.

[0027] Figure 2 Shows a schematic diagram of the radar-AIS guiding the optoelectronic turret to track the target ship. The target ship detected by the radar-AIS in step 100 P has a target longitude and latitude lat 1, lon 1], the straight-line distance from the target ship to the detection device O (optoelectronic turret) d , the target reflection width information w 1. The above information, combined with the longitude and latitude O’ of the detection device itself lat 0, lon 0], and installation height h 0, can be used to calculate the horizontal distance P between the target ship and the observation point d’ , relative azimuth angle θ and relative elevation angle β, the detection process is as Figure 2 shown. The horizontal distance d’ can be calculated by the following haversine formula: where, R is the radius of the earth; a , c are both intermediate quantities; The relative azimuth θ is determined by the following formula: Since the observation target is a ship and its altitude is defaulted to 0, the relative pitch angle β is: In this way, the optoelectronic turret rotates following the relative azimuth θ and the relative pitch angle β to keep the center of the camera image coincident with the center of the radar-guided azimuth.

[0028] In a preferred embodiment, the field of view angle of the camera is roughly adjusted according to the target length included in the target information, so that the target length detected by radar-AIS accounts for 1 / 4 of the camera's image field of view.

[0029] Specifically, based on the target reflection width information w 1 of the target ship detected by radar, combined with the straight-line distance d to roughly adjust the field of view angle of the optoelectronic turret camera. Assuming that the target ship is located at the center of the image, the actual distance MN corresponding to the horizontal field of view of the horizontal axis of the target ship in the image can be estimated by the following formula: If the target is to account for about 1 / 4 of the length and width of the image, MN = 4 w 1, then the field of view angle γ of the camera should be adjusted to: In this way, not only can the target ship be clearly observed in the image, but also there is a certain field of view, which is convenient for subsequent processing of guidance errors.

[0030] In step 102, the current image collected by the optoelectronic turret is recognized to obtain a number of current ships, including: Input the current image into the target detection model, and output the prediction boxes marked with ships; Filter the prediction boxes according to the size, confidence level and overlap of the prediction boxes to obtain the target prediction boxes marked with the current ships.

[0031] Specifically, the current image with the azimuth of the optoelectronic turntable and the field of view angle of the camera roughly adjusted is continuously input into the target detection model, and the positions, sizes, and confidence information of all ships in the picture are obtained through real-time inference and processing.

[0032] In the present invention, a target detection model trained based on the YOLOv5 network architecture and a ship data set composed of a large number of actually captured ship pictures. The algorithm inputs an image of 640*640, and after model inference and result post-processing, prediction boxes are output. The prediction boxes are used to frame the positions, sizes, and confidence information of all ships in the picture.

[0033] In a preferred embodiment, the prediction boxes are screened according to the size, confidence, and overlap of the prediction boxes to obtain target prediction boxes marked with the current ship, including: S1, after removing the prediction boxes corresponding to the confidence levels lower than the preset confidence threshold, taking the prediction box with the highest current confidence as the optimal box, and calculating the intersection over union ratio between the optimal box and the first prediction boxes retained after removal; S2, removing the first prediction boxes corresponding to the intersection over union ratio greater than the first preset threshold, then pairing them two by two to obtain matching pairs, and calculating the similarity score of the matching pairs to screen the first prediction boxes based on the similarity score to obtain the second prediction boxes; S3, screening the second prediction boxes according to the relative position relationship of the ship bows in the ship to obtain the target prediction boxes.

[0034] In a preferred embodiment, after step S2 removes the first prediction boxes corresponding to the intersection over union ratio greater than the first preset threshold, it includes: S21, pairing the remaining first prediction boxes two by two to obtain matching pairs, and calculating the maximum intersection area of the matching pairs; S22, determining the matching pairs corresponding to the maximum intersection area greater than the second preset threshold as overlapping matching pairs; S23, determining the aspect ratio according to the size of the first prediction box; S24, for each overlapping matching pair, perform: calculating the ratio of the minimum aspect ratio to the maximum aspect ratio of the first prediction boxes included in the overlapping matching pair, and taking the product of the ratio and the maximum intersection area of the first prediction boxes included in the overlapping matching pair as the similarity score; S25, screening the overlapping matching pairs and the first prediction boxes according to the similarity score and the preset score threshold to obtain the screened matching pairs and the second prediction boxes.

[0035] It should be noted that for the matching pair including the first prediction box a and the first prediction box b of the matching pair, aIntersection area IOA a = ([[]] S a ∩ S b ) / S a ; b Intersection area IOA b = ([[]] S a ∩ S b ) / S b ; where, S a , S b are respectively a area, b area, S a ∩ S b is a and b area of the intersection part. The maximum intersection area of the matching pairs = max( IOA a , IOA b ). For example, if IOA a < IOA b , then the maximum intersection area of the matching pairs is IOA b .

[0036] Specifically, for the first prediction boxes retained after the first screening, the maximum intersection area ( IOA ) between the matching pairs is calculated and determined by traversing in a pairwise pairing manner. If the maximum IOA is greater than the second preset threshold (i.e., the preset maximum intersection area threshold), then this pair of matching pairs is regarded as a suspicious overlapping matching pair, and a list of overlapping matching pairs is obtained by traversing each matching pair. Then, the ship prediction boxes with high overlapping similarity are further screened out through the preset scoring threshold, and the remaining screened matching pairs and the second prediction boxes are obtained. For example, in the actual execution process, setting the second preset threshold to 0.85 can effectively screen out the prediction boxes with a relatively high overlapping area.

[0037] It should be noted that for the matching pairs corresponding to the maximum intersection area not greater than the second preset threshold, they are directly retained, that is, it is determined that the prediction boxes in this matching pair are all second prediction boxes.

[0038] Specifically, for each overlapping matching pair in the list of overlapping matching pairs, the similarity score of each overlapping matching pair is determined by the following formula: where sim ab is the similarity score of the overlapping matching pair including the first prediction box a and the first prediction box b ; IOA a , IOA b are respectively the maximum intersection areas of a , b in the overlapping matching pair; γ a , γ b are respectively a , b aspect ratios; is to take the minimum value between γ a and γ b ; is to take the maximum value between γ a and γ b .

[0039] Since the object detection model for ships usually includes several detection heads for detecting features of different scales to detect multi-scale objects. Therefore, for the same ship, due to the different observation scales of different detection heads, different detection results of different sizes are output by different detection heads for the ship, and there is only one ship object, so it is necessary to filter out the redundant detection boxes. In the present invention, by considering the aspect ratio to determine the shape similarity of two prediction boxes, the overlapping boxes with similar aspect ratios are more accurately removed, and the overlapping boxes with dissimilar shapes are retained; at the same time, by combining the maximum intersection area, the adaptive recognition of ship objects is improved.

[0040] In a preferred embodiment, step S25 includes: Judging whether the similarity score of the overlapping matching pair is less than a preset score threshold; If so, both the first prediction boxes in the overlapping matching pair are used as the second prediction boxes, and the overlapping matching pair is used as a screening matching pair; If not, the first prediction box with the smaller maximum intersection area in the overlapping matching pair is used as the second prediction box.

[0041] Specifically, if the similarity score of an overlapping match pair is less than the preset score threshold, then the two first prediction boxes within the overlapping match pair are retained; otherwise, in the prediction results after preliminary screening, the first prediction box with the largest IOA in the overlapping match pair is deleted, and the overlapping match pair is removed from the list of suspicious overlapping match pairs. The remaining overlapping match pairs are the screened match pairs, and the remaining first prediction boxes are the second prediction boxes. For example, in the actual execution process, setting the preset score threshold to 0.7 can more accurately filter out the repeated prediction boxes of the same ship target.

[0042] In a preferred embodiment, step S3 includes: The screened match pairs include a second prediction box A and a second prediction box B, and the size of the second prediction box A is larger than the size of the second prediction box B; For each screened match pair, the following operations are performed: Inside the second prediction box A, taking each vertex of the second prediction box A as an anchor point, construct virtual boxes with the same size as the second prediction box B; where the number of virtual boxes is the same as the number of anchor points; Calculate the intersection over union (IoU) between the second prediction box B and the virtual boxes; When the IoU is not less than the third preset threshold, delete the second prediction box B from the second prediction boxes to obtain the target prediction boxes.

[0043] Specifically, each screened match pair includes a second prediction box A with a larger size and a second prediction box B with a smaller size (i.e., the size of the second prediction box A is larger than the size of the second prediction box B). For each screened match pair, let the four vertices of the second prediction box A be ( x A1 , y A1 ), ( x A2 , y A2 ), ( x A3 , y A3 ), ( x A4 , y A4 ). The width of the second prediction box B is w B , and the height is h B . Respectively taking the four vertices of the second prediction box A as anchor points, construct virtual boxes with the same width and height as the second prediction box B inside the second prediction box A. Taking ( x A1 , y A1The point is the anchor point, and the coordinates of the four vertices of the constructed virtual box are ( x A1 , y A1 ), ( x A1 , y A1 + h B ), ( x A1 + w B , y A1 + h B ), ( x A1 + w B , y A1 ); Taking the point ( x A2 , y A2 ) as the anchor point, the coordinates of the four vertices of the constructed virtual box are ( x A2 , y A2 - h B ), ( x A2 , y A2 ), ( x A2 + w B , y A2 ), ( x A2 + w B , y A2 - h B ); Taking the point ( x A3 , y A3 ) as the anchor point, the coordinates of the four vertices of the constructed virtual box are ( x A3 - w B , y A3 - h B ), ( x A3 - wB , y A3 ), ( x A3 , y A3 ), ( x A3 , y A3 - h B ); Taking ( x A4 , y A4 ) as the anchor point, the coordinates of the four vertices of the constructed virtual box are ( x A4 - w B , y A4 ), ( x A4 - w B , y A4 + h B ), ( x A4 , y A4 + h B ), ( x A4 , y A4 ). After the four virtual boxes are constructed, the intersection over union (IoU) is calculated using these virtual boxes and the second prediction box B. If the calculated IoU values are all less than the third preset threshold (i.e., the preset IoU threshold for the bow), the second prediction box B is retained; otherwise, the second prediction box B is deleted. In this way, the finally retained second prediction box is the final target prediction box.

[0044] In the present invention, if the bow part of a certain ship is detected repeatedly, then the prediction box of this bow will definitely be located at the corner position of the prediction box of this ship. Based on this relative position relationship between the bow and the ship target, for the second prediction box retained after the secondary screening, the prediction box with a larger IoA in the screening matching pairs is further filtered to remove the possible smaller prediction box that may be the bow, and finally the target prediction box for the ship target is obtained, thereby solving the problems of target box overlap and repeated detection of the bow position in the existing detection methods, and effectively improving the accuracy and robustness of ship target detection.

[0045] In step 104, the current ship is tracked, and the visual feature information and trajectory information of each current ship are obtained, including: Use a multi - target tracking algorithm to associate the current vessels in the current image at different times, and determine the track information of each current vessel; wherein, the track information includes the direction of position change and the rate of position change. Extract features from the current image to obtain the visual feature information of each current vessel; wherein, the visual feature information includes the ship type, color composition, position of the superstructure, number of superstructure floors, type of goods, and orientation.

[0046] In a preferred embodiment, using a multi - target tracking algorithm to associate the current vessels in the current image at different times includes: For the current time, use a multi - target tracking algorithm to number the current vessels in the current image at this time respectively, and determine the ID of each said current vessel. Within a preset time period after the current time, determine that the current vessels to which the IDs that can be obtained at different times belong are stable vessel targets, so as to record and obtain the track information of the stable vessel targets based on the IDs.

[0047] Specifically, use a multi - target tracking algorithm to associate the vessels between different detection frames (i.e., the current images at different times), assign a unique ID to the same vessel, and record the track information (including the direction and rate of position change) of each vessel in the images within the preset time period according to the ID. For the vessel targets that can be stably detected, use the prediction boxes output by the target detection model to intercept the images, obtain the small images of each current vessel, and then pre - process the small images and input them into the vessel feature extraction network for feature extraction to obtain the visual feature information such as the ship type, color composition, position of the superstructure, number of superstructure floors, type of goods, and orientation of each current vessel. It should be noted that the direction of position change refers to the change direction of the prediction box of the vessel within the radar guidance interval; the rate of position change refers to the change rate of the vessel target box from the horizontal and vertical central axes of the image within a period of time.

[0048] In the present invention, the multi - target tracking algorithm includes but is not limited to using the ByteTrack algorithm. The ByteTrack algorithm establishes a Kalman tracker for each detected target and uses the position correspondence relationship to complete the matching association of the same target. At the same time, when facing the detection results of low - confidence target vessels caused by occlusion and other situations, the algorithm can avoid the loss of target vessels by dynamically adjusting the matching probability of target vessels, thereby improving the stability of the existence of target vessels.

[0049] It should be noted that, preferably, the ship feature extraction network is based on the MobileNetV4 network architecture, and uses 12 types of ship type datasets, ship color datasets, superstructure datasets, and ship direction datasets to train the ship type, color composition, superstructure position, number of superstructure floors, cargo type, and orientation classifiers respectively. Therefore, when the small image of the ship with a stable ID is input into the ship feature extraction network, the network will output the ship visual feature information composed of the above information.

[0050] In step 106, the target information includes the target position change direction, target change rate, target ship type, target color composition, target superstructure position, target number of superstructure floors, target cargo type, and target orientation. The visual feature information includes the ship type, color composition, superstructure position, number of superstructure floors, cargo type, and orientation. According to the target information, visual feature information, and track information, determining the target ship from the current ships for tracking includes: Screening the target track information that is the same as the target position change direction and target position change rate from the track information including the position change direction and position change rate, and determining the current ship corresponding to the target track information as the initially screened ship; Determining the comprehensive matching score of each initially screened ship by comparing the visual feature information and the target information; Determining the initially screened ship corresponding to the highest comprehensive matching score as the target ship.

[0051] Specifically, after the visual feature information is obtained, each current ship is screened and matched by integrating the target information transmitted back by the radar-AIS to complete the confirmation of the target ship, and then the position of the target ship is framed in the picture, and the feature information from the radar-AIS and the visual algorithm is superimposed on it, including: Integrating the visual feature information, track information, and target information to perform screening and matching of the current ships, specifically including: matching the position change direction and rate in the track information with the target position change direction and target change rate in the target information from the radar-AIS. If the matching result is the same or basically the same, the initially screened ship after preliminary screening is determined; then continue to match the visual feature information and the target information to further determine the matching target ship.

[0052] In the present invention, based on the traditional radar-AIS-guided optoelectronic tracking ship, in view of the problem that it is difficult to identify the ship to be tracked among multiple ships within the field of view in the original method, visual feature information is introduced to optimize the identification process of the ship to be tracked, which not only improves the identification success rate of the selected ship target, but also enriches the target feature information of the ship, providing convenience for the subsequent control work; at the same time, it further improves the linkage effect among basic coastal defense equipment such as radar, AIS, and optoelectronic turntable, and further strengthens the intelligent duty ability of the coastal defense perception end.

[0053] In a preferred embodiment, by comparing the visual feature information and the target information, the comprehensive matching score of each preliminarily screened ship is determined, including: Determining the first matching score of the target ship type and the ship type by comparison; Determining the second matching score of the target cargo type and the cargo type by comparison; Determining the third matching score of the target orientation and the orientation by comparison; Determining the fourth matching score between the target color composition and the color composition, the target shiphouse position and the shiphouse position, and the target ship floor number and the ship floor number by comparison; Determining the comprehensive matching score according to the first matching score, the second matching score, the third matching score, and the fourth matching score.

[0054] Specifically, the comprehensive matching score is determined by the following formula: Among them, P is the comprehensive matching score; α 1, α 2, α 3, α 4 are respectively weight coefficients; c is the confidence of the ship type; b 1 is the first matching score; b 2 is the second matching score; is the third matching score, f i is the i similarity between the th target orientation and the orientation; among them, the orientation includes the bow direction, the right front side direction of the ship, the starboard direction, the right rear side direction of the ship, the stern direction, the left rear side direction of the ship, the port direction, and the left front direction of the ship; is the fourth matching score, d 1 is the similarity between the target color composition and the color composition, d 2 is the similarity between the target shiphouse position and the shiphouse position, d 3 is the similarity between the target ship floor number and the ship floor number; α 1 + α2+ α 3+ α 4 = 1。

[0055] It should be noted that the first matching score is the similarity between the target ship type and the ship type; the second matching score is the similarity between the target cargo type and the cargo type.

[0056] In the present invention, the influence of confidence on ship type features is considered; and the cargo type is strictly controlled through the second matching score, enhancing the influence of cargo type differences on the comprehensive matching score and improving the accurate recognition of cargo types; and by comprehensively considering each orientation and suppressing the influence of the orientation to a large extent, reducing the excessive interference of the orientation on the comprehensive matching score; and also by suppressing features such as color composition, position of superstructure, and number of superstructure floors to a certain extent, reducing the strict limitation of these factors on ship positioning, and finally determining the comprehensive matching score for discriminating the target ship, and using the complementarity of each factor to jointly ensure the accurate recognition of the target ship.

[0057] Specifically, in the present invention, after determining the target ship from the current ships, the radar-AIS and visual feature information are superimposed on the screen, so as to complete the accurate positioning of the target ship during the tracking process. At the same time, after determining the target ship from the current ships, a single-object tracking algorithm is adopted, and the target off-target amount is calculated according to the difference between the predicted position of the target ship and the center of the screen, and the control of the optoelectronic turret is changed from radar-AIS guidance to using the tracking off-target amount control, so as to stabilize the predicted frame of the target ship and ensure that the tracking target is long-term located at the center of the screen. It should be noted that the single-object tracking algorithm can be the SiamRPN++ algorithm based on the Siamese network architecture, and this algorithm has a certain ability to resist occlusion and scale changes and can achieve relatively stable tracking.

[0058] The present invention uses visual feature information to optimize the identification process of the radar-AIS guided target, making it easier to identify the specified ship target to be tracked from multiple ships in the video screen, and realizing the fusion and enrichment of multi-source target features, effectively improving the effect of radar, AIS, and optoelectronic fusion perception in the coastal defense field and strengthening the intelligent duty ability of the equipment.

[0059] Such as Figure 3 、 Figure 4 shown, the embodiment of the present invention provides a ship tracking device based on radar-AIS optimized by visual features. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. From the hardware level, as Figure 3 shown, it is a hardware architecture diagram of a computing device where the ship tracking device based on radar-AIS optimized by visual features provided by the embodiment of the present invention is located. Except for Figure 3In addition to the processor, memory, network interface, and non-volatile memory shown, the computing device where the device is located in the embodiment generally may also include other hardware, such as a forwarding chip responsible for processing packets, etc. Taking software implementation as an example, as Figure 4 shown, as a logically meaningful device, it is formed by the CPU of the computing device where it is located reading the corresponding computer program in the non-volatile memory into the memory and running it. A ship tracking device based on radar-AIS optimized by visual features provided in this embodiment includes: A monitoring and guiding module 400, configured to guide the optoelectronic turret to turn to the azimuth of the target ship according to the target information of the target ship obtained by the radar-AIS; An identification module 402, configured to identify the current image collected by the optoelectronic turret to obtain several current ships; A tracking module 404, configured to track the current ships, and obtain the visual feature information and track information of each current ship; and determine the target ship from the current ships according to the target information, visual feature information, and track information for tracking.

[0060] In some specific implementation manners, the monitoring and guiding module 400 may be configured to execute the above step 100, the identification module 402 may be configured to execute the above step 102, and the tracking module 404 may be configured to execute the above step 104 and step 106.

[0061] In some specific implementation manners, the monitoring and guiding module 400 is further configured to perform the following operations: Calculate the azimuth angle and pitch angle of the target ship according to the target longitude and latitude, target distance included in the target information, and the longitude and latitude and installation height of the optoelectronic turret; Rotate the optoelectronic turret following the azimuth angle and pitch angle so that the center of the camera's picture on the optoelectronic turret coincides with the guiding azimuth center of the radar-AIS.

[0062] In some specific implementation manners, the identification module 402 is further configured to perform the following operations: Input the current image into the target detection model and output a prediction box marked with a ship; Screen the prediction boxes according to the size, confidence level, and overlap situation of the prediction boxes to obtain a target prediction box marked with the current ship.

[0063] In some specific implementation manners, the identification module 402 is further configured to perform the following operations: S1. After removing the prediction boxes corresponding to the confidence levels lower than the preset confidence threshold, use the prediction box with the highest current confidence level as the optimal box, and calculate the intersection-over-union ratio between the optimal box and the first prediction box retained after removal; S2. Discard the first prediction boxes corresponding to the intersection over union greater than the first preset threshold, then pair them up two by two to obtain matching pairs, and calculate the similarity scores of the matching pairs, so as to screen the first prediction boxes based on the similarity scores to obtain the second prediction boxes; S3. Screen the second prediction boxes according to the relative position relationship of the bow in the ship to obtain the target prediction boxes.

[0064] In some specific embodiments, the tracking module 404 is further configured to perform the following operations: Use the multi-object tracking algorithm to associate the current ships in the current image at different times to determine the trajectory information of each current ship; wherein, the trajectory information includes the position change direction and the position change rate; Extract features from the current image to obtain the visual feature information of each current ship; wherein, the visual feature information includes the ship type, color composition, position of the superstructure, number of superstructure floors, type of goods, and orientation.

[0065] In some specific embodiments, the target information includes the target position change direction, target change rate, target ship type, target color composition, target superstructure position, target superstructure floor number, target type of goods, and target orientation; The visual feature information includes the ship type, color composition, position of the superstructure, number of superstructure floors, type of goods, and orientation.

[0066] In some specific embodiments, the tracking module 404 is further configured to perform the following operations: Screen the target trajectory information with the same target position change direction and target position change rate from the trajectory information including the position change direction and the position change rate, and determine the current ship corresponding to the target trajectory information as the preliminarily screened ship; Determine the comprehensive matching score of each preliminarily screened ship by comparing the visual feature information and the target information; Determine the ship with the highest comprehensive matching score among the preliminarily screened ships as the target ship.

[0067] In some specific embodiments, the tracking module 404 is further configured to perform the following operations: Determine the first matching score by comparing the target ship type and the ship type; Determine the second matching score by comparing the target type of goods and the type of goods; Determine the third matching score by comparing the target orientation and the orientation; Determine the fourth matching score between the target color composition and the color composition, the target superstructure position and the superstructure position, and the target superstructure floor number and the superstructure floor number; Determine the comprehensive matching score according to the first matching score, the second matching score, the third matching score, and the fourth matching score.

[0068] It should be understood that the structure illustrated in the embodiments of the present invention does not specifically limit a ship tracking device based on radar-AIS optimized by visual features. In some other embodiments of the present invention, a ship tracking device based on radar-AIS optimized by visual features may include more or fewer components than those illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0069] Regarding the information interaction, execution process, etc. between the various modules within the above-mentioned device, since it is based on the same concept as the method embodiments of the present invention, the specific content can be referred to the description in the method embodiments of the present invention, and will not be elaborated here.

[0070] The embodiments of the present invention further provide a computing device, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, it implements a ship tracking method based on radar-AIS optimized by visual features in any one of the embodiments of the present invention.

[0071] The embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it causes the processor to execute a ship tracking method based on radar-AIS optimized by visual features in any one of the embodiments of the present invention.

[0072] The embodiments of the present application further provide a computer program product, which includes a computer program. The processor of a computer device reads the computer program from a computer-readable storage medium, and the processor executes the computer program, so that the computer device executes a ship tracking method based on radar-AIS optimized by visual features described in any one of the above embodiments.

[0073] Specifically, a system or device equipped with a storage medium can be provided. On the storage medium, software program codes for implementing the functions in any one of the above embodiments are stored, and the computer (or CPU or MPU) of the system or device reads and executes the program codes stored in the storage medium.

[0074] In this case, the program codes read from the storage medium itself can implement the functions of any one of the above embodiments. Therefore, the program codes and the storage medium storing the program codes constitute a part of the present invention.

[0075] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROM. Optionally, the program code can be downloaded from a server computer via a communication network.

[0076] In addition, it should be clear that not only can the actual operations be completed in part or in whole by executing the program code read by the computer, but also by the operating system and the like operating on the computer based on the instructions of the program code, thereby realizing the functions of any one of the above embodiments.

[0077] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion module connected to the computer, and then the CPU and the like installed on the expansion board or the expansion module execute part or all of the actual operations based on the instructions of the program code, thereby realizing the functions of any one of the above embodiments.

[0078] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0079] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes various media such as ROM, RAM, magnetic disks, or optical disks that can store program code.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A ship tracking method for radar-AIS optimized based on visual features, characterized in that, Including: According to the target information of the target ship obtained by radar-AIS, guiding the optoelectronic turret to turn to the azimuth of the target ship; Identifying the current image collected by the optoelectronic turret to obtain a number of current ships; Tracking the current ships and obtaining the visual feature information and track information of each current ship; Determining the target ship from the current ships according to the target information, the visual feature information and the track information for tracking.

2. The method according to claim 1, wherein The guiding the optoelectronic turret to turn to the azimuth of the target ship according to the target information of the target ship obtained by radar-AIS includes: Calculating the azimuth angle and elevation angle of the target ship according to the target longitude and latitude, target distance included in the target information and the longitude and latitude, installation height of the optoelectronic turret; Rotating the optoelectronic turret following the azimuth angle and the elevation angle so that the center of the camera screen on the optoelectronic turret coincides with the guiding azimuth center of the radar-AIS; And / or The tracking the current ships and obtaining the visual feature information and track information of each current ship includes: Using a multi-target tracking algorithm to associate the current ships in the current images at different times to determine the track information of each current ship; wherein, the track information includes the position change direction and the position change rate; Performing feature extraction on the current image to obtain the visual feature information of each current ship; wherein, the visual feature information includes the ship type, color composition, ship house position, number of ship floors, cargo type and orientation.

3. The method according to claim 1, characterized in that, The identifying the current image collected by the optoelectronic turret to obtain a number of current ships includes: Inputting the current image into a target detection model and outputting a prediction box marked with a ship; Screening the prediction boxes according to the size, confidence level and overlap situation of the prediction boxes to obtain a target prediction box marked with the current ship.

4. The method according to claim 3, characterized in that The screening the prediction boxes according to the size, confidence level and overlap situation of the prediction boxes to obtain a target prediction box marked with the current ship includes: After removing the prediction boxes corresponding to the confidence levels lower than the preset confidence threshold, taking the prediction box with the highest current confidence level as the optimal box, and calculating the intersection over union between the optimal box and the first prediction boxes retained after removal; Removing the first prediction boxes corresponding to the intersection over union greater than the first preset threshold, then pairing them in pairs to obtain matching pairs, and calculating the similarity scores of the matching pairs to screen the first prediction boxes based on the similarity scores to obtain second prediction boxes; Screening the second prediction boxes according to the relative position relationship of the bows in the ships to obtain the target prediction boxes.

5. The method according to any one of claims 1 to 4, characterized in that The target information includes the target position change direction, target change rate, target ship type, target color composition, target ship house position, number of target ship floors, target cargo type and target orientation; The visual feature information includes the ship type, color composition, ship house position, number of ship floors, cargo type and orientation; Determining the target ship from the current ships for tracking according to the target information, the visual feature information, and the track information includes: Screening out target track information that is the same as the target position change direction and the target position change rate from the track information including the position change direction and the position change rate, and determining the current ship corresponding to the target track information as the preliminarily screened ship; Determining the comprehensive matching score of each preliminarily screened ship by comparing the visual feature information with the target information; Determining the preliminarily screened ship corresponding to the highest comprehensive matching score as the target ship.

6. The method according to claim 5, wherein The determining the comprehensive matching score of each preliminarily screened ship by comparing the visual feature information with the target information includes: Determining a first matching score of the target ship type and the ship type by comparison; Determining a second matching score of the target cargo type and the cargo type by comparison; Determining a third matching score of the target orientation and the orientation by comparison; Determining a fourth matching score among the target color composition and the color composition, the target superstructure position and the superstructure position, and the target number of ship decks and the number of ship decks by comparison; Determining the comprehensive matching score according to the first matching score, the second matching score, the third matching score, and the fourth matching score.

7. A ship tracking device for radar-AIS optimized based on visual features, characterized in that, Including: A monitoring and guiding module, configured to guide the optoelectronic turret to turn to the azimuth of the target ship according to the target information of the target ship obtained by radar-AIS; An identification module, configured to identify the current image collected by the optoelectronic turret to obtain a plurality of current ships; A tracking module, configured to track the current ships and obtain the visual feature information and the track information of each current ship; And determining the target ship from the current ships for tracking according to the target information, the visual feature information, and the track information.

8. A computing device, including a memory and a processor, where a computer program is stored in the memory, and when the processor executes the computer program, the method according to any one of claims 1-6 is implemented.

9. A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed in a computer, the computer is made to execute the method according to any one of claims 1-6.

10. A computer program product, characterized in that, Including computer instructions, and when the computer instructions are executed by a processor, the steps of the method according to any one of claims 1-6 are implemented.

Images