Non-cooperative ship trajectory dynamic complementing and monitoring system fusing remote sensing image and AIS data

Abstract

The invention discloses a non-cooperative ship trajectory dynamic complementing and monitoring system fusing remote sensing images and AIS data, and relates to the field of ocean monitoring and ship management.The system is characterized in that a space-time range determining module determines two retrieval ranges based on an AIS data sequence of a non-cooperative ship; the target recognition module is used for obtaining labeling box information corresponding to the remote sensing images of the ships at the moments in the two retrieval ranges; the space-time position calculation and space-time matching module is used for determining geographic coordinates of each ship at each moment according to the labeling box information, and carrying out space-time matching to obtain a non-cooperative ship remote sensing image; the image matching module is used for performing image matching on the remote sensing image of the ship to be matched by taking the remote sensing image of the non-cooperative ship as a reference; according to the system and the method, the dynamic completion and monitoring of the tracks of the non-cooperative ships can be realized, and the problem of data missing caused by the closing of the AIS can be solved.

Description

Technical Field

[0001] The present application relates to the field of ocean monitoring and ship management, and in particular to a non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data. Background Art

[0002] The Automatic Identification System (AIS) is the main means of positioning and identification in the current ship management system. The existing ship monitoring system mainly relies on AIS, because AIS can provide information such as the ship's identity, location, heading, speed, etc., to achieve automatic identification and tracking of ships.

[0003] Some ships that turn off AIS due to technical reasons or human factors are called "non-cooperative" ships. For ships that turn off AIS, the traditional AIS-based monitoring method will be affected, resulting in the system's lack of effective monitoring of these "non-cooperative" ships. For ships that lack AIS, radar technology is often used as an auxiliary means to detect the location of the ship. However, the application of radar technology has certain limitations. The detection range of radar is relatively limited, and is limited by the observation angle and image resolution, which makes radar relatively effective in detecting ships in nearshore areas. On the contrary, for ships performing ocean voyages, the detection capability of radar is significantly limited, making it difficult to achieve effective long-distance monitoring. This situation poses significant challenges to maritime safety, supervision, and border protection. Therefore, a system for dynamically completing and monitoring the trajectory of non-cooperative ships is now needed to solve the problem of missing data caused by the closure of AIS, thereby improving the accuracy and real-time nature of monitoring. Summary of the invention

[0004] The purpose of this application is to provide a non-cooperative ship trajectory dynamic completion and monitoring system that integrates remote sensing images and AIS data, which can realize dynamic completion and monitoring of non-cooperative ship trajectories and solve the problem of data missing due to AIS closure.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In the first aspect, the present application provides a non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data, including:

[0007] A time and space range determination module is used to determine a first search range and a second search range based on the AIS data sequence of a non-cooperative ship; the first search range is a time range and a space range where AIS data is not missing; the second search range is a time range and a space range where AIS data is missing; the AIS data sequence of a non-cooperative ship is missing AIS data at at least one time;

[0008] The target recognition module is used to process the remote sensing images of each ship in the first search range and the second search range at each time by using a target detection algorithm to obtain the annotation box information of the remote sensing images of each ship in the first search range and the second search range at each time; the annotation box information includes: the coordinates of the center point, width, height, rotation angle, confidence and category;

[0009] The spatiotemporal position calculation and spatiotemporal matching module is used to determine the geographic coordinates of each ship within the first search range and the second search range at each time according to the annotation frame information of the remote sensing images of each ship within the first search range and the second search range at each time, and to perform spatiotemporal matching on the geographic coordinates of each ship within the first search range at each time with the AIS data sequence of the non-cooperative ship, so as to obtain the remote sensing images of the non-cooperative ship corresponding to each time within the first search range;

[0010] An image matching module is used to perform image matching on each remote sensing image in a remote sensing image set of ships to be matched, using the remote sensing images of non-cooperative ships corresponding to each moment in the first search range as a reference; the remote sensing image set of ships to be matched includes the remote sensing images of each ship in the second search range at each moment;

[0011] The trajectory completion module is used to complete the geographical coordinates and time corresponding to the remote sensing images with successful image matching in the remote sensing image set of the ship to be matched into the AIS data sequence of the non-cooperative ship; the geographical coordinates corresponding to the remote sensing images with successful image matching in the remote sensing image set of the ship to be matched are determined according to the annotation box information of the remote sensing images with successful image matching in the remote sensing image set of the ship to be matched;

[0012] The dynamic monitoring and alarm module is used to obtain the AIS data of non-cooperative ships in real time, and to issue an alarm after obtaining the AIS data of non-cooperative ships.

[0013] Optionally, the non-cooperative ship trajectory dynamic completion and monitoring system that integrates remote sensing images and AIS data also includes: a data acquisition module, used to acquire the AIS data sequence of non-cooperative ships, the remote sensing images of each ship within the first retrieval range at each time, and the remote sensing images of each ship within the second retrieval range at each time.

[0014] Optionally, the spatiotemporal position calculation and spatiotemporal matching module includes:

[0015] A first spatiotemporal position calculation unit is used to determine the geographic coordinates of each ship within the first search range at each time by using a geographic coordinate conversion algorithm according to the labeling frame information corresponding to the remote sensing image of each ship within the first search range at each time and the position coordinates corresponding to the remote sensing image of each ship within the first search range at each time;

[0016] The second spatiotemporal position calculation unit is used to determine the geographic coordinates of each ship within the second search range at each moment according to the annotation box information corresponding to the remote sensing image of each ship within the second search range at each moment and the position coordinates corresponding to the remote sensing image of each ship within the second search range at each moment, using a geographic coordinate conversion algorithm.

[0017] Optionally, the spatiotemporal position calculation and spatiotemporal matching module includes:

[0018] The time-space matching unit is used to perform time-space matching on the time-space data of each ship in the first search range and the AIS data sequence of the non-cooperative ship at any time in the first search range to obtain the ship with successful time-space matching; the time-space data of the ship in the first search range is the geographical coordinates of the ship in the first search range at the time and the time;

[0019] The non-cooperative ship image determination unit is used to determine the remote sensing image of the ship with successful spatiotemporal matching within the first search range at the said time as the non-cooperative ship remote sensing image corresponding to the said time within the first search range.

[0020] Optionally, the target detection algorithm is mmrotate to rotate the target detection framework.

[0021] Optionally, the image matching module includes:

[0022] The image matching unit is used to perform image matching on each remote sensing image in the matching ship remote sensing image set, based on the non-cooperative ship remote sensing image corresponding to any moment in the first search range, and taking the non-cooperative ship remote sensing image corresponding to the moment in the first search range as a reference.

[0023] Optionally, the image matching unit includes:

[0024] The image matching subunit is used to use an image matching algorithm to perform image matching on each remote sensing image in the matching ship remote sensing image set, using the non-cooperative ship remote sensing image corresponding to any moment in the first search range as a benchmark.

[0025] Optionally, the non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data further includes:

[0026] The data storage and preprocessing module is used to store the AIS data of non-cooperative ships in chronological order to obtain the AIS data sequence of non-cooperative ships, and is also used to preprocess the remote sensing images of each ship at each time in the first search range and the remote sensing images of each ship at each time in the second search range.

[0027] Optionally, the non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data further includes:

[0028] The track visualization module is used to visualize the completed AIS data sequence of non-cooperative ships obtained by the track completion module.

[0029] According to the specific embodiments provided in this application, this application has the following technical effects:

[0030] The present application provides a non-cooperative ship trajectory dynamic completion and monitoring system that integrates remote sensing images and AIS data. The remote sensing image data can cover a large range of sea areas, and can realize all-day and all-weather monitoring of the ocean without weather and time restrictions. It can actively monitor ships without AIS signals. The present application can realize dynamic completion of non-cooperative ship trajectories by integrating remote sensing images and AIS data, and can solve the problem of data missing caused by AIS closure, thereby improving the accuracy and real-time nature of monitoring, and realizing monitoring of non-cooperative ships through dynamic monitoring and alarm modules. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 A block diagram of a non-cooperative ship trajectory dynamic completion and monitoring system that integrates remote sensing images and AIS data provided by an embodiment of the present application;

[0033] Figure 2 A flowchart of a non-cooperative ship trajectory dynamic completion and monitoring system that integrates remote sensing images and AIS data is provided in one embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0036] In an exemplary embodiment, Figure 1 As shown, a non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data is provided, including:

[0037] The time and space range determination module is used to determine a first search range and a second search range based on the AIS data sequence of a non-cooperative ship; the first search range is a time range and a space range where AIS data is not missing; the second search range is a time range and a space range where AIS data is missing; and the AIS data sequence of a non-cooperative ship lacks AIS data at least at one time.

[0038] The target recognition module is used to process the remote sensing images of each ship in the first search range and the second search range at each time using the target detection algorithm to obtain the annotation box information of the remote sensing images of each ship in the first search range and the second search range at each time; the annotation box information includes: the coordinates of the center point, width, height, rotation angle, confidence and category. The annotation box information of the remote sensing image is obtained by the target detection algorithm to provide more accurate and efficient input data for the subsequent image matching process.

[0039] The spatiotemporal position calculation and spatiotemporal matching module is used to determine the geographic coordinates of each ship within the first search range and the second search range at each time according to the annotation box information of the remote sensing images of each ship within the first search range and the second search range at each time, and to perform spatiotemporal matching on the geographic coordinates of each ship within the first search range at each time with the AIS data sequence of the non-cooperative ship to obtain the remote sensing images of the non-cooperative ship corresponding to each time within the first search range.

[0040] The image matching module is used to perform image matching on each remote sensing image in a remote sensing image set of ships to be matched, using the remote sensing images of non-cooperative ships corresponding to each moment within the first search range as a benchmark; the remote sensing image set of ships to be matched includes the remote sensing images of each ship at each moment within the second search range.

[0041] The trajectory completion module is used to complete the geographic coordinates and time corresponding to the remote sensing images with successful image matching in the remote sensing image set of the ship to be matched into the AIS data sequence of the non-cooperative ship; the geographic coordinates corresponding to the remote sensing images with successful image matching in the remote sensing image set of the ship to be matched are determined according to the annotation box information of the remote sensing images with successful image matching in the remote sensing image set of the ship to be matched.

[0042] The dynamic monitoring and alarm module is used to obtain the AIS data of non-cooperative ships in real time. When the AIS data of non-cooperative ships is obtained, an alarm is issued. The system provides dynamic monitoring function based on data completion. If a non-cooperative ship is detected, an alarm is triggered immediately.

[0043] In another exemplary embodiment of the present application, the non-cooperative ship trajectory dynamic completion and monitoring system that integrates remote sensing images and AIS data also includes: a data acquisition module, used to acquire the AIS data sequence of non-cooperative ships, the remote sensing images of each ship within the first retrieval range at each time, and the remote sensing images of each ship within the second retrieval range at each time.

[0044] In another exemplary embodiment of the present application, the spatiotemporal position calculation and spatiotemporal matching module includes:

[0045] The first spatiotemporal position calculation unit is used to determine the geographic coordinates of each ship in the first search range at each time according to the annotation frame information corresponding to the remote sensing image of each ship in the first search range at each time and the position coordinates corresponding to the remote sensing image of each ship in the first search range at each time (the position coordinates of the remote sensing image) by using a geographic coordinate conversion algorithm. This process is a well-known process, and the general steps are: first, the pixel coordinates of each ship in the remote sensing image are obtained through the annotation frame information, and then the GDAL geospatial data processing library and the remote sensing image are processed using the geographic coordinate conversion algorithm to convert the pixel coordinates into WGS84 geographic coordinates.

[0046] The second spatiotemporal position calculation unit is used to determine the geographic coordinates of each ship within the second search range at each moment according to the annotation box information corresponding to the remote sensing image of each ship within the second search range at each moment and the position coordinates corresponding to the remote sensing image of each ship within the second search range at each moment, using a geographic coordinate conversion algorithm.

[0047] In another exemplary embodiment of the present application, the spatiotemporal position calculation and spatiotemporal matching module includes:

[0048] The space-time matching unit is used to perform space-time matching on the space-time data of each ship in the first search range and the AIS data sequence of the non-cooperative ship at any time in the first search range to obtain the ship with successful space-time matching; the space-time data of the ship in the first search range is the geographical coordinates of the ship in the first search range at the time and the time. Specifically, the space-time data of each ship in the first search range is matched with the AIS data sequence of the non-cooperative ship to determine whether each ship in the first search range meets the trajectory characteristics of the non-cooperative target. If the space-time matching result corresponding to a certain ship is within a certain threshold (the longitude matching range is set to 0.01, the latitude matching range is set to 0.01, and the time matching range is set to 60 seconds), the ship is determined to be a space-time matching ship, otherwise it is discarded. One moment corresponds to a space-time matching ship.

[0049] The non-cooperative ship image determination unit is used to determine the remote sensing image of the ship with successful spatiotemporal matching within the first search range at the said time as the non-cooperative ship remote sensing image corresponding to the said time within the first search range.

[0050] In another exemplary embodiment of the present application, the target detection algorithm is mmrotate rotating target detection framework.

[0051] In another exemplary embodiment of the present application, the image matching module includes:

[0052] The image matching unit is used to perform image matching on each remote sensing image in the remote sensing image set of the matching ship, based on the non-cooperative ship remote sensing image corresponding to any time in the first search range, and determine the image matching degree between the two according to the matching degree evaluation standard.

[0053] In another exemplary embodiment of the present application, the image matching unit includes:

[0054] The image matching subunit is used to use an image matching algorithm to perform image matching on each remote sensing image in the matching ship remote sensing image set, using the non-cooperative ship remote sensing image corresponding to any moment in the first search range as a benchmark.

[0055] In another exemplary embodiment of the present application, the non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data further includes:

[0056] The data storage and preprocessing module is used to store the AIS data of non-cooperative ships in chronological order to obtain the AIS data sequence of non-cooperative ships, establish the time and space range index, and also to preprocess the remote sensing images of each ship at each time in the first retrieval range and the remote sensing images of each ship at each time in the second retrieval range (image denoising, enhancement and ship target detection to ensure the accuracy of target recognition).

[0057] In another exemplary embodiment of the present application, the non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data further includes:

[0058] The track visualization module is used to visualize the completed AIS data sequence of non-cooperative ships obtained by the track completion module.

[0059] This application can solve the problem of missing AIS data by dynamically completing the trajectory of non-cooperative ships through remote sensing image recognition and positioning.

[0060] This application achieves accurate dynamic monitoring of multi-source data through time-space matching and data fusion technology.

[0061] This application supports real-time data updates, alarm push, and trajectory visualization, allowing managers to monitor ship movements in real time.

[0062] This application uses remote sensing images to supplement the missing AIS data, effectively improving the tracking capability and response speed of the ship monitoring system to uncooperative ships. The real-time alarm function of the system provides strong support for maritime supervision.

[0063] like Figure 2 As shown, the workflow of the system is to obtain the original AIS data, i.e., the above-mentioned AIS data sequence; perform data conversion (storage by day); determine the spatiotemporal range of the known trajectory, and determine the first search range according to the spatiotemporal range of the known trajectory; retrieve the remote sensing images within the first search range; perform ship target detection on the ships within the first search range based on the remote sensing images within the first search range; calculate the spatiotemporal positions of the ships within the first search range based on the ship target detection results within the first search range; perform spatiotemporal matching based on the calculation results to obtain the target ship images at each moment (i.e., the non-cooperative ship remote sensing images mentioned above).

[0064] Determine the spatiotemporal range of the unknown trajectory, and determine a second search range according to the spatiotemporal range of the unknown trajectory; retrieve remote sensing images within the second search range; perform ship target detection on ships within the second search range based on the remote sensing images within the second search range; calculate the spatiotemporal positions of ship targets within the second search range based on the ship target detection results within the second search range; determine the remote sensing images within the second search range as the ship images to be matched.

[0065] The target ship image at any moment is matched with the image of the ship to be matched, and it is determined whether the match is successful. If successful, the time and space position corresponding to the image of the ship to be matched with the successful image matching is completed in the original AIS data, and then it is determined whether the target ship images at each moment have been matched. If there are unmatched ones, one of the unmatched target ship images is selected to match the image of the ship to be matched, until the target ship images at each moment have been matched. If the match is not successful, it is determined whether the target ship images at each moment have been matched. If there are unmatched ones, one of the unmatched target ship images is selected to match the image of the ship to be matched, until the target ship images at each moment have been matched.

[0066] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0067] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0068] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0069] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data, characterized in that: The non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data includes: A time and space range determination module is used to determine a first search range and a second search range based on the AIS data sequence of a non-cooperative ship; the first search range is a time range and a space range where AIS data is not missing; the second search range is a time range and a space range where AIS data is missing; the AIS data sequence of a non-cooperative ship is missing AIS data at at least one time; The target recognition module is used to process the remote sensing images of each ship in the first search range and the second search range at each time by using a target detection algorithm to obtain the annotation box information of the remote sensing images of each ship in the first search range and the second search range at each time; the annotation box information includes: the coordinates of the center point, width, height, rotation angle, confidence and category; The spatiotemporal position calculation and spatiotemporal matching module is used to determine the geographic coordinates of each ship within the first search range and the second search range at each time according to the annotation frame information of the remote sensing images of each ship within the first search range and the second search range at each time, and to perform spatiotemporal matching on the geographic coordinates of each ship within the first search range at each time with the AIS data sequence of the non-cooperative ship, so as to obtain the remote sensing images of the non-cooperative ship corresponding to each time within the first search range; An image matching module is used to perform image matching on each remote sensing image in a remote sensing image set of ships to be matched, using the remote sensing images of non-cooperative ships corresponding to each moment in the first search range as a reference; the remote sensing image set of ships to be matched includes the remote sensing images of each ship in the second search range at each moment; The trajectory completion module is used to complete the geographical coordinates and time corresponding to the remote sensing images with successful image matching in the remote sensing image set of the ship to be matched into the AIS data sequence of the non-cooperative ship; the geographical coordinates corresponding to the remote sensing images with successful image matching in the remote sensing image set of the ship to be matched are determined according to the annotation box information of the remote sensing images with successful image matching in the remote sensing image set of the ship to be matched; The dynamic monitoring and alarm module is used to obtain the AIS data of non-cooperative ships in real time, and to issue an alarm after obtaining the AIS data of non-cooperative ships.

2. The non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data according to claim 1 is characterized in that: The non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data also includes: a data acquisition module, which is used to acquire the AIS data sequence of the non-cooperative ship, the remote sensing image of each ship in the first search range at each time, and the remote sensing image of each ship in the second search range at each time.

3. The non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data according to claim 1 is characterized in that: The spatiotemporal position calculation and spatiotemporal matching module includes: A first spatiotemporal position calculation unit is used to determine the geographic coordinates of each ship within the first search range at each time by using a geographic coordinate conversion algorithm according to the labeling frame information corresponding to the remote sensing image of each ship within the first search range at each time and the position coordinates corresponding to the remote sensing image of each ship within the first search range at each time; The second spatiotemporal position calculation unit is used to determine the geographic coordinates of each ship within the second search range at each moment according to the annotation box information corresponding to the remote sensing image of each ship within the second search range at each moment and the position coordinates corresponding to the remote sensing image of each ship within the second search range at each moment, using a geographic coordinate conversion algorithm.

4. The non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data according to claim 1 is characterized in that: The spatiotemporal position calculation and spatiotemporal matching module includes: The time-space matching unit is used to perform time-space matching on the time-space data of each ship in the first search range and the AIS data sequence of the non-cooperative ship at any time in the first search range to obtain the ship with successful time-space matching; the time-space data of the ship in the first search range is the geographical coordinates of the ship in the first search range at the time and the time; The non-cooperative ship image determination unit is used to determine the remote sensing image of the ship with successful spatiotemporal matching within the first search range at the said time as the non-cooperative ship remote sensing image corresponding to the said time within the first search range.

5. The non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data according to claim 1 is characterized in that: The target detection algorithm is mmrotate rotation target detection framework.

6. The non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data according to claim 1 is characterized in that: The image matching module comprises: The image matching unit is used to perform image matching on each remote sensing image in the matching ship remote sensing image set, based on the non-cooperative ship remote sensing image corresponding to any moment in the first search range, and taking the non-cooperative ship remote sensing image corresponding to the moment in the first search range as a reference.

7. The non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data according to claim 6 is characterized in that: The image matching unit comprises: The image matching subunit is used to use an image matching algorithm to perform image matching on each remote sensing image in the matching ship remote sensing image set, using the non-cooperative ship remote sensing image corresponding to any moment in the first search range as a benchmark.

8. The non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data according to claim 1 is characterized in that: The non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data also includes: The data storage and preprocessing module is used to store the AIS data of non-cooperative ships in chronological order to obtain the AIS data sequence of non-cooperative ships, and is also used to preprocess the remote sensing images of each ship at each time in the first search range and the remote sensing images of each ship at each time in the second search range.

9. The non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data according to claim 1, characterized in that: The non-cooperative ship trajectory dynamic completion and monitoring system integrating remote sensing images and AIS data also includes: The track visualization module is used to visualize the completed AIS data sequence of non-cooperative ships obtained by the track completion module.

Images