Ship target recognition method and system based on UAV-borne multi-sensor data

Abstract

The present invention provides a ship target recognition method and system based on multi-sensor data onboard a drone, belonging to the technical field of ship safety supervision, and the method includes: obtaining multi-sensor data onboard a drone, the multi-sensor data onboard a drone including AIS data, ship visual data and IMU / GNSS data; establishing a coordinate conversion relationship between a coordinate system where the ship visual data is located and a coordinate system where the AIS data is located based on the IMU / GNSS data, and performing coordinate system conversion on the AIS data based on the coordinate conversion relationship to obtain AIS conversion data; determining the association result between the ship visual data and the AIS conversion data based on the ship position constraint, the ship heading constraint and the camera field of view constraint, and fusing the ship visual data and the AIS conversion data based on the association result to obtain a ship target recognition result. The present invention realizes the spatial co-dimension of the AIS data and the ship visual data, and improves the accuracy and precision of the ship target recognition result.

Description

Technical Field

[0001] The present invention relates to the technical field of ship safety supervision, and in particular to a ship target recognition method and system based on unmanned aerial vehicle (UAV) airborne multi-sensor data. Background Art

[0002] Drones have been widely used in the field of maritime supervision due to their advantages such as high-altitude perspective, easy to carry, simple operation, and flexible maneuverability. The single sensors commonly used in current drones have problems such as limited detection dimensions and low perception accuracy, making it difficult to obtain complete feature information of surface targets. For example, the identity and motion information of a ship cannot be obtained only through the visual data of the ship; the ship automatic identification system (AIS) can send static information (MMSI code, ship type, ship name, etc.) and dynamic information (latitude and longitude, speed, heading, etc.) of the ship, but the visual information of the ship cannot be directly obtained only through AIS data. Relying on existing perception methods is difficult to meet the growing demand of drone platforms for situational awareness of water traffic safety supervision. The fusion of ship vision and AIS data can simultaneously obtain the visual, identity and motion information of the ship, providing more efficient and comprehensive perception data for maritime monitoring tasks. Therefore, it is necessary to study new technologies for maritime perception equipment and drone intelligent perception, which is also an important development direction for improving the efficiency of maritime intelligent supervision.

[0003] Existing ship target recognition methods mainly face the following two problems: the airborne multi-sensor data has the problem of different spatial dimensions; it is difficult to achieve accurate fusion effect by only considering the single attribute of the ship, which leads to low accuracy of ship target recognition. Summary of the invention

[0004] In view of this, it is necessary to provide a ship target recognition method and system based on UAV airborne multi-sensor data to solve the technical problems in the prior art that the airborne multi-sensor data has different spatial dimensions and it is difficult to achieve accurate fusion effect by considering only a single attribute of the ship, resulting in low accuracy of ship target recognition.

[0005] On the one hand, in order to solve the above technical problems, the present invention provides a ship target recognition method based on multi-sensor data onboard a drone, comprising:

[0006] Acquire multi-sensor data onboard the drone, wherein the multi-sensor data onboard the drone includes AIS data, ship vision data, and IMU / GNSS data;

[0007] Establishing a coordinate conversion relationship between a coordinate system where the ship visual data is located and a coordinate system where the AIS data is located based on the IMU / GNSS data, and performing a coordinate conversion on the AIS data based on the coordinate conversion relationship to obtain AIS conversion data;

[0008] The association result of the ship visual data and the AIS conversion data is determined based on the ship position constraint, the ship heading constraint and the camera field of view constraint, and the ship visual data and the AIS conversion data are fused based on the association result to obtain a ship target recognition result.

[0009] In a possible implementation, the coordinate system of the ship visual data is a pixel coordinate system, the coordinate system of the AIS data is a WGS84 coordinate system, the AIS data includes the ship longitude, ship latitude, speed, heading and ship name, the GNSS data includes the drone longitude, drone latitude, drone altitude, and the IMU data includes the azimuth and pitch angle of the airborne camera; then the coordinate conversion relationship between the coordinate system of the ship visual data and the coordinate system of the AIS data is established based on the IMU / GNSS data, including:

[0010] Determine a first conversion relationship between the WGS84 coordinate system and the world coordinate system based on the ship longitude, the ship latitude, the drone longitude and the drone latitude;

[0011] Determine a second transformation relationship between the world coordinate system and the body coordinate system, a third transformation relationship between the camera coordinate system and the body coordinate system, a fourth transformation relationship between the camera coordinate system and the image coordinate system, and a fifth transformation relationship between the image coordinate system and the pixel coordinate system;

[0012] The coordinate transformation relationship is determined based on the first transformation relationship, the second transformation relationship, the third transformation relationship, the fourth transformation relationship, and the fifth transformation relationship.

[0013] In a possible implementation manner, determining the coordinate transformation relationship based on the first transformation relationship, the second transformation relationship, the third transformation relationship, the fourth transformation relationship, and the fifth transformation relationship includes:

[0014] Determine an initial coordinate transformation relationship based on the first transformation relationship, the second transformation relationship, the third transformation relationship, the fourth transformation relationship, and the fifth transformation relationship;

[0015] Obtain camera distortion parameters, and calibrate the initial coordinate transformation relationship based on the camera distortion parameters to obtain the coordinate transformation relationship.

[0016] In a possible implementation, the ship position constraint for:

[0017]

[0018]

[0019] The ship heading constraint for:

[0020]

[0021] The camera field of view constraint is: filtering out the AIS conversion data outside the camera field of view;

[0022] In the formula, is the ship position in the i-th AIS conversion data; is the coordinate of the center point of the rotating frame in the jth ship visual data; is the Euclidean distance between the position coordinates of the same ship in the i-th AIS data and the j-th ship visual data; is the distance threshold; is the ship heading in the i-th AIS conversion data; is the ship heading in the jth ship visual data; is the angle threshold.

[0023] In a possible implementation, the determining the association result between the ship visual data and the AIS conversion data based on the ship position constraint, the ship heading constraint and the camera field of view constraint includes:

[0024] Filtering the AIS conversion data based on the camera field of view constraint to obtain target AIS conversion data;

[0025] Determining a ship position constraint matrix between the ship visual data and the target AIS conversion data based on the ship position constraint;

[0026] Determine a ship heading constraint matrix between the ship visual data and the target AIS conversion data based on the ship heading constraint;

[0027] A comprehensive constraint matrix is ​​determined based on the ship position constraint matrix and the ship heading constraint matrix, and a correlation result between the ship visual data and the AIS conversion data is determined based on the comprehensive constraint matrix.

[0028] In a possible implementation, the number of rows of the comprehensive constraint matrix is ​​the total number of the target AIS conversion data, and the number of columns of the comprehensive constraint matrix is ​​the total number of the ship visual data; and determining the association result between the ship visual data and the AIS conversion data based on the comprehensive constraint matrix includes:

[0029] Determining the minimum non-zero value in each row of the comprehensive constraint matrix;

[0030] The successfully associated target ship visual data and target AIS conversion data are determined based on the row number and column number corresponding to the minimum value.

[0031] In a possible implementation, the ship target recognition method based on multi-sensor data onboard a drone also includes:

[0032] Determining, based on the association result, matching AIS data in the AIS data that is successfully associated with the ship visual data;

[0033] The ship visual data is displayed, and the matching AIS data is superimposed and displayed on the ship visual data.

[0034] In a possible implementation, after acquiring the multi-sensor data onboard the drone, the method further includes:

[0035] The multi-sensor data on the drone is synchronized with the hardware time and the software time.

[0036] In a possible implementation, after the multi-sensor data onboard the drone, the method further includes:

[0037] The AIS data is decoded and cleaned.

[0038] On the other hand, the present invention also provides a ship target recognition system based on multi-sensor data onboard a drone, comprising:

[0039] A multi-sensor data acquisition unit, used to acquire multi-sensor data onboard the UAV, wherein the multi-sensor data onboard the UAV includes AIS data, ship vision data, and IMU / GNSS data;

[0040] A multi-sensor data conversion unit, used to establish a coordinate conversion relationship between a coordinate system where the ship visual data is located and a coordinate system where the AIS data is located based on the IMU / GNSS data, and to perform coordinate system conversion on the AIS data based on the coordinate conversion relationship to obtain AIS conversion data;

[0041] The multi-sensor data association and matching unit is used to determine the association result of the ship visual data and the AIS conversion data based on the ship position constraint, the ship heading constraint and the camera field of view constraint, and fuse the ship visual data and the AIS conversion data based on the association result to obtain a ship target recognition result.

[0042] The beneficial effects of the present invention are as follows: the ship target recognition method based on the multi-sensor data onboard the drone provided by the present invention obtains the multi-sensor data onboard the drone. Compared with the technology of ship target recognition through the ship end and the shore end in the prior art, the application of drones in maritime scenes is realized, and it has the specificity and novelty of high-altitude perspective, easy to carry, simple operation, flexibility and maneuverability. Furthermore, by setting it to convert the AIS data into a coordinate system based on the established coordinate conversion relationship before associating the multi-sensor data onboard the drone, the AIS conversion data is obtained, and the AIS data and the ship visual data are unified in the same coordinate space, so as to realize the same dimension of the airspace of the AIS data and the ship visual data. In addition, when associating the ship visual data and the AIS conversion data, the present invention takes into account the three constraints of ship position constraint, ship heading constraint and camera field of view constraint, so that the association result is more accurate, thereby improving the accuracy and precision of the ship target recognition result. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A schematic flow chart of an embodiment of a method for ship target recognition based on multi-sensor data onboard a drone provided by the present invention;

[0045] Figure 2 A schematic diagram of a flow chart of an embodiment of establishing a coordinate transformation relationship provided by the present invention;

[0046] Figure 3 A schematic diagram of each coordinate system provided by the present invention;

[0047] Figure 4 For the present invention Figure 2 A schematic flow chart of an embodiment of S203;

[0048] Figure 5 For the present invention Figure 1 A schematic diagram of an embodiment of determining the association result in S103;

[0049] Figure 6 A schematic diagram of an embodiment of data visualization provided by the present invention;

[0050] Figure 7 A schematic diagram of an embodiment of the visualization result provided by the present invention;

[0051] Figure 8A schematic diagram of a specific embodiment of the ship target recognition method based on multi-sensor data onboard a drone provided by the present invention;

[0052] Fig. 9 A schematic diagram of the structure of an embodiment of a ship target recognition system based on multi-sensor data onboard a drone provided by the present invention;

[0053] Fig.10 A schematic structural diagram of an embodiment of a ship target identification device provided by the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0055] It should be understood that the schematic drawings are not drawn to scale. The flowchart used in the present invention shows the operations implemented according to some embodiments of the present invention. It should be understood that the operations of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, under the guidance of the content of the present invention, can add one or more other operations to the flowchart, and can also remove one or more operations from the flowchart. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor systems and / or microcontroller systems.

[0056] Reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0057] The present invention provides a ship target recognition method and system based on unmanned aerial vehicle (UAV) airborne multi-sensor data, which are described below respectively.

[0058] Figure 1 The present invention provides a flow chart of an embodiment of a method for identifying a ship target based on multi-sensor data onboard a drone, such as Figure 1As shown, the ship target recognition method based on UAV airborne multi-sensor data includes:

[0059] S101, obtaining drone airborne multi-sensor data, which includes AIS (Automatic Identification System) data, ship vision data, and IMU (Inertial Measurement Unit) / GNSS (Global Navigation Satellite System) data;

[0060] S102, establishing a coordinate conversion relationship between a coordinate system where the ship visual data is located and a coordinate system where the AIS data is located based on the IMU / GNSS data, and performing coordinate conversion on the AIS data based on the coordinate conversion relationship to obtain AIS conversion data;

[0061] S103, determining the association result of the ship visual data and the AIS conversion data based on the ship position constraint, the ship heading constraint and the camera field of view constraint, and fusing the ship visual data and the AIS conversion data based on the association result to obtain a ship target recognition result.

[0062] The method for acquiring the multi-sensor data onboard the drone in step S101 is: acquiring the multi-sensor data onboard the drone based on the visual perception module, AIS receiving module, GNSS sensor and IMU sensor integrated on the drone. Specifically, the visual perception module is a visual camera, and the visual data of the ship is collected based on the visual camera.

[0063] It should be noted that the ship visual data includes the ship frame determined based on the visual camera. The specific process of collecting ship visual data based on the visual camera is as follows: first, the visual camera is calibrated, that is, the camera imaging model is established and distortion correction is performed to obtain the camera internal parameter matrix and distortion parameters. Then, the ship visual image data is processed using rotating target detection and the SORT tracking method improved based on the rotating frame, the ship in the ship visual image data is identified, and the rotating frame of the ship is obtained.

[0064] It should be understood that the camera imaging model refers to a model for converting a world coordinate system into a pixel coordinate system.

[0065] Among them, GNSS data includes the latitude, longitude and altitude of the drone, IMU data includes the azimuth and pitch angle of the camera, and AIS data includes MMSI number, ship type, ship name, longitude and latitude, heading and speed, etc.

[0066] It should be noted that: in step S103, the ship visual data and the AIS conversion data are fused based on the association result to obtain the ship target recognition result: if the association result is a successful association, the AIS data corresponding to the successfully associated ship visual data and the AIS conversion data are synchronously obtained and superimposed, and the superimposed data is the fused data of the ship.

[0067] Compared with the prior art, the ship target recognition method based on the drone airborne multi-sensor data provided by the embodiment of the present invention obtains the drone airborne multi-sensor data. Compared with the technology of ship target recognition through the ship end and the shore end in the prior art, the application of drones in maritime scenes is realized, and it has the specificity and novelty of high-altitude perspective, easy to carry, simple operation, flexibility and maneuverability. Furthermore, by setting it to convert the AIS data into a coordinate system based on the established coordinate conversion relationship before associating the drone airborne multi-sensor data, obtain the AIS conversion data, unify the AIS data and the ship visual data into the same coordinate space, and realize the airspace dimension of the AIS data and the ship visual data. In addition, when associating the ship visual data and the AIS conversion data, the embodiment of the present invention takes into account the three constraints of ship position constraint, ship heading constraint and camera field of view constraint, so that the association result is more accurate, thereby improving the accuracy and precision of the ship target recognition result.

[0068] In some embodiments of the present invention, the coordinate system of the ship visual data is a pixel coordinate system, the coordinate system of the AIS data is a WGS84 coordinate system, the AIS data includes the ship longitude and ship latitude, the GNSS data includes the UAV longitude, UAV latitude, and UAV altitude, and the IMU data includes the azimuth and pitch angles of the airborne camera.

[0069] If Figure 2 As shown, the step S102 of establishing the coordinate conversion relationship between the coordinate system where the ship visual data is located and the coordinate system where the AIS data is located includes:

[0070] S201, determining a first conversion relationship between the WGS84 coordinate system and the world coordinate system based on the ship longitude, the ship latitude, the drone longitude, and the drone latitude;

[0071] S202, determining a second transformation relationship between the world coordinate system and the body coordinate system, a third transformation relationship between the camera coordinate system and the body coordinate system, a fourth transformation relationship between the camera coordinate system and the image coordinate system, and a fifth transformation relationship between the image coordinate system and the pixel coordinate system;

[0072] S203: Determine a coordinate transformation relationship based on the first transformation relationship, the second transformation relationship, the third transformation relationship, the fourth transformation relationship, and the fifth transformation relationship.

[0073] In a specific embodiment of the present invention, the WGS84 coordinate system, the world coordinate system, the body coordinate system, the camera coordinate system, the image coordinate system and the pixel coordinate system are as follows: Figure 3 shown.

[0074] Among them, the first conversion relationship is:

[0075]

[0076]

[0077]

[0078]

[0079] In the formula, , , , are the ship latitude, ship longitude, drone latitude and drone longitude in the world coordinate system; Indicates the azimuth from the starting point to the end point, with due north as 0° and clockwise as positive; represents the radius of the earth, which is 6371 kilometers.

[0080] In the world coordinate system, the first conversion relationship between the WGS84 coordinate system and the world coordinate system can be obtained according to the distance and azimuth between the ship and the UAV.

[0081] The transformation between the world coordinate system and the fuselage coordinate system is a rigid transformation, and only translation and rotation transformations need to be performed between the coordinate systems. First, the world coordinate system is rotated around its X-axis, Y-axis, and Z-axis at different angles to obtain the corresponding rotation matrix, and then a translation vector is added. The final second transformation relationship is:

[0082]

[0083] In the formula, is the identity matrix; is the rotation matrix between the world coordinate system and the body coordinate system; is the translation vector between the world coordinate system and the fuselage coordinate system; , , is the fuselage coordinate system X , Y , Z Axis coordinates, , , In the world coordinate system X , Y , Z Axis coordinates.

[0084] Among them, the fuselage coordinate system is based on the center point of the drone is the origin, and each axis is the same as the camera coordinate system. The position error between the body coordinate system and the camera coordinate system is:

[0085]

[0086] in,( ， ， ) is the camera installation error angle, is the spatial distance from the camera to the center point of the drone. Therefore, the third transformation relationship between the camera coordinate system and the fuselage coordinate system is:

[0087]

[0088] Among them, the conversion between the camera coordinate system and the image coordinate system. The conversion from the camera coordinate system to the image coordinate system is a conversion from three-dimensional to two-dimensional coordinates, which belongs to perspective projection. If the distance from the image plane to the optical center (focal length) is , then from the triangle similarity relationship, we can infer that the fourth conversion relationship is:

[0089]

[0090] The conversion between the image coordinate system and the pixel coordinate system is that the two are ideally located in the same image plane, but the coordinate origin position and unit are different. The origin of the image coordinate system is the focus of the optical axis and the image plane. , in millimeters ( ); The origin of the pixel coordinate system is usually in the upper left corner of the image The unit is pixel ( ). Ideally, the fifth transformation relationship from the image coordinate system to the pixel coordinate system is as follows:

[0091]

[0092]

[0093] in, Represents an image List OK, and Indicates the length of each column and row.

[0094] Combining the six transformation relationships between the aforementioned coordinate systems, the coordinate transformation relationship between the world coordinate system and the pixel coordinate system under ideal conditions is as follows:

[0095]

[0096] Among them, the camera intrinsic parameter matrix and the external parameter matrix The expressions are:

[0097]

[0098]

[0099] Considering that it is difficult to ensure that the thickness of the lens is completely uniform during the manufacturing process of the camera, camera distortion may occur. In order to solve this technical problem, in some embodiments of the present invention, Figure 4 As shown, step S203 includes:

[0100] S401, determining an initial coordinate transformation relationship based on a first transformation relationship, a second transformation relationship, a third transformation relationship, a fourth transformation relationship, and a fifth transformation relationship;

[0101] S402: Obtain camera distortion parameters, and calibrate the initial coordinate transformation relationship based on the camera distortion parameters to obtain a coordinate transformation relationship.

[0102] The embodiment of the present invention can eliminate the influence of camera distortion on the coordinate transformation relationship by calibrating the initial coordinate transformation relationship based on the camera distortion parameters after the initial coordinate transformation relationship is determined, ensure the accuracy of the coordinate transformation relationship, and further ensure the accuracy of ship target recognition.

[0103] In a specific embodiment of the present invention, the camera distortion parameters include k 1 and k 2 Parameters are used to calibrate the distortion of the image center and edge areas respectively.

[0104] In some embodiments of the present invention, the vessel position constraint for:

[0105]

[0106]

[0107] Ship heading constraints for:

[0108]

[0109] The camera field of view constraint is: filter out the AIS conversion data outside the camera field of view;

[0110] In the formula, is the ship position in the i-th AIS conversion data; is the coordinate of the center point of the rotating frame in the jth ship visual data; is the Euclidean distance between the position coordinates of the same ship in the i-th AIS data and the j-th ship visual data; is the distance threshold; is the ship heading in the i-th AIS conversion data; is the ship heading in the jth ship visual data; is the angle threshold.

[0111] Among them, the ship heading in the ship visual data The angle corresponding to the long side of the ship's rotation frame. Ship heading in AIS data The calculation process is as follows: First, record the i-th AIS data The ship heading provided in , which represents the ship's heading and true north in the world coordinate system ( ). In the world coordinate system, the position of the ship is , take any point in the ship's heading direction and record it as ,but:

[0112]

[0113] in, is a constant and can be any number except 0. Then, M and N The two points are converted from the world coordinate system to the pixel coordinate system, recorded as and ,but Line and axis The angle between is the ship heading in the AIS conversion data, then for:

[0114] .

[0115] In some embodiments of the present invention, Figure 5 As shown, in step S103, the association result of the ship visual data and the AIS conversion data is determined based on the ship position constraint, the ship heading constraint and the camera field of view constraint, including:

[0116] S501, filtering AIS conversion data based on camera field of view constraints to obtain target AIS conversion data;

[0117] S502, determining a ship position constraint matrix between ship visual data and target AIS conversion data based on the ship position constraint;

[0118] S503, determining a ship heading constraint matrix between the ship visual data and the target AIS conversion data based on the ship heading constraint;

[0119] S504: Determine a comprehensive constraint matrix based on the ship position constraint matrix and the ship heading constraint matrix, and determine a correlation result between the ship visual data and the AIS conversion data based on the comprehensive constraint matrix.

[0120] The embodiment of the present invention filters out the AIS data outside the camera field of view based on the camera field of view constraint, and only retains the AIS data within the camera field of view, thereby improving the efficiency of the subsequent target matching process. Subsequently, based on the ship heading constraint and the ship position constraint, the AIS data and the ship visual data can be matched and associated from two dimensions of heading and position, thereby improving the accuracy and reliability of the matching and association results.

[0121] It should be noted that the number of rows of the comprehensive constraint matrix is ​​the total number of target AIS conversion data, and the number of columns of the comprehensive constraint matrix is ​​the total number of ship visual data.

[0122] In the embodiment of the present invention, the total number of ship visual data and target AIS conversion data is used as the number of rows and columns of the comprehensive constraint matrix. Since the rows and columns are one-to-one corresponding, the one-to-one correspondence of the associated matching results can be achieved, thereby further ensuring the accuracy of the matching results.

[0123] In a specific embodiment of the present invention, step S504 is specifically:

[0124] Determine the minimum non-zero value in each row of the comprehensive constraint matrix; determine the successfully associated target ship visual data and target AIS conversion data based on the row number and column number corresponding to the minimum value.

[0125] Specifically: The comprehensive constraint matrix is , determine the minimum value in each row of the matrix, and record the column where the minimum value of each row is located, recorded as :

[0126]

[0127] in, The function here specifically refers to finding the minimum value of each row of the matrix; and Should be one-to-one correspondence, indicating that each ship visual target ( ) has only one AIS target ( ) to match it, that is, the correlation matching of the UAV’s onboard multi-sensor data is realized.

[0128] To realize the visualization of all ship data, in some embodiments of the present invention, Figure 6As shown, the ship target recognition method based on the UAV airborne multi-sensor data also includes:

[0129] S601, determining matching AIS data in the AIS data that is successfully associated with the ship visual data based on the association result;

[0130] S602: Display the ship visual data, and overlay the matching AIS data on the ship visual data.

[0131] Wherein, the ship visual data is a ship visual image, and step S502 specifically includes: directly displaying the ship visual image, and superimposing the matching AIS data on the ship visual image.

[0132] In a specific embodiment of the present invention, the visual display result is as follows: Figure 7 As shown, there are three ships in the figure, which will be matched with AIS data such as MMSI, ship type ( Figure 7 Type in), ship name ( Figure 7 Name in), longitude and latitude ( Figure 7 Lon, Lat), heading and speed ( Figure 7 Parameters such as Speed ​​in the display are superimposed.

[0133] Since the multi-sensor data onboard the drone still has the problem of time domain asynchrony, in order to avoid erroneous association and matching of ship data, in some embodiments of the present invention, after step S101, the following is further included:

[0134] Perform hardware and software time synchronization on multi-sensor data on drones.

[0135] The embodiment of the present invention eliminates the technical problem of time domain asynchrony of the drone's onboard multi-sensor data to the greatest extent by performing hardware time synchronization and software time synchronization on the drone's onboard multi-sensor data, further ensures the accuracy of the association matching results, and further improves the accuracy of ship identification.

[0136] Among them, hardware time synchronization is specifically: the time synchronization function is realized through the TIMESYNC pin on the drone, and the data of multiple sensors on board are synchronized to the same clock source using the hardware synchronization signal. Specifically, the drone time is synchronized to UTC time using the second pulse signal of the drone GNSS sensor, and the precise UTC time of the rising edge of the time synchronization output pulse is provided. Finally, this UTC timestamp is applied to the data collected by other sensors to achieve accurate hardware time synchronization.

[0137] Among them, software time synchronization specifically includes: aligning the ship visual data timestamp and AIS data timestamp.

[0138] Since the format of AIS data is determined by the ship automatic identification system, in order to facilitate subsequent analysis of AIS data, in some embodiments of the present invention, after step S101, the following is further included:

[0139] Decode and clean the AIS data.

[0140] Specifically, the AIS data is decoded according to the preset structured recording requirements.

[0141] Data cleaning includes cleaning duplicate data and erroneous data. When there are two or more sets of completely identical data, only one set of data is retained; according to common sense of ship navigation, the values ​​containing erroneous data are found and deleted.

[0142] The embodiment of the present invention decodes and cleans the AIS data, so that the AIS data is easy to store and analyze in a structured manner, and the data accuracy is improved, thereby ensuring the accuracy of subsequent matching association.

[0143] In summary, the process of the ship target recognition method based on multi-sensor data onboard a drone provided by the embodiment of the present invention is as follows: Figure 8 As shown in the figure, firstly, the ship visual data, AIS data, GNSS data and IMU data are collected based on the multi-source sensors on the UAV. The ship visual data and AIS data need to be preprocessed. The preprocessing process of the ship visual data is as follows: first, the visual camera is calibrated, that is, the camera imaging model is established and the distortion correction is performed to obtain the camera internal parameter matrix and distortion parameters. Then, the ship visual image data is processed by rotating target detection and the SORT tracking method improved based on the rotating frame. The AIS data preprocessing includes data decoding and data cleaning. After the preprocessing of the two types of data is completed, the multi-sensor data on the UAV is synchronized in time to eliminate the time domain asynchrony. Then, the visual target is determined based on the ship visual data (that is, the ship position and other parameters determined by the visual data), and the AIS target is determined based on the AIS data (that is, the ship position and other parameters determined by the AIS), and the AIS data is coordinate transformed (that is, spatial registration is realized based on the transformation relationship between multiple coordinate systems). After the coordinate transformation, the target association matching is performed based on the three mixed multi-constraint conditions of ship position, ship heading and camera field of view to obtain the association result. Finally, the data is fused based on the association result to obtain the ship recognition result.

[0144] In order to better implement the ship target recognition method based on the multi-sensor data onboard the unmanned aerial vehicle in the embodiment of the present invention, on the basis of the ship target recognition method based on the multi-sensor data onboard the unmanned aerial vehicle, the embodiment of the present invention also provides a ship target recognition system based on the multi-sensor data onboard the unmanned aerial vehicle, such as Fig. 9As shown, the ship target recognition system 900 based on the multi-sensor data onboard the drone includes:

[0145] The multi-sensor data acquisition unit 901 is used to acquire the multi-sensor data onboard the UAV, where the multi-sensor data onboard the UAV includes AIS data, ship vision data, and IMU / GNSS data;

[0146] The multi-sensor data conversion unit 902 is used to establish a coordinate conversion relationship between the coordinate system of the ship visual data and the coordinate system of the AIS data based on the IMU / GNSS data, and to perform coordinate system conversion on the AIS data based on the coordinate conversion relationship to obtain AIS conversion data;

[0147] The multi-sensor data association and matching unit 903 is used to determine the association result of the ship visual data and the AIS conversion data based on the ship position constraint, the ship heading constraint and the camera field of view constraint, and fuse the ship visual data and the AIS conversion data based on the association result to obtain the ship target recognition result.

[0148] The ship target recognition system 900 based on UAV airborne multi-sensor data provided in the above embodiment can implement the technical solution described in the above embodiment of the ship target recognition method based on UAV airborne multi-sensor data. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above embodiment of the ship target recognition method based on UAV airborne multi-sensor data, which will not be repeated here.

[0149] like Fig.10 As shown, the present invention also provides a ship target identification device 1000. The ship target identification device 1000 includes a processor 1001, a memory 1002 and a display 1003. Fig.10 Only some components of the ship target identification device 1000 are shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components may be implemented instead.

[0150] In some embodiments, the processor 1001 may be a graphics processing unit (GPU) for running program codes or processing data stored in the memory 1002, such as the ship target recognition method based on multi-sensor data onboard a drone in the present invention.

[0151] In some embodiments, the processor 1001 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, the processor 1001 may be local or remote. In some embodiments, the processor 1001 may be implemented in a cloud platform. In one embodiment, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-cloud, etc., or any combination thereof.

[0152] In some embodiments, the memory 1002 may be an internal storage unit of the ship target identification device 1000, such as a hard disk or memory of the ship target identification device 1000. In other embodiments, the memory 1002 may also be an external storage device of the ship target identification device 1000, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the ship target identification device 1000.

[0153] Furthermore, the memory 1002 may include both an internal storage unit of the ship target identification device 1000 and an external storage device. The memory 1002 is used to store application software installed in the ship target identification device 1000 and various data.

[0154] In some embodiments, the display 1003 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 1003 is used to display information of the ship target recognition device 1000 and to display a visual user interface.

[0155] It should be noted that, for the convenience of observation, the display 1003 is installed at the shore-based end or the central control end.

[0156] The components 1001-1003 of the ship target identification device 1000 communicate with each other via a system bus.

[0157] In some embodiments of the present invention, when the processor 1001 executes the ship target recognition program based on the multi-sensor data onboard the drone in the memory 1002, the following steps may be implemented:

[0158] Acquire multi-sensor data onboard drones, including AIS data, ship vision data, and IMU / GNSS data;

[0159] A coordinate conversion relationship between the coordinate system of the ship's visual data and the coordinate system of the AIS data is established based on the IMU / GNSS data, and the AIS data is converted into a coordinate system based on the coordinate conversion relationship to obtain AIS conversion data;

[0160] The association result of the ship visual data and the AIS conversion data is determined based on the ship position constraint, the ship heading constraint and the camera field of view constraint, and the ship visual data and the AIS conversion data are fused based on the association result to obtain the ship target recognition result.

[0161] It should be understood that: when the processor 1001 executes the ship target identification program in the memory 1002, in addition to the above functions, other functions can also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0162] Furthermore, the ship target identification device 1000 in the embodiment of the present invention is mounted on a drone.

[0163] Correspondingly, an embodiment of the present invention also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, it can implement the steps or functions of the ship target identification method based on drone-mounted multi-sensor data provided by the above-mentioned method embodiments.

[0164] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0165] The above is a detailed introduction to a ship target recognition method and system based on drone-borne multi-sensor data provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A ship target recognition method based on multi-sensor data onboard an unmanned aerial vehicle, characterized in that: include: Acquire multi-sensor data onboard the drone, wherein the multi-sensor data onboard the drone includes AIS data, ship visual data, IMU data, and GNSS data; Establishing a coordinate conversion relationship between a coordinate system where the ship visual data is located and a coordinate system where the AIS data is located based on the IMU data and the GNSS data, and performing a coordinate conversion on the AIS data based on the coordinate conversion relationship to obtain AIS conversion data; Determine the association result between the ship visual data and the AIS conversion data based on the ship position constraint, the ship heading constraint and the camera field of view constraint, and fuse the ship visual data and the AIS conversion data based on the association result to obtain a ship target recognition result; The vessel position constraint for: The ship heading constraint for: The camera field of view constraint is: filtering out the AIS conversion data outside the camera field of view; In the formula, is the ship position in the i-th AIS conversion data; is the coordinate of the center point of the rotating frame in the jth ship visual data; is the Euclidean distance between the position coordinates of the same ship in the i-th AIS data and the j-th ship visual data; is the distance threshold; is the ship heading in the i-th AIS conversion data; is the ship heading in the jth ship visual data; is the angle threshold.

2. The ship target recognition method based on multi-sensor data onboard an unmanned aerial vehicle according to claim 1 is characterized in that: The coordinate system where the ship visual data is located is a pixel coordinate system, the coordinate system where the AIS data is located is a WGS84 coordinate system, the AIS data includes the ship's longitude and latitude, speed, heading and ship name, the GNSS data includes the drone's longitude, latitude and altitude, and the IMU data includes the azimuth and pitch angle of the airborne camera; then the coordinate conversion relationship between the coordinate system where the ship visual data is located and the coordinate system where the AIS data is located, established based on the IMU data and the GNSS data, includes: Determine a first conversion relationship between the WGS84 coordinate system and the world coordinate system based on the ship longitude, the ship latitude, the drone longitude and the drone latitude; Determine a second transformation relationship between the world coordinate system and the body coordinate system, a third transformation relationship between the camera coordinate system and the body coordinate system, a fourth transformation relationship between the camera coordinate system and the image coordinate system, and a fifth transformation relationship between the image coordinate system and the pixel coordinate system; The coordinate transformation relationship is determined based on the first transformation relationship, the second transformation relationship, the third transformation relationship, the fourth transformation relationship, and the fifth transformation relationship.

3. The ship target recognition method based on multi-sensor data onboard an unmanned aerial vehicle according to claim 2 is characterized in that: The determining the coordinate transformation relationship based on the first transformation relationship, the second transformation relationship, the third transformation relationship, the fourth transformation relationship and the fifth transformation relationship comprises: Determine an initial coordinate transformation relationship based on the first transformation relationship, the second transformation relationship, the third transformation relationship, the fourth transformation relationship, and the fifth transformation relationship; Obtain camera distortion parameters, and calibrate the initial coordinate transformation relationship based on the camera distortion parameters to obtain the coordinate transformation relationship.

4. The ship target recognition method based on multi-sensor data onboard an unmanned aerial vehicle according to claim 1 is characterized in that: The determining of the association result between the ship visual data and the AIS conversion data based on the ship position constraint, the ship heading constraint and the camera field of view constraint comprises: Filtering the AIS conversion data based on the camera field of view constraint to obtain target AIS conversion data; Determine a ship position constraint matrix between the ship visual data and the target AIS conversion data based on the ship position constraint; Determine a ship heading constraint matrix between the ship visual data and the target AIS conversion data based on the ship heading constraint; A comprehensive constraint matrix is ​​determined based on the ship position constraint matrix and the ship heading constraint matrix, and a correlation result between the ship visual data and the AIS conversion data is determined based on the comprehensive constraint matrix.

5. The ship target recognition method based on multi-sensor data onboard an unmanned aerial vehicle according to claim 4 is characterized in that: The number of rows of the comprehensive constraint matrix is ​​the total number of target AIS conversion data, and the number of columns of the comprehensive constraint matrix is ​​the total number of ship visual data; The determining the association result between the ship visual data and the AIS conversion data based on the comprehensive constraint matrix includes: Determining the minimum non-zero value in each row of the comprehensive constraint matrix; The successfully associated target ship visual data and target AIS conversion data are determined based on the row number and column number corresponding to the minimum value.

6. The ship target recognition method based on multi-sensor data onboard an unmanned aerial vehicle according to claim 1 is characterized in that: The ship target recognition method based on the UAV airborne multi-sensor data also includes: Determining, based on the association result, matching AIS data in the AIS data that is successfully associated with the ship visual data; The ship visual data is displayed, and the matching AIS data is superimposed and displayed on the ship visual data.

7. The ship target recognition method based on multi-sensor data onboard an unmanned aerial vehicle according to claim 1 is characterized in that: After obtaining the multi-sensor data onboard the drone, the method further includes: The multi-sensor data on the drone is synchronized with the hardware time and the software time.

8. The ship target recognition method based on multi-sensor data onboard an unmanned aerial vehicle according to claim 1 is characterized in that: Following the multi-sensor data onboard the drone, it also includes: The AIS data is decoded and cleaned.

9. A ship target recognition system based on multi-sensor data onboard an unmanned aerial vehicle, characterized in that: include: A multi-sensor data acquisition unit, used to acquire multi-sensor data onboard the drone, wherein the multi-sensor data onboard the drone includes AIS data, ship vision data, IMU data, and GNSS data; A multi-sensor data conversion unit, used to establish a coordinate conversion relationship between a coordinate system where the ship visual data is located and a coordinate system where the AIS data is located based on the IMU data and the GNSS data, and to perform coordinate system conversion on the AIS data based on the coordinate conversion relationship to obtain AIS conversion data; A multi-sensor data association and matching unit, used to determine the association result of the ship visual data and the AIS conversion data based on the ship position constraint, the ship heading constraint and the camera field of view constraint, and fuse the ship visual data and the AIS conversion data based on the association result to obtain a ship target recognition result; The vessel position constraint for: The ship heading constraint for: The camera field of view constraint is: filtering out the AIS conversion data outside the camera field of view; In the formula, is the ship position in the i-th AIS conversion data; is the coordinate of the center point of the rotating frame in the jth ship visual data; is the Euclidean distance between the position coordinates of the same ship in the i-th AIS data and the j-th ship visual data; is the distance threshold; is the ship heading in the i-th AIS conversion data; is the ship heading in the jth ship visual data; is the angle threshold.

Images