Ship collision risk assessment method and system based on multi-source heterogeneous data

Abstract

The invention relates to a ship collision risk assessment method and system based on multi-source heterogeneous data, and the method comprises the steps: obtaining a plurality of continuous frames of images shot at a fixed position, and extracting the center pixel coordinates of all ships in each frame of image; acquiring AIS data of the ship; aiming at the center pixel coordinates of all ships in each frame of image, identifying the ships with breakpoints in the tracks and the ships detected by mistake, and realizing the correction of the ship tracks; based on the pixel coordinates of the corrected ship trajectory, converting the pixel coordinates into world coordinates to obtain visual data; obtaining fusion data by matching the AIS data with the visual data; and based on the fusion data, ship collision risk assessment is realized. Compared with the prior art, the method has the advantages of improving the evaluation accuracy under the condition that the AIS data is partially missing, overcoming the problem of feature drift and the like.

Description

Technical Field

[0001] The present invention relates to the field of ship risk assessment, and in particular to a ship collision risk assessment method and system based on multi-source heterogeneous data. Background Art

[0002] With the continuous increase in global trade volume, the shipping industry plays a vital role in international transportation. As an important part of water transportation, inland waterways are important channels for cargo transportation. However, the unique geographical characteristics and high traffic density of inland waterways have brought considerable pressure to water traffic management. Among various types of maritime accidents, collisions are considered to be a high-incidence type of accidents in different waters. According to historical accident reports, global ship collision accidents accounted for 59.18% of global ship accidents in the past decade. The high incidence and severity of ship collision accidents pose a severe challenge and threat to water traffic safety.

[0003] At present, the research on ship collision risk assessment is mainly based on two data sources: ship automatic identification system (AIS) data and video surveillance data. The ship automatic identification system is a navigation aid system that can provide dynamic position information and static information of ships. It has been widely used in ship collision risk assessment in the past. However, AIS data may sometimes be inaccurate or incomplete. For example, some inland vessels (e.g., fishing boats, small cargo ships, etc.) may turn off their AIS equipment due to some illegal acts, which may make it impossible to accurately assess their collision risk.

[0004] In summary, there is currently a lack of a ship collision risk assessment method and system to solve or partially solve the problem of being unable to accurately assess the ship collision risk when some AIS data is missing. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a ship collision risk assessment method and system based on multi-source heterogeneous data, so as to solve or partially solve the problem that the ship collision risk cannot be accurately assessed when some AIS data is missing.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] One aspect of the present invention provides a method for assessing ship collision risk based on multi-source heterogeneous data, comprising the following steps:

[0008] Obtain multiple consecutive frames of images taken at a fixed position, and extract the center pixel coordinates of all ships in each frame of the image;

[0009] Get the ship's AIS data;

[0010] Based on the central pixel coordinates of all ships in each frame of image, identify ships with breakpoints in their trajectories and misdetected ships, and correct the ship trajectories;

[0011] Based on the pixel coordinates of the corrected ship trajectory, visual data is obtained by transforming the pixel coordinates into world coordinates;

[0012] Obtaining fused data by matching the AIS data and the visual data;

[0013] Based on the fused data, ship collision risk assessment is achieved.

[0014] As a preferred technical solution, the process of extracting the center pixel coordinates of all ships in each frame of image includes the following steps:

[0015] The computer vision model based on the image target tracking algorithm is trained with the goal of maximizing the intersection-over-union ratio of the area of ​​the true box and the area of ​​the detected detection box.

[0016] The image is divided into grids and used as the input of the computer vision model. The non-maximum suppression method is used to obtain the center pixel coordinates, width and height of the detection box with the maximum intersection-over-union ratio, as well as the confidence score of the detection box based on the offset between the prediction box and the anchor box, the activation function, and the weight matrix of the anchor box and set the threshold.

[0017] Based on DeepSORT, the connection between multiple frames is established, and the position of the ship is updated by calculating the error covariance:

[0018]

[0019] G t =CG t-1 C T +Q

[0020] K=G t W T (WG t W T +H)

[0021]

[0022] G t =(I-KU)G t-1

[0023] in, are the x coordinates of the ship at the previous moment and the current moment, G t-1 for The error covariance, G t for The error covariance of is, C and W are the state transfer matrix and the observation matrix respectively, CT and W T They represent the transposed matrices of the state transfer matrices C and W, Q and H are the system noise and observation noise, respectively, K is the Kalman gain, is the x-coordinate tracked in the tth frame, l represents the unit matrix, and U is the measurement matrix.

[0024] As a preferred technical solution, after obtaining the AIS data of the ship, it also includes:

[0025] The AIS data is subjected to regional screening, data deduplication and data denoising, and the ship type, length and width data are obtained by matching the dynamic and static AIS data of the ship.

[0026] As a preferred technical solution, the process of identifying ships with breakpoints in their trajectories and misdetected ships includes the following steps:

[0027] Based on the center pixel coordinates of the ship in each frame image, the ship trajectory is obtained in time sequence;

[0028] Based on the ship trajectory, identify the ships with breakpoints in the trajectory;

[0029] The area where the image was taken is taken as the monitoring area A, and the monitoring area B is obtained by adjustment to be smaller than the monitoring area A. The ships that appear in the monitoring area B but not in the monitoring area A are screened as misdetected ships.

[0030] As a preferred technical solution, the ship trajectory correction process includes the following steps:

[0031] The time similarity and position similarity between the ship with breakpoints in the trajectory and the misdetected ship are calculated respectively;

[0032] Based on the time similarity and position similarity, the ship with breakpoints in the calculated trajectory is matched with the trajectory of the misdetected ship in a manner of preferentially matching the information pairs with higher values, so as to realize the correction of the ship trajectory.

[0033] As a preferred technical solution, the process of transforming the pixel coordinates into world coordinates includes the following steps:

[0034] Using the pre-built transformation matrix, the pixel coordinates of the corrected ship trajectory are transformed into world coordinates using a multi-region projection method:

[0035]

[0036] Among them, u, ν are the pixel coordinates of the ship; x' and y' are the longitude and latitude coordinates of the ship after coordinate projection, w and w' are the scaling factors, and a 11 -a 33 are the parameters to be calibrated in the transformation matrix, u(n) ,v (n) is the pixel coordinate of the ship in the nth region, x n ' and y n ' is the latitude and longitude coordinates of the ship in the nth area, a 11(n) –a 33(n) are the parameters of the transformation matrix in the nth region respectively.

[0037] As a preferred technical solution, the process of acquiring fusion data includes the following steps:

[0038] Calculating the time similarity, speed similarity and position similarity between the AIS data and the visual data, and constructing a total allocation cost matrix;

[0039] Based on time similarity, speed similarity and location similarity, a weighted bipartite graph is constructed with visual data and AIS data as nodes;

[0040] The Hungarian algorithm is used to search for the best match from the total allocation cost matrix to obtain the fused data.

[0041] As a preferred technical solution, the total allocation cost matrix is:

[0042]

[0043] Among them, m i,j is the total cost between ship points i and j, that is, the element in the i-th row and j-th column of the allocation cost matrix, represents the normalization function, Ω and Γ are parameters, and H(·) is the step function.

[0044] As a preferred technical solution, the process of implementing ship collision risk assessment based on the fusion data includes the following steps:

[0045] Based on the fused data, the relative speed, time interval and the angle between the two propagation lines and the velocity vectors are calculated. When the following equation is satisfied, there is a collision risk between the two ships:

[0046]

[0047] in,

[0048]

[0049] In the above formula, ΔT is the distance traveled by ship i in the time interval Δt, the relative speed of ship i relative to ship j at time t, the line connecting ship points i and j and V ij The angle between the velocity vectors, are the angles between ship i and ship j and the vertical direction at time t, is the distance between ship i and ship j at time t, and R is the preset collision risk radius.

[0050] Another aspect of the present invention provides a ship collision risk assessment system based on multi-source heterogeneous data, comprising:

[0051] The central pixel coordinate extraction module of the ship is used to obtain multiple consecutive frames of images taken at a fixed position and extract the central pixel coordinates of all ships in each frame of the image;

[0052] AIS module, used to obtain AIS data of ships;

[0053] The ship track correction module is used to identify the ships with breakpoints in the track and the misdetected ships according to the central pixel coordinates of all ships in each frame image, so as to correct the ship track;

[0054] A coordinate conversion module, for obtaining visual data by converting the pixel coordinates into world coordinates based on the pixel coordinates of the corrected ship trajectory;

[0055] A multi-heterogeneous data fusion module, used to obtain fused data by matching the AIS data and the visual data;

[0056] The risk assessment module is used to implement ship collision risk assessment based on the fused data.

[0057] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0058] (1) Improving the assessment accuracy when AIS data is partially missing: Compared with the risk perception system based on a single data source, the present invention provides a method based on the fusion of AIS data and visual data, and a coordinate conversion method based on video surveillance data and multi-region projection to convert video pixel coordinates into longitude and latitude coordinates, thereby realizing a ship collision risk assessment system based on multi-source heterogeneous data fusion.

[0059] (2) Overcoming the problem of feature drift: Compared with existing visual perception methods, the present invention provides a solution based on the drift of ship visual features. By considering the similarity threshold of visual data timestamp and position, the original video detection data is corrected by matching the misdetected ships with the ships with interrupted tracks. The present invention can not only eliminate the shortcomings of AIS data (e.g., missing ship track data), but also reduce the visual data errors caused by water surface reflection and camera distortion. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 Schematic diagram of a ship collision risk assessment method based on multi-source heterogeneous data in an embodiment;

[0061] Figure 2 This is a schematic diagram of correction for the drift problem of ship visual features in an embodiment;

[0062] Figure 3 Schematic diagram of the process of detecting ship collision in the embodiment;

[0063] Figure 4 Schematic diagram of a ship collision risk assessment system based on multi-source heterogeneous data in an embodiment DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0065] Example 1

[0066] In view of the problems existing in the aforementioned prior art, considering that video surveillance technology has good real-time and intuitiveness, some drawbacks of AIS data are avoided. However, there are still some problems with video surveillance data. For example, visual surveillance data cannot extract detailed static ship information. In addition, external factors such as water surface reflection and camera distortion may cause visual feature information to drift, thereby leading to misjudgment of collision risk. Therefore, this embodiment provides a ship collision risk assessment method based on multi-source heterogeneous data, which integrates AIS data with video data to propose a scientific and reliable ship collision risk assessment, which is crucial to reducing the error in ship collision risk assessment.

[0067] See also Figure 1 , the method comprises the following steps:

[0068] Step S1, shore-based image acquisition and processing step: by acquiring video data within the shore-based monitoring video range, the video data is frame-processed to obtain multiple shore-based images, and the image target tracking algorithm is used to extract the central pixel coordinates of all ship targets within the monitoring range from the shore-based images. Step S1 may include the following sub-steps:

[0069] Step S101, using a computer vision model based on an image target tracking algorithm to perform target detection on target ships in each maritime image.

[0070] During detection, each maritime image is divided into a grid, and IOU is used as the detection metric. IOU represents the overlap rate between the detection frame and the true frame by calculating the intersection-over-union ratio of the area of ​​the true frame and the area of ​​the detected detection frame, so as to accurately capture the characteristics of the ship. Then the deep convolutional neural network outputs the coordinates, width and height of the detection frames of all target ships.

[0071] The output of the computer vision model is the predicted detection box position. The non-maximum suppression method is used to determine the detection box with the largest value based on the offset between the predicted box and the anchor box, the activation function, and the weight matrix of the anchor box, and to set a threshold to obtain accurate target ship detection results.

[0072] The final output of the image target tracking algorithm includes not only the coordinates, width and height of all target ship detection frames, but also the confidence score of the detected ship. The confidence score is composed of the overlap rate between the detection frame and the true frame and the weight factor. When the center of the target ship is located within the grid cell, the confidence score is set to 1, otherwise it is set to 0. IOU represents the overlap rate between the detection frame and the true frame, which can be expressed as:

[0073]

[0074] Among them, A gb Represents the area of ​​the real box; B db Represents the area of ​​the detection box; ∩ represents the intersection operation; ∪ represents the union operation.

[0075] Specifically, step S101 uses a YOLOv5 network to build and train a computer vision model to accurately detect ship features by predicting the coordinates, width, and height of the bounding box. First, the original input image of the ship is divided into an S×S grid for predicting the position, width, and height of the ship. The grid cell output captures the confidence score of the ship, which is defined as In order to further detect the ship features, non-maximum suppression (NMS) is used to determine the box with the largest IOU. Therefore, the detection result of the ship features can be expressed as:

[0076]

[0077] in, and are the coordinates of the center point of the ship on the x-axis and y-axis in the original image respectively; and Respectively represent the width and height of the captured ship target; Represents the offset between the predicted box and the anchor box; λ is the activation function of SiLU; and is the weight matrix of the anchor box; a x and ay are the horizontal and vertical distances from the center point of the ship to the upper left corner of the original image, respectively.

[0078] Step S102, in order to track the ship in the video, DeepSORT is applied to establish the connection between multiple frames of video, match the ship position in the continuous images, and then use the Kalman algorithm to predict the x-axis position of the ship in the tth frame with the help of the ship's position at the previous moment to obtain the error covariance. The Mahalanobis distance is used to calculate the distance between the prediction result and the detection result to associate the motion information, and the Hungarian matching algorithm is used to combine the motion metric and the appearance feature metric to associate the predicted motion frame with the detection frame to achieve ship matching, and finally the motion trajectory of the ship can be obtained.

[0079] Specifically, in order to track the ship in the video, step S102 further applies DeepSORT to establish the connection between multiple frames of the video. Specifically, by detecting the ship features, the ship position is consistently matched in the continuous images. The Kalman algorithm is used to obtain the ship's position at the previous moment. Predict the x-axis position of the ship at frame t and get the error covariance:

[0080]

[0081] Additionally, the error covariance and the position of the ship on the x-axis are updated, which can be shown in the equation:

[0082] K=G t W T (WG t W T +H) (6)

[0083]

[0084] G t =(I-KU)G t-1 (8)

[0085] Where: G represents the x coordinate of the ship at time t-1; t-1 for The error covariance, G t for The error covariance of; C and W are the state transfer matrix and the observation matrix respectively; C T and W T represents the transposed matrix of the state transfer matrix C and W respectively; Q and H are the system noise and observation noise respectively; K is the Kalman gain; is the x-coordinate tracked in the tth frame; l represents the unit matrix. The y-coordinate of the ship's position is updated in the same way as the x-coordinate.

[0086] The Mahalanobis distance is used to calculate the distance between the prediction result and the detection result, and the target motion information is associated based on the minimum Mahalanobis distance between the prediction result and the detection result. The formula is as follows:

[0087]

[0088] where d j is the jth position information detected; t i is the i-th position prediction information of the tracker, S i is the covariance matrix between detection and tracked positions.

[0089] For ship matching, the Hungarian matching algorithm is combined with motion metric and appearance feature metric, the predicted motion box is associated with the detection box, and the trajectory selection is confirmed to obtain the motion trajectory of the ship.

[0090] Step S2, AIS data acquisition and processing step: AIS data is acquired, and data cleaning is completed by performing regional screening, data deduplication, and data denoising on the AIS data, and then the static data of ship type, length, and width are acquired respectively through matching of dynamic and static AIS data of the ship.

[0091] The initial AIS data contains multiple data. First, the AIS data preprocessing of the target area is completed, including area screening, data deduplication, data denoising, etc. Then, the static data of ship type, length and width are obtained through the matching of dynamic and static AIS data of ships.

[0092] Step S3, feature drift processing step: sort the center pixel coordinates of the ship by time, identify ships with short track points and misdetected ships, and complete data correction by measuring the similarity between time and position. Step S3 includes the following sub-steps:

[0093] Step S301, firstly, the captured ship pixel points are sorted in chronological order, and ships with discontinuous track gauges are extracted (i.e., ships with interrupted tracks are extracted). With the help of the continuity of the ship's motion track, the monitoring area is adjusted from the original area A to a smaller area B. Ships that appear in area B but not in area A are regarded as misdetected ships.

[0094] Specifically, step S301 first sorts all captured ship pixel points in chronological order and extracts ships with track breakpoints. Secondly, since the frequency of ship tracks detected in the video is once per second, the tracks of all ships should be complete. According to the characteristics of the waterway, the monitoring area is adjusted from the original area A to a smaller area B. Among them, area B is defined by uniformly reducing the boundary of area A by 10% of the waterway width toward the center of the monitoring area. Therefore, ships that appear in area B but not in area A are regarded as false detections. Figure 2shown.

[0095] Step S302, after identifying the misdetected ship, the original video detection data is corrected by matching the misdetected ship with the ship whose track is interrupted by considering the threshold of timestamp and position similarity. The approximate matching mechanism is used to calculate the similarity between the misdetected ship and the ship whose track is interrupted in terms of time, position, speed, etc., and the information pairs with high similarity are matched first to complete the data correction of the ship whose track is interrupted.

[0096] Specifically, step S302 corrects the original video detection data by matching the misdetected ship with the ship whose track is interrupted by considering the threshold of the timestamp and position similarity. This embodiment adopts an approximate matching mechanism, that is, matching the information pair with a smaller value first. Specifically, the timestamp (Ts) and the position similarity (Ps) can be expressed as follows:

[0097]

[0098] in and are the timestamps of incomplete ships and misdetected ships, respectively; Indicates a break in an incomplete ship track; It is the first point of the misdetected ship’s track.

[0099] Step S4, coordinate conversion step: divide the monitoring area into n rectangular areas, obtain the vertex pixel coordinates and world coordinates of each area, establish a projection matrix, and convert the pixel coordinates of the ship in each area into world coordinates.

[0100] The monitoring area is divided into n rectangular areas, the pixel coordinates and world coordinates of the vertices of each area are obtained, and the image is projected to a new view plane by establishing a projection matrix. The coordinate projection process can be achieved by multiplying the coordinate vector with the transformation matrix to convert the pixel coordinates of the ship in each area into the world coordinates.

[0101] Specifically, after completing the data correction process, coordinate projection is performed, that is, the process of projecting the image to a new view plane by establishing a projection matrix. Note that the coordinate projection process can be achieved by multiplying the coordinate vector with the transformation matrix, as shown in the following formula:

[0102]

[0103] Where u, v are the pixel coordinates of the ship; x' and y' are the longitude and latitude coordinates of the ship after coordinate projection; w and w' are scaling factors, which were usually set to 1 in the past; a 11 -a 33 are the parameters to be calibrated in the transformation matrix.

[0104] It should be noted that in most solutions, the control area is usually considered as a single area of ​​coordinate projection. However, the single-area projection based on the bridge camera is easily affected by camera distortion in the regular edge area. Taking this into account, this embodiment designs a multi-area projection to reduce the influence of camera aberration. Assuming that the control edge can be divided into n areas, the coordinate projection process is as follows:

[0105]

[0106] where u (n) ,v (n) is the pixel coordinate of the ship in the nth region; x n ' and y n ' is the latitude and longitude coordinates of the ship in the nth area; a 11(n) –a 33(n) are the parameters of the transformation matrix in the nth region respectively.

[0107] Step S5, video data and AIS data matching step: Based on the AIS data and video data processed in the above steps, the similarity values ​​of time, speed and position of each ship are calculated, and then the video data and AIS data are matched using the Hungarian algorithm.

[0108] The data fusion problem is modeled as a linear assignment problem. Based on the AIS data and video data processed in the above steps, the similarity values ​​of time, speed, and position of each ship are calculated. Based on the feature similarity measure, a weighted bipartite graph is constructed with nodes (visual and AIS points). The weights of the edges are determined by the cost matrix. Finally, the Hungarian algorithm is used to find the optimal match from the total assignment cost matrix.

[0109] Specifically, after completing the multi-region coordinate projection process, data fusion will be performed to match the AIS data with the ship trajectory extracted from the visual data. Multi-feature similarity measurement is adopted. Specifically, the timestamp, speed and position similarity measurement are as follows:

[0110]

[0111] Where T f ,S f ,P f are timestamp, speed and position similarity respectively; and These are the timestamps of the visual data and AIS data, respectively; and Represent the ship speed obtained from visual data and AIS data respectively; and They are the ship positions obtained from visual data and AIS data respectively.

[0112] According to the above similarity index, the allocation cost matrix M of time, speed and position is calculated respectively. T , each matrix element represents the feature difference between different points in the visual data and the AIS data. For example:

[0113]

[0114] Similarly, construct M S and M P .

[0115] The above matrix is ​​then normalized, that is, the total allocation cost matrix M is composed of the multi-dimensional feature similarities of trajectory points from different data sources. The total allocation cost matrix between ship points i and j after normalization can be expressed as:

[0116]

[0117] in represents the normalization function, Ω is set to 10 9 , Γ is set to 5s. Where: H(·) is a step function, which can be given by the following formula:

[0118]

[0119] Based on the feature similarity measure, a weighted bipartite graph is constructed with nodes (including visual data and AIS points). The weights of the edges are determined by the cost matrix. Finally, the Hungarian algorithm is used to minimize the time, speed and position differences as the optimization goal, and the optimal solution is found through iteration, that is, the matching result of each pair of visual points and AIS points.

[0120] Step S6, risk assessment step: obtain the ship position and speed data after data matching, calculate the relative speed between each ship, the ship's travel distance, and the distance between each ship, and based on this calculate the ship conflict event value to achieve ship collision risk assessment based on multi-source data fusion.

[0121] After data matching, the ship position and speed data are obtained, and the relative speed between each ship, the ship travel distance, and the distance between each ship are calculated. Using the fused data, a conflict identification model based on the ship field is constructed to determine the potential risk of ship collision. Ship conflict refers to an emergency situation in which a ship is expected to enter the field of another ship in the next time interval. If the distance traveled by the target ship in the time interval is less than the product of the relative speed of the target ship relative to the other ship and the time interval, it is determined to be a ship conflict.

[0122] Specifically, based on the fusion of ship conflict identification. After obtaining the fusion data containing ship-related information, such as Maritime Mobile Service Identifier (MMSI), geographic coordinates, ship length and ship speed, a conflict identification model based on the ship domain is constructed to determine the potential risk of ship collision. A ship conflict refers to an emergency situation in which a ship is expected to enter the domain of another ship within the next time interval. An example of a ship conflict is Figure 3 shown.

[0123] Assume that the two ships after data fusion are recorded as i and j. Therefore, the relative speed of ship i relative to ship j at time t can be calculated as follows:

[0124]

[0125] The distance traveled by ship i in the time interval Δt can be calculated by the following formula:

[0126]

[0127] The distance between ship i and ship j at time t can be obtained by Euclidean distance. ij The angle θ between the velocity vectors can be expressed as:

[0128]

[0129] Therefore, a collision event will occur between ship i and ship j if the following conditions are met:

[0130]

[0131] In summary, this method first uses the YOLOv5 network to predict the coordinates, width and height of the bounding box to accurately detect the ship features. The original input image of the ship is divided into an S×S grid for predicting the position, width and height of the ship. In order to further detect the ship features, non-maximum suppression (NMS) is used to determine the box with the largest IOU. In order to track the ship in the video, DeepSORT is further applied to establish the connection between multiple frames of the video. Considering the drift of the ship's visual features, the AIS and visual data are fused. All captured ship pixels are sorted in chronological order, and the ships with track breakpoints are extracted. After completing the data correction process, a projection matrix is ​​established to project the image to the new view plane. After completing the multi-region coordinate projection process, data fusion is performed to match the AIS data with the ship track extracted from the visual data, and fused data containing ship-related information can be obtained.

[0132] This method has the following characteristics:

[0133] (1) Compared with the risk perception system based on a single data source, this embodiment provides a method for fusing AIS data and video surveillance data, and a coordinate conversion method based on video surveillance data and multi-region projection to convert video pixel coordinates into longitude and latitude coordinates, thereby realizing a ship collision risk assessment system based on multi-source heterogeneous data fusion.

[0134] (2) Compared with the existing visual perception method, this embodiment provides a solution based on the drift of ship visual features. By considering the similarity threshold of the visual data timestamp and position, the misdetected ship is matched with the ship with interrupted track to correct the original video detection data. Not only can the shortcomings of AIS data (e.g., missing ship track data) be eliminated, but also the visual data errors caused by water surface reflection and camera distortion can be mitigated.

[0135] Example 2

[0136] Based on Example 1, see Figure 4 This embodiment provides a ship collision risk assessment system based on multi-source heterogeneous data, including:

[0137] The central pixel coordinate extraction module of the ship is used to obtain multiple consecutive frames of images taken at a fixed position and extract the central pixel coordinates of all ships in each frame of the image;

[0138] AIS module, used to obtain AIS data of ships;

[0139] The ship track correction module is used to identify the ships with breakpoints in the track and the misdetected ships according to the central pixel coordinates of all ships in each frame image, so as to correct the ship track;

[0140] A coordinate conversion module, for obtaining visual data by converting the pixel coordinates into world coordinates based on the pixel coordinates of the corrected ship trajectory;

[0141] A multi-heterogeneous data fusion module, used to obtain fused data by matching the AIS data and the visual data;

[0142] The risk assessment module is used to implement ship collision risk assessment based on the fused data.

[0143] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A ship collision risk assessment method based on multi-source heterogeneous data, characterized in that: The steps include: Obtain multiple consecutive frames of images taken at a fixed position, and extract the center pixel coordinates of all ships in each frame of the image; Get the ship's AIS data; Based on the central pixel coordinates of all ships in each frame of image, identify ships with breakpoints in their trajectories and misdetected ships, and correct the ship trajectories; Based on the pixel coordinates of the corrected ship trajectory, visual data is obtained by transforming the pixel coordinates into world coordinates; Obtaining fused data by matching the AIS data and the visual data; Based on the fused data, ship collision risk assessment is achieved.

2. A ship collision risk assessment method based on multi-source heterogeneous data according to claim 1, characterized in that: The process of extracting the center pixel coordinates of all ships in each frame of image includes the following steps: The computer vision model based on the image target tracking algorithm is trained with the goal of maximizing the intersection-over-union ratio of the area of ​​the true box and the area of ​​the detected detection box. The image frame is divided into a grid and used as the input of the computer vision model. The non-maximum suppression method is used to obtain the center pixel coordinates, width and height of the detection box with the maximum intersection-over-union ratio, as well as the confidence score of the detection box based on the offset between the prediction box and the anchor box, the activation function, and the weight matrix of the anchor box and set the threshold. Based on DeepSORT, the connection between multiple frames is established, and the position of the ship is updated by calculating the error covariance: G t =CG t-1 C T +Q K=G t W T (WG t W T +H) G t =(I-KU)G t-1 in, are the x coordinates of the ship at the previous moment and the current moment, G t-1 for The error covariance, G t for The error covariance of is, C and W are the state transfer matrix and the observation matrix respectively, C T and W T They represent the transposed matrices of the state transfer matrices C and W, Q and H are the system noise and observation noise, respectively, K is the Kalman gain, is the x-coordinate tracked in the tth frame, l represents the unit matrix, and U is the measurement matrix.

3. The ship collision risk assessment method based on multi-source heterogeneous data according to claim 1 is characterized in that: After obtaining the ship's AIS data, it also includes: The AIS data is subjected to regional screening, data deduplication and data denoising, and the ship type, length and width data are obtained by matching the dynamic and static AIS data of the ship.

4. The ship collision risk assessment method based on multi-source heterogeneous data according to claim 1 is characterized in that: The process of identifying ships with breakpoints in their trajectories and misdetected ships includes the following steps: Based on the center pixel coordinates of the ship in each frame image, the ship trajectory is obtained in time sequence; Based on the ship trajectory, identify the ships with breakpoints in the trajectory; The area where the image was taken is taken as the monitoring area A, and the monitoring area B is obtained by adjustment to be smaller than the monitoring area A. The ships that appear in the monitoring area B but not in the monitoring area A are screened as misdetected ships.

5. The ship collision risk assessment method based on multi-source heterogeneous data according to claim 1 is characterized in that: The ship track correction process includes the following steps: The time similarity and position similarity between the ship with breakpoints in the trajectory and the misdetected ship are calculated respectively; Based on the time similarity and position similarity, the ship with breakpoints in the calculated trajectory is matched with the trajectory of the misdetected ship in a manner of preferentially matching the information pairs with higher values, so as to realize the correction of the ship trajectory.

6. The ship collision risk assessment method based on multi-source heterogeneous data according to claim 1 is characterized in that: The process of transforming the pixel coordinates into world coordinates comprises the following steps: Using the pre-built transformation matrix, the pixel coordinates of the corrected ship trajectory are transformed into world coordinates using a multi-region projection method: Among them, u, v are the pixel coordinates of the ship; x' and y' are the longitude and latitude coordinates of the ship after coordinate projection, w and w' are the scaling factors, and a 11 -a 33 are the parameters to be calibrated in the transformation matrix, u (n) ,v (n) is the pixel coordinate of the ship in the nth region, x n ' and y n ' is the latitude and longitude coordinates of the ship in the nth area, a 11(n) –a 33(n) are the parameters of the transformation matrix in the nth region respectively.

7. The ship collision risk assessment method based on multi-source heterogeneous data according to claim 1 is characterized in that: The process of acquiring fusion data includes the following steps: Calculating the time similarity, speed similarity and position similarity between the AIS data and the visual data, and constructing a total allocation cost matrix; Based on time similarity, speed similarity and location similarity, a weighted bipartite graph is constructed with visual data and AIS data as nodes; The Hungarian algorithm is used to search for the best match from the total allocation cost matrix to obtain the fused data.

8. The ship collision risk assessment method based on multi-source heterogeneous data according to claim 7 is characterized in that: The total allocation cost matrix is: Among them, m i,j is the total cost between ship points i and j, that is, the element in the i-th row and j-th column of the allocation cost matrix, represents the normalization function, Ω and Γ are parameters, and H(·) is the step function.

9. The ship collision risk assessment method based on multi-source heterogeneous data according to claim 1 is characterized in that: Based on the fused data, the process of implementing ship collision risk assessment includes the following steps: Based on the fused data, the relative speed, time interval and the angle between the two propagation lines and the velocity vectors are calculated. When the following equation is satisfied, there is a collision risk between the two ships: in, In the above formula, ΔT is the distance traveled by ship i in the time interval Δt, the relative speed of ship i relative to ship j at time t, the line connecting ship points i and j and V ij The angle between the velocity vectors, are the angles between ship i and ship j and the vertical direction at time t, is the distance between ship i and ship j at time t, and R is the preset collision risk radius.

10. A ship collision risk assessment system based on multi-source heterogeneous data, characterized in that: include: The central pixel coordinate extraction module of the ship is used to obtain multiple consecutive frames of images taken at a fixed position and extract the central pixel coordinates of all ships in each frame of the image; AIS module, used to obtain AIS data of ships; The ship track correction module is used to identify the ships with breakpoints in the track and the misdetected ships according to the central pixel coordinates of all ships in each frame image, so as to correct the ship track; A coordinate conversion module, for obtaining visual data by converting the pixel coordinates into world coordinates based on the pixel coordinates of the corrected ship trajectory; A multi-heterogeneous data fusion module, used to obtain fused data by matching the AIS data and the visual data; The risk assessment module is used to implement ship collision risk assessment based on the fused data.

Images