Ship trajectory prediction method and device, storage medium, computer program product and computing equipment

Abstract

The invention provides a ship trajectory prediction method and device, a storage medium, a computer program product and computing equipment. The ship trajectory prediction method comprises the steps that an image set collected for the same channel area is acquired; performing ship detection on the plurality of channel images to obtain a pixel coordinate position of a ship in each channel image; calculating confidence degree scores of the target ship and the candidate ship according to the pixel coordinate position of the target ship and the pixel coordinate position of the candidate ship; associating the target ship with the candidate ship corresponding to the maximum value of the confidence score, and generating an image trajectory data set of the target ship; and predicting the trajectory of the target ship in the future first duration according to the image trajectory data set. The invention provides a scheme for predicting the trajectory of the ship by automatically utilizing the image data, and the accuracy of ship trajectory prediction is improved.

Description

Technical Field

[0001] This application relates to the field of transportation technology, and in particular to a method and apparatus for predicting ship trajectories, a storage medium, a computer program product, and a computing device. Background Technology

[0002] Waterway transportation plays a crucial role in the logistics sector. However, due to human factors, traffic density, and hydrological and meteorological conditions, accidents such as contact, collisions, groundings, and sinkings occur frequently, severely impacting waterway traffic safety and efficiency. Some small inland waterway vessels illegally disable their Automatic Identification System (AIS) terminals, preventing maritime authorities from effectively monitoring them and triggering accidents. Similarly, due to the interval settings for AIS data transmission and issues with VHF channel allocation within the jurisdiction, the spacing between AIS points is often too long, and dynamic characteristics such as latitude and longitude are significantly lacking. This also significantly affects the monitoring and tracking of normally navigating vessels by maritime authorities, ultimately increasing the risk of accidents within the jurisdiction.

[0003] Therefore, maritime authorities have introduced a Vessel Traffic Service (VTS) system, which allows for the analysis and judgment of vessel movement information in video footage through manual monitoring, in order to prevent accidents.

[0004] However, manual monitoring suffers from efficiency and accuracy issues, and is also costly in terms of manpower. Summary of the Invention

[0005] This application provides a method and apparatus for predicting ship trajectories, offering a scheme for automatically predicting ship trajectories using image data, thereby improving the accuracy of ship trajectory prediction.

[0006] To achieve the above objectives, this application provides the following technical solution:

[0007] Firstly, a ship trajectory prediction method is provided, comprising: acquiring an image set collected for the same waterway area, the image set including multiple waterway images with a preset sampling interval between adjacent images; performing ship detection on the multiple waterway images to obtain the pixel coordinate positions of ships in each waterway image; calculating confidence scores for the target ship and the candidate ships based on the pixel coordinate positions of the target ship and the candidate ships, wherein the target ship is at least one ship in a first waterway image among the multiple waterway images, and the candidate ships are ships in other waterway images besides the first waterway image; associating the target ship with the candidate ship corresponding to the maximum value of the confidence score to generate an image trajectory dataset of the target ship; and predicting the trajectory of the target ship within a future first time period based on the image trajectory dataset.

[0008] Optionally, calculating the confidence score between the target ship and the candidate ship includes: calculating the appearance similarity between the target ship and the candidate ship, the distance between the target ship and the candidate ship, and the area difference between the target ship and the candidate ship based on the pixel coordinate positions of the target ship and the candidate ship; and generating the confidence score based on the appearance similarity, the distance, and the area difference.

[0009] Optionally, generating the image trajectory dataset of the target vessel includes: associating the target vessel with candidate vessels corresponding to the maximum confidence score to generate a set of pixel positions of the target vessel in multiple image frames of the multiple waterway images; obtaining a transformation matrix between the waterway images and the remote sensing images; converting each pixel position in the set of pixel positions of the multiple image frames into a projected position in the remote sensing image according to the transformation matrix, and calculating the latitude and longitude coordinates of the target vessel according to the projected position and the known latitude and longitude coordinates in the remote sensing image; calculating at least one of the following navigation information of the target vessel according to the latitude and longitude coordinates of the target vessel and the sampling interval of adjacent images: speed, heading, nearest encounter distance, and nearest encounter time; and constructing the image trajectory dataset according to the navigation information of the target vessel.

[0010] Optionally, predicting the trajectory of the target vessel within a future first time period based on the image trajectory dataset includes: inputting the image trajectory dataset and AIS data for the same waterway area into a prediction model to obtain the trajectory of the target vessel within the future first time period; the prediction model constructs a spatiotemporal graph set and a weighted adjacency matrix based on the image trajectory dataset and the AIS data, and generates the trajectory based on the spatiotemporal graph set and the weighted adjacency matrix; the spatiotemporal graph set represents the position of the target vessel at each time point, and the weighted adjacency matrix represents the interaction relationship between vessels.

[0011] Optionally, the prediction model constructs the weighted adjacency matrix based on the navigation information of the target vessel in the image trajectory dataset. The weighted adjacency matrix includes at least one of the following: speed-weighted adjacency matrix, heading-weighted adjacency matrix, nearest encounter distance-weighted adjacency matrix, and nearest encounter time-weighted adjacency matrix.

[0012] Optionally, the prediction model includes a spatiotemporal convolutional network and a temporal extrapolator convolutional neural network. The spatiotemporal convolutional network extracts a first embedding representation of spatiotemporal nodes based on the spatiotemporal graph set and the weighted adjacency matrix, encodes trajectory features, and outputs a second embedding representation. The temporal extrapolator convolutional neural network decodes the second embedding representation to obtain the trajectory of the target ship in the future within a first time period.

[0013] Secondly, this application also discloses a ship trajectory prediction device, comprising: an acquisition module for acquiring a set of images collected for the same waterway area, the image set including multiple waterway images with a preset sampling interval between adjacent images; a detection module for detecting ships in the multiple waterway images to obtain the pixel coordinate positions of ships in each waterway image; a confidence score calculation module for calculating a confidence score between the target ship and the candidate ship based on the pixel coordinate positions of the target ship and the candidate ships, wherein the target ship is at least one ship in a first waterway image among the multiple waterway images, and the candidate ships are ships in other waterway images besides the first waterway image; a dataset generation module for associating the target ship with the candidate ship corresponding to the maximum value of the confidence score to generate an image trajectory dataset of the target ship; and a prediction module for predicting the trajectory of the target ship within a future first time period based on the image trajectory dataset.

[0014] Optionally, the confidence score calculation module includes: a parameter calculation unit, used to calculate the appearance similarity between the target ship and the candidate ship, the distance between the target ship and the candidate ship, and the area difference between the target ship and the candidate ship based on the pixel coordinate positions of the target ship and the candidate ship; and a score calculation unit, used to generate the confidence score based on the appearance similarity, the distance, and the area difference.

[0015] Optionally, the dataset generation module includes: a pixel location set determination unit, used to associate the target vessel with the candidate vessel corresponding to the maximum value of the confidence score, and generate a set of pixel locations of the target vessel in multiple frames of image data in multiple channel images; a transformation matrix acquisition unit, used to acquire the transformation matrix between the channel image and the remote sensing image; a conversion unit, used to convert each pixel location in the set of pixel locations in the multiple frames of image data into a projected location in the remote sensing image according to the transformation matrix, and calculate the latitude and longitude coordinates of the target vessel according to the projected location and the known latitude and longitude coordinates in the remote sensing image; a navigation information determination unit, used to calculate at least one of the following navigation information of the target vessel according to the latitude and longitude coordinates of the target vessel and the sampling interval of adjacent images: speed, heading, nearest encounter distance, and nearest encounter time; and an image trajectory dataset construction unit, used to construct the image trajectory dataset according to the navigation information of the target vessel.

[0016] Optionally, the prediction module inputs the image trajectory dataset and the Automatic Identification System (AIS) data for the same waterway area into the prediction model to obtain the trajectory of the target vessel within a first time period in the future. The prediction model constructs a spatiotemporal graph set and a weighted adjacency matrix based on the image trajectory dataset and the AIS data, and generates the trajectory based on the spatiotemporal graph set and the weighted adjacency matrix. The spatiotemporal graph set represents the position of the target vessel at each time, and the weighted adjacency matrix represents the interaction relationship between vessels.

[0017] Thirdly, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being executed by a processor to perform the method provided in the first aspect.

[0018] Fourthly, a computer program product is provided, on which a computer program is stored, the computer program being executed by a processor to perform the method provided in the first aspect.

[0019] Sixthly, embodiments of this application also provide a chip that stores a computer program, which, when executed by the chip, implements the steps of the above-described method.

[0020] In a seventh aspect, embodiments of this application also provide a system chip for use in a terminal. The chip system includes at least one processor and an interface circuit, the interface circuit and the at least one processor being interconnected via a line. The at least one processor is used to execute instructions to perform the method provided in the first aspect.

[0021] Compared with the prior art, the technical solution of this application has the following beneficial effects:

[0022] In this application's technical solution, a series of processes are performed on the image set collected for the same waterway area to extract reliable dynamic features of the ship from the visual data, thereby obtaining the ship's navigation trajectory information in the image set, i.e., the image trajectory dataset. This image trajectory dataset will be used to predict the trajectory of the target ship in the first time period in the future. By introducing visual data, the ship's navigation information can be enriched, avoiding the loss of ship information caused by the lack of AIS data, and improving the accuracy of ship trajectory prediction.

[0023] Furthermore, the technical solution of this application generates a confidence score by using the appearance similarity distance and area difference between the target ship and the candidate ship to represent the similarity between them. By comprehensively considering constraints such as appearance features, distance factors, and area factors to calculate the similarity, target ships in adjacent frames are associated, identifying the same ship across frames, thereby obtaining the ship's trajectory and improving the reliability of ship tracking.

[0024] Furthermore, the prediction model in this application constructs a weighted adjacency matrix to represent the interaction relationship between ships. Specifically, it comprehensively evaluates the ship interaction based on the ship's nearest encounter distance, ship's nearest encounter time, speed, and heading, and uses multi-dimensional high interaction force characteristics to predict the future ship trajectory, thereby further improving the accuracy of ship trajectory prediction. Attached Figure Description

[0025] Figure 1 This is a flowchart of a ship trajectory prediction method provided in an embodiment of this application;

[0026] Figure 2 This is a flowchart of another ship trajectory prediction method provided in the embodiments of this application;

[0027] Figure 3 This is a flowchart of another ship trajectory prediction method provided in the embodiments of this application;

[0028] Figure 4 This is a schematic diagram of the structure of a prediction model provided in an embodiment of this application;

[0029] Figure 5 This is a schematic diagram of a ship trajectory prediction configuration provided in an embodiment of this application. Detailed Implementation

[0030] As described in the background section, in the absence of AIS data, manual monitoring suffers from efficiency and accuracy issues, and is also costly in terms of manpower.

[0031] Specifically, to improve monitoring efficiency and reduce manual labor, researchers are attempting to enhance the auxiliary capabilities of video data through methods such as deep learning to meet the needs of real-time automated maritime surveillance. For example, they are using waterway monitoring video combined with target tracking algorithms to obtain ship trajectories, then using AIS data for calibration, integrating ship number, type, speed, and heading information from the AIS data into the visual data. They are also using target detection methods to obtain ship positions in waterway images, filtering AIS data based on data integrity, detection range, and ship heading, and employing a module based on ship arrival time estimation to complete data fusion, all of which improve the reliability of ship trajectory tracking.

[0032] However, the system relies on AIS data, using it as supplementary information to visual data through methods such as data calibration. While this achieves data fusion, it fails to effectively mine information from the visual data. Visual data is merely used to verify the results of AIS inference; it doesn't directly extract dynamic features through inference from the visual data, which is crucial for observing ship motion.

[0033] In real-world scenarios, there may be vessels lacking AIS data in inland waterways and other waterways. This application extracts reliable dynamic features of vessels from visual data by processing a series of image sets collected for the same waterway area. By introducing visual data, the navigation information of vessels can be enriched, avoiding the loss of vessel information caused by missing AIS data and improving the accuracy of vessel trajectory prediction.

[0034] Furthermore, in practical applications, navigation in inland waterways is complex, with rapid changes in vessel course and speed. Timely prediction based on vessel motion is crucial to ensure safety and assist management personnel in risk assessment and decision-making. Therefore, predicting future vessel trajectories based on acquired navigation tracks has become a research hotspot in waterway management in recent years. Early work on vessel trajectory prediction often used recurrent neural networks sensitive to time-series data to improve trajectory prediction methods, including Gated Recurrent Units (GRUs), Long Short-Term Memory (LSTM) networks, and Sequence-to-Sequence (Seq2Seq) networks. However, due to interactions between vessels during navigation, such as distance, speed, and course, vessel motion and waterway flow are often affected. Simply using trajectory point data and time-series deep learning algorithms is insufficient to guarantee high-quality vessel trajectory prediction.

[0035] This application considers the interaction between ships, and reflects the motion relationship between ships more accurately through the interaction between them, thereby improving the accuracy of trajectory prediction.

[0036] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0037] See Figure 1 The method provided in this application specifically includes steps 101 to 105.

[0038] It is understood that, in specific implementations, the ship trajectory prediction method can be implemented using software programs, which run within a processor integrated into a chip or chip module. This method can also be implemented using a combination of software and hardware; this application does not impose any limitations on this approach.

[0039] In step 101, an image set is acquired for the same waterway area. The image set includes multiple waterway images, and there is a preset sampling interval between adjacent images.

[0040] In practice, the image set can be sourced from waterway monitoring videos, and the image set is obtained by sampling from the waterway monitoring videos according to a preset sampling interval. Each waterway image may include one or more vessels.

[0041] Specifically, due to the complexity of the aquatic environment and the limitations of the computing power of the terminal equipment, the sampling time interval between adjacent frames of the channel image in the acquired image set is relatively large. For example, the average sampling time interval between adjacent frames is 40 seconds (s), resulting in significant changes in the distance, attitude, and size of the same ship between adjacent frames. The sampling interval between any two images can be the same or different.

[0042] In step 102, ship detection is performed on multiple waterway images to obtain the pixel coordinates of ships in each waterway image.

[0043] In practice, object detection models can be used to detect and obtain the pixel coordinates of ships in each waterway image. For example, the You Only Look Once (YOLO) model can be used to obtain the pixel coordinates of each ship in each waterway image. The pixel coordinates of each ship can be a set of pixel coordinates of at least one pixel of the ship in the waterway image.

[0044] In step 103, confidence scores for the target vessel and candidate vessels are calculated based on their pixel coordinates. The target vessel is at least one vessel in the first channel image among multiple channel images, and the candidate vessels are vessels in other channel images besides the first channel image.

[0045] In practice, the confidence score can represent the similarity between the target ship and the candidate ship.

[0046] In one specific implementation, a confidence score can be calculated using a ship re-identification model. In this case, the confidence score represents the appearance similarity between the target ship and the candidate ship, such as the similarity between the ship's shape, size, and dimensions.

[0047] Specifically, a Siamese network architecture is used to train the ship re-identification model. Considering real-time requirements, the lightweight MobileNet_v1 network is selected as the backbone network, and the Additive Angular Margin Loss (ArcFace) is used as the loss function. The Additive Angular Margin Loss function adds a margin to the existing loss function, which can enhance the classification ability. The loss function includes representation learning and metric learning. Representation learning treats the re-identification task as an image classification or validation problem. Metric learning considers the similarity between two images during training, including contrastive loss, triplet loss, etc. Compared with other loss functions, the Additive Angular Margin Loss can more effectively reduce intra-class distance and increase inter-class distance, effectively distinguishing different ships while maintaining the identification of the same ship, thus improving the accuracy of cross-frame ship re-identification.

[0048] In another specific implementation, confidence scores are generated by using appearance similarity, distance, and area difference to address the constraints of ship distance and size and avoid ship jumps and occlusion problems.

[0049] Refer to together Figure 2 , Figure 2 The specific procedure for calculating the confidence score is shown.

[0050] In step 201, the appearance similarity between the target ship and the candidate ship, the distance between the target ship and the candidate ship, and the area difference between the target ship and the candidate ship are calculated based on the pixel coordinate positions of the target ship and the candidate ship.

[0051] In step 202, a confidence score is generated based on appearance similarity, distance, and area difference.

[0052] Specifically, the confidence score can be calculated using the following formula (1):

[0053] score=α×similarity+β×(1-distance)+γ×(1-area), (1)

[0054] Wherein, `score` represents the confidence score, `similarity` is the appearance similarity between the target ship and the candidate ship, with a value ranging from [0,1]. `distance` is the normalized distance between the target ship and the candidate ship. `area` is the difference between the detection box areas of the target ship and the candidate ship, where the detection box area is the normalized area. `α`, `β`, and `γ` are the corresponding weighting values, responsible for adjusting the influence of different factors on the confidence score.

[0055] It should be noted that the specific values ​​of α, β and γ can be adaptively set according to the actual application scenario, and this application does not impose any restrictions on this.

[0056] This application embodiment combines factors such as distance and area to comprehensively constrain and score confidence, which serves as a matching metric for the similarity between the target ship and candidate ships, thereby achieving automatic association of ship data at different shooting times.

[0057] Continue to refer to Figure 1 In step 104, the target ship is associated with the candidate ship corresponding to the maximum confidence score to generate an image trajectory dataset of the target ship.

[0058] In practice, the highest confidence score is used as the matching weight for the similarity between the target ship and candidate ships, and the target ships in adjacent frames are sequentially associated. Specifically, for each tracking trajectory, the confidence scores of the target detection box (representing the target ship) in the previous frame of the channel image and all candidate detection boxes (representing candidate ships) in the current frame of the channel image are calculated sequentially. If the confidence score of a candidate detection box meets the criterion or reaches the maximum, then this candidate detection box is associated with the target detection box. This generates an image trajectory dataset of the target ship, which can include multiple positional information of the target ship within the sampling time.

[0059] In step 105, the trajectory of the target vessel in the first future time period is predicted based on the image trajectory dataset.

[0060] In practice, image trajectory datasets can be used to predict the trajectory of target vessels. Compared to AIS data, image trajectory datasets extracted from visual data contain more comprehensive and accurate information about vessels, thus ensuring the accuracy of target vessel predictions.

[0061] In some non-limiting embodiments, the image trajectory dataset can also represent the latitude and longitude coordinates of the target vessel. In this case, the pixel coordinates of the target vessel in the waterway image need to be converted into latitude and longitude coordinates in the remote sensing image.

[0062] Refer to together Figure 3 , Figure 3 The specific process of coordinate transformation is shown.

[0063] In step 301, the target vessel is associated with the candidate vessel corresponding to the maximum confidence score to generate a set of multi-frame image pixel positions of the target vessel in multiple waterway images.

[0064] In specific implementation, the set of pixel positions in multiple frames of images can include information about the coordinate positions of multiple pixels of the target vessel within the sampling time in multiple channel images. Specifically, the set of pixel positions in multiple frames of images is G1 = (x1, y1), (x2, y2), ..., (x... n ,y n ), where (x1, y1) represents the pixel coordinates of the target vessel in the first frame of the channel image, and (x2, y2) represents the pixel coordinates of the target vessel in the second frame of the channel image. n ,y n ) represents the pixel coordinates of the target vessel in the nth frame of the waterway image.

[0065] In step 302, the transformation matrix between the waterway image and the remote sensing image is obtained.

[0066] In practice, heterogeneous image registration aligns the coordinates in the waterway image with those in the remote sensing image, enabling the identification of ships within the waterway image data on the remote sensing map. Specifically, sufficient matching points are selected in both the waterway image data and the remote sensing map. Perspective transformation is then used to establish a mapping relationship between the pixel coordinates in the waterway image and the pixel coordinates in the remote sensing map, thus obtaining a transformation matrix. This transformation matrix can then be used to project the pixel coordinates of ships onto the remote sensing map.

[0067] Specifically, the transformation matrix T can be referred to as in formula (2):

[0068]

[0069] Wherein, the coefficient a in the transformation matrix 11 ,a 12 ,a 21 ,a 22 Responsible for linear transformation, coefficient a 13 ,a 23 Responsible for perspective transformation, coefficient a 31 ,a 32 It is responsible for translation transformations.

[0070] In step 303, the pixel positions in the set of pixel positions of the multi-frame image are converted into the projected positions in the remote sensing image according to the transformation matrix, and the latitude and longitude coordinates of the target ship are calculated according to the projected positions and the known latitude and longitude coordinates in the remote sensing image.

[0071] For any pixel coordinate (x, y) in the waterway image, the projection position (x', y') in the remote sensing image can be calculated using formula (3):

[0072]

[0073] The latitude and longitude coordinates of the target ship at multiple sampling times are calculated by combining the projected position in the remote sensing image with the known latitude and longitude coordinates in the remote sensing image.

[0074] In step 304, at least one of the following navigation information of the target vessel is calculated based on the latitude and longitude coordinates of the target vessel and the sampling interval of adjacent images: speed, heading, nearest encounter distance, and nearest encounter time.

[0075] In practice, the target ship's heading and speed are calculated based on its latitude and longitude coordinates at adjacent sampling times. The actual distance DR between adjacent trajectory points (i.e., adjacent sampling times) is obtained using the target ship's latitude and longitude coordinates, and the target ship's speed v is calculated by combining this distance with the known time interval t between adjacent frames.

[0076] In practice, the azimuth angle A between the current trajectory point and the trajectory point in the next frame is obtained by solving the azimuth angle formula based on latitude and longitude. Z To obtain the heading of the target ship's trajectory points.

[0077] In practice, the nearest encounter distance and the nearest encounter time between ships can be further calculated by using the target ship's course and speed.

[0078] It should be noted that the specific calculation formulas for ship course, nearest encounter distance between ships, and nearest encounter time can be found in existing technologies and will not be elaborated here.

[0079] In step 305, an image trajectory dataset is constructed based on the navigation information of the target vessel.

[0080] In specific implementation, the target vessel's image trajectory dataset may include at least one of the following navigation information of the target vessel: speed, heading, nearest encounter distance, and nearest encounter time. The target vessel's image trajectory dataset can be represented as: G photo =(lat1,lon1,course1,speed1,DCPA1,TCPA1),...,(lat n ,lon n course n speed n DCPA n TCPA n ), where lat1,lon1 represents the latitude and longitude of the target vessel at sampling time 1, course1 represents the heading of the target vessel at sampling time 1, speed1 represents the speed of the target vessel at sampling time 1, DCPA1 represents the nearest encounter distance of the target vessel at sampling time 1, TCPA1 represents the nearest encounter time of the target vessel at sampling time 1; and so on, lat n ,lon n This represents the latitude and longitude of the target ship at sampling time n. n The speed represents the heading of the target ship at sampling time n. n The target ship's speed at sampling time n, DCPA n TCPA represents the nearest encounter distance of the target ship at sampling time n. n This represents the closest encounter time of the target vessel at sampling time n.

[0081] In this embodiment of the application, the image trajectory dataset may include not only the location information of the target ship, but also the dynamic feature information of the target ship, such as the ship's latitude and longitude, speed, and heading at the time of image sampling, so as to enrich the feature dimensions of the visual data.

[0082] It should be noted that the sequence number of each step in this embodiment does not represent a limitation on the execution order of each step.

[0083] In some non-limiting embodiments, image trajectory datasets and Automatic Identification System (AIS) data for the same waterway area are input into a prediction model to obtain the trajectory of the target vessel in the first time period in the future.

[0084] As mentioned earlier, there are sampling intervals between multiple channel images, and the position changes of ships are discontinuous. Therefore, the spatial characteristics of the target ship at different sampling times can be incorporated into the experimental design to eliminate the impact of discontinuity on trajectory prediction.

[0085] Given N ships in a scenario and their time interval T O Latitude and longitude positions observed from within It is necessary to predict the future time period T of N ships. P Inner trajectory

[0086] In this embodiment, the prediction model constructs a spatiotemporal graph set and a weighted adjacency matrix based on the image trajectory dataset and AIS data, and generates a trajectory based on the spatiotemporal graph set and the weighted adjacency matrix. The spatiotemporal graph set represents the position of the target ship at each time, and the weighted adjacency matrix represents the interaction relationship between ships.

[0087] Specifically, the prediction model first needs to construct a set of spatiotemporal graphs G. t =(V t E t Let V represent the position of the target ship at each sampling time t, where V t The set of vertices representing the coordinates of the target ship can be specifically represented as: This indicates the latitude and longitude coordinates of the target vessel. E t Representing the spacetime graph G t The edge set can be specifically represented as if and Connected, =1, otherwise It is 0.

[0088] In this embodiment, the motion of a ship is influenced by many factors, such as wave loads, structural stress, wind force, and even steering force. The motion of the ship's trajectory is considered a complex nonlinear behavior. Furthermore, the interaction forces between ships have varying degrees of influence on the ship's motion, necessitating the consideration of evaluation methods that are highly correlated with the ship's motion.

[0089] In one specific embodiment, the weighted adjacency matrix includes at least one of the following: speed-weighted adjacency matrix, heading-weighted adjacency matrix, nearest encounter distance-weighted adjacency matrix, and nearest encounter time-weighted adjacency matrix.

[0090] Among these methods, ablation experiments revealed that it is feasible to assess the interaction forces between ships using the speed *v* between different vertices at different times. This applies to the spacetime graph G. t By modeling the speed of the target ship, a speed-weighted adjacency matrix A can be obtained. V Speed-weighted adjacency matrix A V It can be represented as:

[0091]

[0092] Among these methods, ablation experiments revealed that it is feasible to assess the interaction forces between ships using the heading c between different vertices at different times. This applies to the spatiotemporal graph G. t By modeling the course of the target vessel, a course-weighted adjacency matrix A can be obtained. c Heading weighted adjacency matrix A c It can be represented as:

[0093]

[0094] Among them, in constructing the distance-weighted adjacency matrix A D At that time, three evaluation methods—Euclidean distance, Mahalanobis distance, and closest encounter distance—were compared. After comparative experiments, the closest encounter distance was ultimately adopted as the evaluation method for ship distance. For the spatiotemporal diagram G... t By modeling the nearest encounter distance of the target vessel, a nearest encounter distance-weighted adjacency matrix A can be obtained. D The nearest neighbor will be the distance-weighted adjacency matrix A. D It can be represented as:

[0095] Among these, the time of closest encounter (RTI) has become an indicator for assessing ship collision risk. Ablation experiments have shown that using the RTI between different vertices at different times to assess ship interaction forces is feasible. For the spatiotemporal graph G... t By modeling the nearest encounter time of the target ships, a nearest encounter time-weighted adjacency matrix A can be obtained. T Recently, we will encounter a time-weighted adjacency matrix A. T It can be represented as:

[0096]

[0097] The accuracy of the predicted trajectory tracking model is primarily measured by the distance error between the predicted trajectory and the actual trajectory. The coordinates of the actual trajectory are mainly obtained from AIS data. By combining the position and speed of the same ship in the actual AIS data for verification, the accuracy of the visual data reconstruction can be evaluated.

[0098] In some non-limiting embodiments, please refer to Figure 4 The prediction model 40 includes a spatiotemporal convolutional network 401 and a temporal extrapolator convolutional neural network 402. The spatiotemporal convolutional network 401 extracts the first embedding representation of spatiotemporal nodes based on the spatiotemporal graph set and the weighted adjacency matrix, encodes trajectory features, and outputs the second embedding representation. The temporal extrapolator convolutional neural network (TXP-CNN) 402 decodes the second embedding representation to obtain the trajectory of the target ship in the first time period in the future.

[0099] In practice, the spatiotemporal convolutional network 401 can create spatiotemporal embeddings. It extracts the embeddings of spatiotemporal nodes from the input graph. The input to the spatiotemporal convolutional network 401 is the spatiotemporal graph G. t =(V t E t ), that is, the feature vector of the graph of N observation points at T time steps. and the corresponding weighted adjacency matrix A∈R T×N×N The purpose of the spatiotemporal convolutional network 401 is to extract the first embedding representation of spatiotemporal nodes from the input graph, and output the second embedding representation V∈R by encoding trajectory features. T×N×Co C i and C O This represents the input and output scales of the feature vector.

[0100] Specifically, the spatiotemporal convolutional network 401 includes two spatiotemporal convolutional blocks and an output layer. Each spatiotemporal convolutional block consists of two temporal convolutional blocks and one spatial convolutional block. The spatial convolutional blocks are used to extract the trajectory features of the target ships, while the temporal convolutional blocks extract the temporal information between the target ships. The spatiotemporal convolutional blocks can have a sandwich structure, i.e., two temporal convolutions sandwiching one spatial convolution. Chebyshev polynomials can be used to approximate the result of graph convolution, achieving spatial convolution and avoiding the computationally high complexity of direct graph convolution. The temporal convolution extracts temporal features through one-dimensional causal convolution and an activation function, such as a gated linear unit (GLU).

[0101] In specific implementation, the temporal extrapolator convolutional neural network 402 receives features output by the spatiotemporal convolutional network 401 and treats the time dimension as a feature channel. The temporal extrapolator convolutional neural network 402 includes a series of convolutional neural networks (CNNs) with residual connections. Based on the TCN, the temporal extrapolator convolutional neural network 402 decodes the second embedding representation output from the spatiotemporal convolutional network 401 along the time dimension, thereby realizing the prediction of future trajectory points.

[0102] In a specific application scenario, the spatiotemporal convolutional network 401 can be an improved social bottleneck spatiotemporal graph convolution network (IS-BSTGCN).

[0103] This application validated the prediction performance of the above scheme by collecting actual data. First, it verified the effect of image-based ship trajectory tracking in supplementing missing trajectories. Data was collected in the bend area of ​​the Huangpu River, using continuous frame images from Huangpu River cameras, with a time interval of 40 seconds, totaling 618 images: 336 daytime images and 282 nighttime images. The pixel sizes of the daytime and nighttime images were 800×480 and 2560×1440, respectively. The trajectory prediction error was defined as the shortest distance between the actual latitude and longitude position of the AIS and the predicted trajectory. Finally, the average trajectory error obtained on the dataset was 32.90482m.

[0104] Next, the accuracy of the prediction model for the target ship's trajectory in the first time period was verified. The average displacement error (ADE) and final displacement error (FDE) were used to evaluate the accuracy of the ship's trajectory prediction. The final measured average displacement error and final displacement error were 25.68521 meters and 25.32279 meters, respectively.

[0105] Please refer to Figure 5 , Figure 5 A ship trajectory prediction device 50 is shown, which may include:

[0106] The acquisition module 501 is used to acquire a set of images collected for the same waterway area. The image set includes multiple waterway images, and there is a preset sampling interval between adjacent images.

[0107] The detection module 502 is used to detect ships in multiple waterway images to obtain the pixel coordinates of ships in each waterway image;

[0108] The confidence score calculation module 503 is used to calculate the confidence scores of the target ship and the candidate ship based on the pixel coordinate positions of the target ship and the candidate ship. The target ship is at least one ship in the first channel image among multiple channel images, and the candidate ship is a ship in other channel images other than the first channel image.

[0109] The dataset generation module 504 is used to associate the target ship with the candidate ship corresponding to the maximum confidence score to generate the image trajectory dataset of the target ship;

[0110] The prediction module 505 is used to predict the trajectory of the target ship in the first time period in the future based on the image trajectory dataset.

[0111] In specific implementation, the aforementioned ship trajectory prediction device 50 may correspond to a chip in a terminal device that has ship trajectory prediction function, such as a system-on-a-chip (SOC), a baseband chip, etc.; or correspond to a chip module in a terminal device that includes a ship trajectory prediction function; or correspond to a chip module with a data processing function chip; or correspond to a terminal device.

[0112] Other relevant descriptions of the ship trajectory prediction device 50 can be found in the descriptions in the foregoing embodiments, and will not be repeated here.

[0113] Regarding the modules / units included in the various devices and products described in the above embodiments, they can be software modules / units, hardware modules / units, or a combination of both. For example, for devices and products applied to or integrated into a chip, all modules / units can be implemented using hardware methods such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits. For devices and products applied to or integrated into a chip module, all modules / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using hardware methods such as circuits. The implementation is achieved through a software program that runs on a processor integrated within the chip module. The remaining modules / units (if any) can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into terminal equipment, each of their modules / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components within the terminal equipment. Alternatively, at least some modules / units can be implemented using a software program that runs on a processor integrated within the terminal equipment, while the remaining modules / units (if any) can be implemented using hardware methods such as circuits.

[0114] This application also discloses a storage medium, which is a computer-readable storage medium storing a computer program thereon. When the computer program is executed, it can perform the steps of the method shown in the foregoing embodiments. The storage medium may include read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc. The storage medium may also include non-volatile memory or non-transitory memory, etc.

[0115] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0116] In the embodiments of this application, "multiple" refers to two or more.

[0117] The descriptions of "first," "second," etc., appearing in the embodiments of this application are for illustrative purposes and to distinguish the objects being described. They have no order and do not indicate any special limitation on the number of devices in the embodiments of this application, nor do they constitute any limitation on the embodiments of this application.

[0118] The term "connection" in this application refers to various connection methods, such as direct connection or indirect connection, to achieve ship trajectory prediction between devices. This application does not limit this in any way.

[0119] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means.

[0120] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or ship trajectory prediction connection shown or discussed may be through some interfaces; the indirect coupling or ship trajectory prediction connection of apparatuses or units may be electrical, mechanical, or other forms.

[0122] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0123] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0124] The integrated unit implemented as a software functional unit described above can be stored in a computer-readable storage medium. This software functional unit, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute some steps of the methods described in the various embodiments of this application.

[0125] While this application discloses the above information, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application shall be determined by the scope defined in the claims.

Claims

1. A method for predicting ship trajectories, characterized in that, include: Acquire a set of images collected for the same waterway area, the set of images including multiple waterway images, with a preset sampling interval between adjacent images; Ship detection is performed on the multiple waterway images to obtain the pixel coordinates of ships in each waterway image; Based on the pixel coordinates of the target vessel and the pixel coordinates of the candidate vessels, a confidence score is calculated for the target vessel and the candidate vessels. The target vessel is at least one vessel in the first channel image among the multiple channel images, and the candidate vessels are vessels in other channel images besides the first channel image. The target vessel is associated with the candidate vessel corresponding to the maximum confidence score to generate an image trajectory dataset of the target vessel; Predict the trajectory of the target vessel within a first time period in the future based on the image trajectory dataset.

2. The ship trajectory prediction method according to claim 1, characterized in that, The calculation of the confidence scores between the target vessel and the candidate vessels includes: The similarity between the target ship and the candidate ship, the distance between the target ship and the candidate ship, and the area difference between the target ship and the candidate ship are calculated based on the pixel coordinates of the target ship and the candidate ship. The confidence score is generated based on the appearance similarity, the distance, and the area difference.

3. The ship trajectory prediction method according to claim 1, characterized in that, The image trajectory dataset for generating the target ship includes: The target vessel is associated with the candidate vessel corresponding to the maximum value of the confidence score, thereby generating a set of multi-frame image pixel positions of the target vessel in the multiple waterway images; Obtain the transformation matrix between the airway image and the remote sensing image; The transformation matrix is ​​used to convert each pixel position in the set of pixel positions of the multi-frame image into a projected position in the remote sensing image, and the latitude and longitude coordinates of the target ship are calculated based on the projected position and the known latitude and longitude coordinates in the remote sensing image. Based on the latitude and longitude coordinates of the target vessel and the sampling interval of adjacent images, calculate at least one of the following navigation information of the target vessel: speed, heading, nearest encounter distance, and nearest encounter time; The image trajectory dataset is constructed based on the navigation information of the target vessel.

4. The ship trajectory prediction method according to claim 1, characterized in that, The step of predicting the trajectory of the target vessel within a future first time period based on the image trajectory dataset includes: The image trajectory dataset and AIS data for the same waterway area are input into the prediction model to obtain the trajectory of the target vessel in the first time period in the future. The prediction model constructs a spatiotemporal graph set and a weighted adjacency matrix based on the image trajectory dataset and the AIS data, and generates the trajectory based on the spatiotemporal graph set and the weighted adjacency matrix. The spatiotemporal graph set represents the position of the target vessel at each time, and the weighted adjacency matrix represents the interaction relationship between vessels.

5. The ship trajectory prediction method according to claim 4, characterized in that, The prediction model constructs the weighted adjacency matrix based on the navigation information of the target vessel in the image trajectory dataset, and the weighted adjacency matrix includes at least one of the following: Speed-weighted adjacency matrix, heading-weighted adjacency matrix, nearest encounter distance-weighted adjacency matrix, nearest encounter time-weighted adjacency matrix.

6. The ship trajectory prediction method according to claim 4, characterized in that, The prediction model includes a spatiotemporal convolutional network and a temporal extrapolator convolutional neural network. The spatiotemporal convolutional network extracts a first embedding representation of spatiotemporal nodes based on the spatiotemporal graph set and the weighted adjacency matrix, encodes trajectory features, and outputs a second embedding representation. The temporal extrapolator convolutional neural network decodes the second embedding representation to obtain the trajectory of the target ship in the future within a first time period.

7. A ship trajectory prediction device, characterized in that, include: The acquisition module is used to acquire a set of images collected for the same waterway area, the set of images including multiple waterway images, with a preset sampling interval between adjacent images; The detection module is used to detect ships in the multiple waterway images to obtain the pixel coordinates of ships in each waterway image. The confidence score calculation module is used to calculate the confidence scores of the target ship and the candidate ship based on the pixel coordinate positions of the target ship and the candidate ship. The target ship is at least one ship in the first channel image among the multiple channel images, and the candidate ship is a ship in other channel images other than the first channel image. The dataset generation module is used to associate the target ship with the candidate ship corresponding to the maximum confidence score, and generate the image trajectory dataset of the target ship; The prediction module is used to predict the trajectory of the target vessel within a first time period in the future based on the image trajectory dataset.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when run by a processor, performs the steps of the ship trajectory prediction method according to any one of claims 1 to 6.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the ship trajectory prediction method according to any one of claims 1 to 6.

10. A computing device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor runs the computer program, it performs the steps of the ship trajectory prediction method according to any one of claims 1 to 6.

Images