Trajectory classification method based on cross-modal spatio-temporal feature dynamic fusion

Abstract

This invention provides a trajectory classification method based on dynamic fusion of cross-modal spatiotemporal features. The method divides the target sea area into multiple grids, maps the trajectory points of ships to a grid number sequence, and generates a trajectory text sequence. A trajectory image is then generated based on these trajectory points. A word embedding model maps the grid numbers in the trajectory text sequence to word vectors, and a bidirectional long short-term memory network combined with an attention mechanism is used to extract text features. A residual network based on dilated convolution is used to extract image features from the trajectory image. The text features and image features are aligned using a cross-attention mechanism, and a gated network is used for dynamic weighted fusion. The text features and the dynamically weighted image features are then fused and input into a classifier to obtain the ship type prediction result. Thus, this invention achieves fine-grained cross-modal fusion of trajectory text and image, improving the accuracy of ship trajectory classification and solving the problem of insufficient classification accuracy for low-resolution trajectory data.

Description

Technical Field

[0001] This invention relates to the field of water transport technology, and in particular to a trajectory classification method that dynamically fuses cross-modal spatiotemporal features. Background Technology

[0002] In recent years, the task of classifying ships at sea based on AIS data has become a research hotspot in both academia and industry. The goal of this task is to accurately classify different types of ships by analyzing their dynamic behavior and track characteristics. The practical significance of ship classification is multifaceted. For example, in the field of navigation safety, identifying different types of ships allows for the analysis of their behavioral patterns, thereby predicting potential risks and implementing targeted management measures; in the monitoring of illegal activities, ship classification technology helps to quickly detect illegal fishing, smuggling activities, or other abnormal behaviors; and in port management, classification information can be used to optimize berth allocation and port logistics scheduling.

[0003] With the increasing demand for intelligent methods in maritime traffic safety and ship operation management, how to efficiently and accurately classify maritime vessel trajectories has become an urgent research problem. Due to the high dimensionality, sparsity, and noise inherent in maritime vessel trajectory data, directly applying existing deep learning models often fails to achieve ideal results. Therefore, designing effective deep learning models that take into account the characteristics of maritime vessel trajectory data has become a key challenge in current research.

[0004] Existing technologies do not provide an effective solution for fully combining the temporal semantics of trajectory text with the spatiotemporal distribution information of trajectory images and performing fine-grained dynamic fusion. Specifically, existing single-modal methods (such as using only text sequences or images) are unable to fully capture the complex patterns of ship motion; while simple feature splicing fusion methods cannot solve the inconsistencies in semantic density and noise distribution between text and image modalities, resulting in the failure to fully leverage the advantages of cross-modal information complementarity, ultimately limiting further improvements in classification accuracy. Summary of the Invention

[0005] To address the problems of existing technologies, this invention proposes a trajectory classification method based on dynamic fusion of cross-modal spatiotemporal features. Through a dynamic alignment module, fine-grained cross-modal fusion of trajectory text and images is achieved, effectively combining spatiotemporal information to significantly improve the accuracy of ship trajectory classification and solve the problem of insufficient classification accuracy for low-resolution trajectory data.

[0006] To achieve the above technical effects, this invention provides a trajectory classification method that dynamically fuses cross-modal spatiotemporal features, comprising the following steps:

[0007] The target sea area is divided into multiple grids, and the trajectory points of ships are mapped to grid number sequences to generate trajectory text sequences; and trajectory images are generated based on the trajectory points.

[0008] The grid numbers in the trajectory text sequence are mapped to word vectors using a word embedding model, and text features are extracted using a bidirectional long short-term memory network combined with an attention mechanism.

[0009] Image features in the trajectory image are extracted using a residual network based on dilated convolution;

[0010] The text features and image features are aligned using a cross-attention mechanism, and the aligned image features are dynamically weighted and fused using a gating network.

[0011] The text features are fused with dynamically weighted image features and input into a classifier to obtain the ship type prediction result.

[0012] Furthermore, the step of mapping the ship's trajectory points to a grid number sequence and generating a trajectory text sequence includes:

[0013] The trajectory points of the same ship are arranged in chronological order to form a grid number sequence;

[0014] Based on pre-defined near-port area identification rules, grid numbers located in near-port areas are removed from the grid number sequence.

[0015] Furthermore, generating a trajectory image based on the trajectory points includes:

[0016] The access frequency of ship trajectory points in each grid is counted, and a single-channel grayscale image is generated; wherein, the grayscale intensity reflects the access frequency.

[0017] The access frequency, trajectory order, and time information are mapped to the RGB three channels respectively to generate a color trajectory image; wherein, the red channel maps the access frequency, the green channel maps the trajectory order, and the blue channel maps the time information.

[0018] Furthermore, the word embedding model is a word2vec model, pre-trained using the CBOW training strategy, with a context window size of 5 and a word embedding dimension of 100.

[0019] Furthermore, the output feature map of the dilated convolutional layer of the residual network based on dilated convolution is calculated using the following formula:

[0020] ;

[0021] in, Represents the convolution kernel. The void ratio, These are the row and column indices for traversing the convolution kernel, respectively. This is the output feature map of the previous layer. Indicates the output feature map The spatial coordinate index of the target pixel.

[0022] Furthermore, aligning the text features with the image features using a cross-attention mechanism includes:

[0023] Using the text features as the query and the image features as the key and value, the aligned image features are calculated through a multi-head cross-attention mechanism.

[0024] Furthermore, the step of dynamically weighting and fusing the aligned image features using a gating network includes:

[0025] The text features are concatenated with the aligned image features, and dynamic weights are generated through a gating network. The aligned image features and the original image features are then weighted and fused using the dynamic weights to obtain the dynamically weighted target image features.

[0026] Furthermore, the gated network includes a fully connected layer, a ReLU activation function layer, and a Sigmoid activation function layer connected in sequence, for outputting the dynamic weights; the dynamic weights are calculated based on the following formula:

[0027] ;

[0028] in, This involves concatenating the text features with the aligned image features. and The weights of the fully connected layer, It is the Sigmoid activation function. For batch size, The length of the text sequence. Dimensions represent image features.

[0029] Furthermore, the process of fusing the text features with the dynamically weighted image features and inputting the result into the classifier to obtain the ship type prediction includes:

[0030] Global pooling is performed on the text features and the dynamically weighted image features respectively to obtain global text features and global image features;

[0031] The text global features and the image global features are concatenated into a fusion feature;

[0032] The fused features are input into a fully connected classification layer and processed by a Softmax function to output the category prediction result.

[0033] Furthermore, the category prediction result is calculated based on the following formula:

[0034] ;

[0035] ;

[0036] in, and These represent the weights and biases of the fully connected classification layer, respectively. The text global features, For the global features of the image, For batch size, For text feature dimensions, Dimensions represent image features.

[0037] The trajectory classification method based on dynamic fusion of cross-modal spatiotemporal features described in this invention first involves gridding and numbering the study sea area, transforming the ship's trajectory into a grid-coded sequence of navigation data. Then, treating the ship trajectory sequence as natural text and the gridded trajectory image as image data, a dynamic alignment module is proposed, utilizing cross-attention and gating mechanisms to dynamically align text and image features. Finally, the image features adjusted by global pooling are concatenated with the text features adjusted by global pooling to form a fused representation. Thus, this invention achieves fine-grained cross-modal fusion of trajectory text and image through the dynamic alignment module, effectively combining spatiotemporal information, improving the ship trajectory classification accuracy by 2%–3%, and solving the problem of insufficient classification accuracy for low-resolution trajectory data. This provides more reliable technical support for intelligent maritime surveillance and port scheduling. Attached Figure Description

[0038] Figure 1 This is a flowchart of the steps of the trajectory classification method for dynamic fusion of cross-modal spatiotemporal features provided in an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of a dataset grid partitioning method used in the trajectory classification method for dynamic fusion of cross-modal spatiotemporal features according to an embodiment of the present invention;

[0040] Figure 3 An example image of a near-port grid dataset is provided for the trajectory classification method that dynamically fuses cross-modal spatiotemporal features according to an embodiment of the present invention.

[0041] Figure 4 A flowchart illustrating the steps of the trajectory classification method for dynamic fusion of cross-membrane spatiotemporal features provided in an embodiment of the present invention to generate the trajectory image;

[0042] Figure 5A flowchart illustrating the steps of the trajectory classification method for dynamic fusion of cross-membrane spatiotemporal features provided in an embodiment of the present invention for classifying and outputting prediction results;

[0043] Figure 6 This is a schematic diagram of the ablation experiment results of the trajectory classification method based on dynamic fusion of cross-membrane spatiotemporal features according to an embodiment of the present invention.

[0044] Figure 7 This is a schematic diagram of the model framework of the trajectory classification method based on dynamic fusion of cross-modal spatiotemporal features provided in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0046] It should be noted that references to "an embodiment," "embodiment," "example embodiment," etc., in this specification refer to the described embodiment including specific features, structures, or characteristics, but not every embodiment must include these specific features, structures, or characteristics. Furthermore, such expressions do not refer to the same embodiment. Moreover, when describing specific features, structures, or characteristics in conjunction with embodiments, whether or not explicitly described, it is indicated that incorporating such features, structures, or characteristics into other embodiments is within the knowledge of those skilled in the art.

[0047] Furthermore, certain terms are used in the specification and subsequent claims to refer to specific components or parts. Those skilled in the art will understand that manufacturers may use different names or terms to refer to the same component or part. This specification and subsequent claims do not distinguish components or parts by differences in name, but rather by differences in function. The terms "comprising" and "including" used throughout the specification and subsequent claims are open-ended and should be interpreted as "including but not limited to".

[0048] Before detailing the embodiments of the present invention, the technical concept of the present invention is briefly described: The present invention encodes ship trajectories into two modalities: text sequence and grid image. Through a dynamic alignment module, guided by text semantic features, the image features are cross-attention alignment and gating weight adjustment are performed to achieve fine-grained adaptive fusion of cross-modal features, thereby improving the classification accuracy of low-resolution, sparse trajectory data.

[0049] The trajectory classification method based on dynamic fusion of cross-modal spatiotemporal features will be described in detail below with reference to the accompanying drawings and specific embodiments. These embodiments are specific illustrations of the present invention and are helpful in understanding the invention, but should not be construed as limiting the invention.

[0050] Figure 1 This invention illustrates a trajectory classification method based on dynamic fusion of cross-modal spatiotemporal features, applicable to maritime vessel trajectory classification. The method includes the following steps:

[0051] S101: Divide the target sea area into multiple grids, map the ship's trajectory points to a grid number sequence, generate a trajectory text sequence, and generate a trajectory image based on the trajectory points.

[0052] S102: The grid numbers in the trajectory text sequence are mapped to word vectors through a word embedding model, and text features are extracted using a bidirectional long short-term memory network combined with an attention mechanism.

[0053] S103: Extract image features from trajectory images using a residual network based on dilated convolution.

[0054] S104: Align the text features with the image features through a cross-attention mechanism, and use a gating network to dynamically weight and fuse the aligned image features.

[0055] S105: The text features are fused with the dynamically weighted image features and input into the classifier to obtain the ship type prediction result.

[0056] The mapping of ship trajectory points to a grid number sequence to generate a trajectory text sequence in this embodiment includes: arranging the trajectory points of the same ship in chronological order to form a grid number sequence; and removing grid numbers located in the near-port area from the grid number sequence based on pre-defined near-port area identification rules. Specifically, the target sea area is first divided into grids, for example, but not limited to, dividing the Taiwan Strait into 28×31 regular grids (868 in total), and assigning a unique number to each grid. The ship's latitude and longitude coordinates are obtained from AIS (Automatic Identification System) data, and mapped to the corresponding grid number based on the grid the coordinates fall into. The trajectory points of the same ship are arranged in ascending order of timestamps to form a grid number sequence, i.e., a trajectory text sequence. To reduce the interference caused by the high density and complex trajectory patterns in the near-port area on model training, this embodiment removes grid numbers located in the near-port area from the sequence.

[0057] See Figure 4 In one specific embodiment, generating a trajectory image based on the trajectory points includes:

[0058] S1011: Calculate the access frequency of ship trajectory points in each grid and generate a single-channel grayscale image; wherein, the grayscale intensity reflects the access frequency;

[0059] S1012: The access frequency, trajectory order, and time information are mapped to the RGB three channels respectively to generate a color trajectory image; wherein, the red channel maps to the access frequency, the green channel maps to the trajectory order, and the blue channel maps to the time information. That is, in terms of image generation, this embodiment adopts two information encoding strategies: first, the access frequency of ship trajectory points in each grid is counted to generate a single-channel grayscale image, where grayscale intensity reflects the frequency; based on this, the access frequency, trajectory order, and time information are further mapped to the RGB three channels respectively, wherein the red channel encodes the frequency, the green channel encodes the sequence, and the blue channel encodes the time, generating a color trajectory image. This encoding method enables low-resolution images to carry richer spatiotemporal semantic information.

[0060] This embodiment specifically uses AIS data from the Taiwan Strait from January to March 2021, such as... Figure 2 As shown, the Taiwan Strait is divided into 28×31 grids, totaling 868 grids.

[0061] The grids in near-port areas, due to their high vessel density and more complex trajectory patterns, often have multifaceted impacts on vessel classification. Vessel activity is more frequent in port areas, with various vessel types converging, resulting in a large and unevenly distributed dataset, easily leading to data imbalance. This causes models to favor features from these high-density areas during training. Simultaneously, near-port vessel trajectories contain more local dynamic details, such as berthing and departure behaviors, providing rich semantics but also increasing noise. Models often struggle to fully capture these fine-grained differences. Therefore, in this embodiment, during data preprocessing, the grid numbers corresponding to near-port areas are removed from the original trajectory sequence, such as... Figure 3 As shown, this aims to reduce the adverse effects of the abnormally high density and complex ship motion characteristics in port areas on model training.

[0062] The dataset used in this embodiment is described in Table 1 below.

[0063] Table 1:

[0064] Dataset Raw AIS data Trajectory sequence January 2021 17,222,608 1,013,094 February 2021 14,966,148 935,384 March 2021 12,362,663 772,666

[0065] Furthermore, examples of trajectory text sequences generated based on the above dataset are shown in Table 2 below.

[0066] Table 2:

[0067] MMSI Path ShipType 636018098 775|711|678|677|581|518|485|420|356|322|226|194|131 Container ship 351291000 868|805|741|740|708|675|611|578|449|385|321|160|63 Bulk carrier

[0068] This embodiment specifically retrieves N ship records from the AIS database. Each record contains the ship's unique identifier (mmsi), timestamp (t), geographic coordinates (lon, lat), and ship type (y). Specifically, it can be denoted as... ,in Indicates the type of ship. A collection of all categories. Definition For the set of grid numbers (a total of 28*31=868 grids), the coordinates ( Mapped to the corresponding grid number After grid encoding, the trajectory points of the same ship are sorted by time to form a trajectory grid sequence. ,in This represents the length of the trajectory text sequence. For each grid cell, it statistically analyzes the frequency of ship visits, trajectory order, and time information, and constructs... Image , This represents the number of channels.

[0069] After the above data preprocessing, the trajectory of each ship can be simultaneously represented as a text sequence. With trajectory image , Indicates the first The sample includes trajectory text, trajectory images, and labels for real vessel types. The goal of maritime vessel trajectory classification is to learn a classification model. , This represents the trajectory text sequence in the embedding space. For any input... The model outputs a predicted category. ; .

[0070] To fully capture the temporal dependencies and key local information in the trajectory text, a word2vec model is preferred to vectorize each grid number. A network model combining bidirectional long short-term memory and self-attention (biLSTM+Attention) is used to extract the global semantic representation. Specifically, this embodiment uses the word2vec model to vectorize the trajectory text sequence. Given the relatively small size of the grid vocabulary (868 grids in this embodiment), a CBOW (Continuous Bag-of-Words) training strategy is adopted. The center word is predicted using a window size of 5, and a 100-dimensional word embedding vector is obtained, mapping each grid number to a low-dimensional dense vector representation.

[0071] After obtaining the word embedding sequence, it is fed into a bidirectional long short-term memory (BiLSTM) network. The BiLSTM network captures the forward and backward dependencies of the sequence through LSTM units in both the forward and backward directions. The hidden state at each time step is formed by concatenating the forward and backward hidden states, thus obtaining a temporal feature representation that incorporates bidirectional contextual information. Subsequently, a self-attention mechanism is introduced into the BiLSTM output layer. Attention weights are calculated for the hidden state at each time step using the learnable context vector, and the weighted summation of the feature vector is used as the global text representation.

[0072] Let the trajectory text sequence be ,in Let i be the grid number in the trajectory. A pre-trained word2vec model is used. ,in A vocabulary list composed of grid numbers. For the embedded dimension, each grid is numbered. Mapped to a dimensional vector Thus, the embedded sequence is obtained. As input to subsequent networks.

[0073] This embodiment preferably uses CBOW as the training method for word2vec. CBOW aims to predict the center word using context words. Its training process typically involves averaging or summing the context word vectors, resulting in lower computational cost and faster convergence, making it particularly suitable for scenarios with small datasets and vocabulary sizes. In this paper, the vocabulary formed after grid encoding is relatively small (868 words in total), and CBOW can obtain stable embedding representations with lower computational cost. In the CBOW method, given a center word... The context, i.e., the c grid numbers before and after. The goal of the model is to predict the center word. Its objective function is:

[0074] ;

[0075] The prediction probability is defined using softmax:

[0076] ;

[0077] In obtaining word embedding sequences After being input, the model calculates the forward and backward hidden states using a bidirectional LSTM, and the output of the forward LSTM is denoted as... and the inverse LSTM output is Then the overall hidden state at time i is:

[0078] ;

[0079] in, This represents the hidden state dimension of a single LSTM.

[0080] A self-attention mechanism is introduced onto the output of the BiLSTM. Each hidden state... An intermediate representation is obtained after a fully connected layer:

[0081] ;

[0082] in, This is the weight matrix. For bias, This represents the attention hidden layer dimension. It utilizes context vectors. For each Calculate attention score The attention weights are obtained by normalizing all scores using the softmax function.

[0083] ;

[0084] Finally, the hidden states at each time step are weighted and summed to form the global text representation:

[0085] ;

[0086] The BiLSTM+Attention structure can fully capture bidirectional temporal dependency information in trajectory text, and automatically focus on grid points that have a key impact on ship motion patterns through the attention mechanism. This is used to extract temporal dependencies and local key information within the text, generating a text representation with global semantics.

[0087] In step S103, for the 28×31 low-resolution trajectory image generated in this embodiment, using a traditional deep convolutional network (such as ResNet) would result in a drastic reduction in feature map size due to multiple downsampling operations, leading to the loss of key local details. Therefore, this embodiment preferably uses a Dilated Residual Network (DRN) as the image encoder. This encoder has a relatively shallow overall structure, designed to avoid the loss of key information caused by multiple downsampling, while utilizing dilated convolution to expand the receptive field.

[0088] For the input image In this embodiment, basic features are first extracted using two standard convolutional layers, with each layer employing... The convolution kernel and the number of output channels are set to 64. Pooling layer; next, two dilated convolutional layers are used to expand the receptive field, with the output feature map of the previous layer being... The formula for calculating a holed convolutional layer is:

[0089] ;

[0090] in, Represents the convolution kernel. The void ratio, , These are the row and column indices for traversing the convolution kernel.

[0091] Step S104 is specifically implemented through a dynamic alignment module, which includes two sub-steps: cross-attention alignment and gating fusion.

[0092] In the cross-attention alignment step, aligning the text features with the image features through a cross-attention mechanism includes: using the text features as the query and the image features as the key and value, calculating through a multi-head cross-attention mechanism to obtain the aligned image features.

[0093] Suppose that the text description of the ship's trajectory, after being processed by a text encoder, yields a feature matrix. .in For batch size, The length of the text sequence. This represents the text feature dimension. Meanwhile, the trajectory image is processed by an image encoder to obtain the feature matrix. .in Indicates the number of image blocks. Dimensions represent image features.

[0094] To enable each token in the text to adaptively focus on its associated image region, this embodiment employs a multi-head cross-attention mechanism. Specifically, for each text token, its corresponding aligned image features... It can be obtained through the following formula: ;

[0095] The formula for calculating each attention head is as follows:

[0096] ;

[0097] , , Let be the projection matrices of the key and value for the query at the i-th head. Dimensions for each head.

[0098] Furthermore, in the gating fusion step, the dynamic weighted fusion of aligned image features using a gating network includes: concatenating the text features with the aligned image features, generating dynamic weights through the gating network, and using the dynamic weights to weightedly fuse the aligned image features and the original image features to obtain dynamically weighted target image features. The gating network includes a fully connected layer, a ReLU activation function layer, and a Sigmoid activation function layer connected in sequence, used to output the dynamic weights.

[0099] Since simple cross-attention can be affected by noise or local mismatches, this paper introduces a gating mechanism to dynamically adjust the weights of image features. First, the text features X are aligned with the image features... By concatenating the components, we obtain the joint representation:

[0100] ;

[0101] Subsequently, dynamic weights were calculated using a gating network. :

[0102] ;

[0103] in, and It is a fully connected layer. This is the Sigmoid activation function. Utilizing... The image features are dynamically weighted to obtain the final fused image features:

[0104] ;

[0105] in, This represents element-wise multiplication. It provides global statistics on image features to supplement global information during the alignment process.

[0106] See Figure 5 In one specific embodiment, step S105 includes:

[0107] S1051: Perform global pooling on the text features and the dynamically weighted image features respectively to obtain global text features and global image features.

[0108] S1052: The text global features and image global features are concatenated into a fused feature.

[0109] S1053: Input the fused features into the fully connected classification layer, process them through the Softmax function, and output the category prediction result.

[0110] Specifically, for text features X and dynamically weighted image features... Perform global pooling separately to obtain:

[0111] ;

[0112] Then, the two are combined to form a fused representation:

[0113] ;

[0114] Finally, the fully connected classification layer outputs the category prediction:

[0115] ;

[0116] Where W and b are the weights and biases of the classification layer.

[0117] The following describes the technical effects achieved in a specific experiment in this embodiment.

[0118] In the experiment, the dataset was divided into training, validation, and test sets in a 6:2:2 ratio, and a 5-class classification experiment was conducted on the entire dataset. After preprocessing, the original AIS data generated trajectory text and trajectory images. The trajectory text was represented by a gridded sequence of numbers, and the trajectory images were fixed at 28×31 pixels. For the trajectory text data, a CBOW training strategy with a window size of 5 was used for word embedding, with a fixed 100-dimensional word embedding and a uniform sentence length of 50 characters; any exceeding this length was segmented. The batch size was set to 128, and the model was trained over 200 training epochs.

[0119] Three metrics are used to evaluate classification performance: precision, recall, and F1 score. The specific calculation formulas are as follows:

[0120] ;

[0121] ;

[0122] ;

[0123] Table 3 below shows the selection of eight comparison models and the verification of the effectiveness of the model of the method described in this embodiment on a real maritime AIS dataset.

[0124] Table 3:

[0125] Method precision recall f1_score accuracy TextRNN 0.9230 0.9100 0.9160 0.9200 TextCNN 0.9355 0.9350 0.9341 0.9368 textRCNN 0.9370 0.8990 0.9170 0.9300 Bilstm+attention 0.9540 0.9150 0.9330 0.9490 lenet 0.7780 0.6760 0.7130 0.7760 resNet18 0.8400 0.8160 0.8240 0.8370 drn 0.8520 0.8290 0.8400 0.8620 simple fusion 0.9622 ]]> ​ 0.9166 0.9347 ]]> ​ 0.9598 ]]> ​ ours 0.9792 0.9330 ]]> ​ 0.9560 0.9761

[0126] Table 3 above shows the weighted experimental results on the dataset from January to March 2021. These results indicate that, in terms of text modality, BiLSTM+Attention performs best among the four models. This demonstrates that using bidirectional LSTM to capture contextual dependencies and then weighting key information through an attention mechanism can more effectively extract temporal and local key information from trajectory text.

[0127] In terms of image modalities, the performance of LeNet, ResNet18 and DRN models is lower than that of text models. The main reason is that the resolution of the input trajectory image is only 28×31, which leads to information loss after multiple downsamplings in traditional deep networks.

[0128] In the cross-modal fusion part, the Simple Fusion method achieves significant performance improvement by directly concatenating global features from text and images to realize information complementarity. This indicates that text and image information are complementary, and simple fusion can leverage this advantage to some extent. Building upon this, the dynamic alignment module proposed in this paper further introduces cross-attention and gating networks to dynamically adjust image features under text guidance. Ultimately, this results in a 1.3%–1.6% improvement in all metrics compared to simple fusion, demonstrating the effectiveness of fine-grained cross-modal alignment for ship trajectory classification.

[0129] The results show that, compared with existing text-based or image-based single-modal models, the proposed cross-modal fusion achieves a 2%-3% improvement in classification accuracy and F1 score, demonstrating the effectiveness of combining gridded trajectory text with trajectory images.

[0130] To further verify the effectiveness of the modules in the proposed model, ablation experiments were conducted on the January 2021 dataset, and the experimental results were analyzed. The following experimental variations were designed:

[0131] w / o CBOW: Replace the CBOW-based training strategy with random initialization to verify the effect of pre-training on the semantic representation of trajectory text.

[0132] w / o DRN: Replace the dilated convolution-based DRN with a simple convolutional layer and examine whether the dilated convolution structure can capture local details more effectively on low-resolution images.

[0133] w / o CrossAtt: Remove the cross-attention mechanism in the dynamic alignment module and directly fuse text and image global features to explore the impact of fine-grained feature alignment on classification performance.

[0134] w / o Gated: Remove the gated network so that the dynamic alignment module retains only the cross-attention output, and verify the effect of adaptive weighted adjustment on the fusion of features of the original and aligned images.

[0135] Ablation test results as follows Figure 6 As shown in the experimental results, removing the word2vec pre-training significantly reduces the semantic expressive power of the text representation, reflecting the necessity of pre-training methods in capturing trajectory semantics. Replacing the dilated convolution-based DRN with traditional convolutional layers compromises the model's ability to preserve image details, demonstrating the advantages of DRN in expressing low-resolution images. Removing the cross-attention mechanism and gating network leads to a decrease in prediction metrics, showcasing the role of the dynamic alignment module in enhancing cross-modal feature complementarity. Experiments demonstrate that the synergistic effect of each module can improve the accuracy of ship trajectory classification, confirming the effectiveness of each component in the proposed model.

[0136] Furthermore, comparing the impact of the two image representation methods on multimodal ship trajectory classification tasks reveals that single-channel grayscale images only reflect the frequency of trajectory grid visits through grayscale intensity. While they can capture basic spatial distribution information of the trajectory, under low-resolution conditions, the information dimension is relatively singular, making it difficult to comprehensively express the details of ship motion. The RGB color value mapping strategy, on the other hand, maps the frequency, order, and temporal information of ship appearance to the red, green, and blue channels respectively, based on the original grayscale image. This allows the generated image to display local motion patterns and temporal changes. Experimental verification shows that introducing additional spatiotemporal information into low-resolution trajectory images not only compensates for the insufficient information content of single-channel images but also better complements text features, further improving the cross-modal fusion effect. This verifies that in maritime ship trajectory classification tasks, the semantic enhancement strategy of image branches has a significant impact on the model's discriminative ability.

[0137] The model framework of the method described in this embodiment is as follows: Figure 7As shown, firstly, in the data preprocessing stage, the latitude and longitude coordinates in the original AIS data are discretized using a pre-defined sea area grid, transforming the continuous trajectory into grid-coded text similar to natural language. Simultaneously, the access frequency of each grid is counted, generating a single-channel grayscale image. This image is then colored using RGB color values, combining trajectory sequence and time information to generate a trajectory image reflecting the ship's motion pattern. In the feature extraction stage, the text portion uses word2vec to map grid numbers to low-dimensional embeddings. Inspired by the trajectory prediction direction, an attention mechanism is used, employing a bidirectional LSTM combined with an attention mechanism to capture the temporal dependence and contextual semantics of the trajectory text. For the image portion, a Dilated Residual Network (DRN) based on dilated convolution is used to extract features from the low-resolution trajectory image. Finally, in the classification training stage, a cross-modal fusion method is designed. This method adaptively fuses global features from the text and image through a dynamic alignment mechanism, then outputs the final ship type prediction after passing through a fully connected layer and Softmax.

[0138] In summary, the trajectory classification method based on dynamic fusion of cross-modal spatiotemporal features provided by this invention constructs trajectory text sequences and trajectory images for each vessel in the research sea area. It uses a BiLSTM+Attention network and a DRN structure based on dilated convolution to learn features in both modalities. Furthermore, it designs a dynamic alignment module to achieve cross-modal interaction and dynamically adjust the fusion of features between the two modalities. Finally, the fused features are used to perform classification prediction via a feedforward neural network. On real maritime trajectory datasets, the method described in this invention demonstrates high accuracy in vessel trajectory classification tasks.

[0139] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0140] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A trajectory classification method based on dynamic fusion of cross-modal spatiotemporal features, characterized in that, Including the following steps: The target sea area is divided into multiple grids, and the trajectory points of ships are mapped to grid number sequences to generate trajectory text sequences; and trajectory images are generated based on the trajectory points. The grid numbers in the trajectory text sequence are mapped to word vectors using a word embedding model, and text features are extracted using a bidirectional long short-term memory network combined with an attention mechanism. Image features in the trajectory image are extracted using a residual network based on dilated convolution; The text features and image features are aligned using a cross-attention mechanism, and the aligned image features are dynamically weighted and fused using a gating network. The text features are fused with dynamically weighted image features and input into a classifier to obtain the ship type prediction result.

2. The method according to claim 1, characterized in that, The process of mapping ship trajectory points to a grid number sequence and generating a trajectory text sequence includes: The trajectory points of the same ship are arranged in chronological order to form a grid number sequence; Based on pre-defined near-port area identification rules, grid numbers located in near-port areas are removed from the grid number sequence.

3. The method according to claim 1, characterized in that, The process of generating a trajectory image based on the trajectory points includes: The access frequency of ship trajectory points in each grid is counted, and a single-channel grayscale image is generated; wherein, the grayscale intensity reflects the access frequency. The access frequency, trajectory order, and time information are mapped to the RGB three channels respectively to generate a color trajectory image; wherein, the red channel maps the access frequency, the green channel maps the trajectory order, and the blue channel maps the time information.

4. The method according to claim 1, characterized in that, The word embedding model is a word2vec model, which is pre-trained using the CBOW training strategy, with a context window size of 5 and a word embedding dimension of 100.

5. The method according to claim 1, characterized in that, The output feature map of the dilated convolutional layer of the residual network based on dilated convolution is calculated using the following formula: ； in, Represents the convolution kernel. The void ratio, These are the row and column indices for traversing the convolution kernel, respectively. This is the output feature map of the previous layer. Indicates the output feature map The spatial coordinate index of the target pixel.

6. The method according to claim 1, characterized in that, The step of aligning the text features with the image features using a cross-attention mechanism includes: Using the text features as the query and the image features as the key and value, the aligned image features are calculated through a multi-head cross-attention mechanism.

7. The method according to claim 6, characterized in that, The method of dynamically weighting and fusing aligned image features using a gated network includes: The text features are concatenated with the aligned image features, and dynamic weights are generated through a gating network. The aligned image features and the original image features are then weighted and fused using the dynamic weights to obtain the dynamically weighted target image features.

8. The method according to claim 7, characterized in that, The gated network comprises a fully connected layer, a ReLU activation function layer, and a Sigmoid activation function layer connected in sequence, used to output the dynamic weights; the dynamic weights are calculated based on the following formula: ； in, This involves concatenating the text features with the aligned image features. and The weights of the fully connected layer, It is the Sigmoid activation function. For batch size, The length of the text sequence. Dimensions represent image features.

9. The method according to claim 1, characterized in that, The process of fusing the text features with dynamically weighted image features and inputting the result into a classifier to obtain a ship type prediction includes: Global pooling is performed on the text features and the dynamically weighted image features respectively to obtain global text features and global image features; The text global features and the image global features are concatenated into a fusion feature; The fused features are input into a fully connected classification layer and processed by a Softmax function to output the category prediction result.

10. The method according to claim 1, characterized in that, The category prediction result is calculated based on the following formula: ； ； in, and These represent the weights and biases of the fully connected classification layer, respectively. The text global features, For the global features of the image, For batch size, For text feature dimensions, Dimensions represent image features.

Images