Intelligent decision support method for maritime safety information based on multimodal fusion network

Abstract

Comprehensively extract and effectively integrate multi-modal maritime safety information to provide intelligent navigation decision support for mariners. [Solution] For the data processing end, a multimodal fusion network is proposed that can train various input information including latitude and longitude information of navigation AIS, text information of NAVTEX, vector information and raster information of electronic charts, and RGB information and HOG information of video images using various networks, and a Concat fusion structure based on multiple attention mechanisms MHA is introduced into the parallel structure of the multimodal network to fully extract the features of the multimodal information and effectively fuse them. For the result display end, a Transformer model based on generative pre-training is proposed, and a multimodal decision support method is proposed by understanding the context of the multimodal information of the fusion features.

Description

[Technical field]

[0001] The present invention relates to the technical field of maritime safety information, and in particular to an intelligent decision-making support method for maritime safety information based on multi-modal fusion network. [Background technology]

[0002] In the process of maritime transportation, in order to ensure the safety of navigation, every ship is equipped with many sensors that collect multimodal maritime safety information, i.e., navigation support information, including longitude and latitude information of navigation AIS, text information of NAVTEX, vector information and raster information of electronic charts, RGB information and HOG information of video images, etc. Therefore, it is very important to fully extract and effectively fuse the multimodal maritime safety information and provide intelligent decision support for navigation. However, the current method of extracting multimodal information features with a single model and simply merging them to perform feature fusion has problems such as reliability and data consistency, and does not achieve the degree of full and accurate use of real-time navigation support information. Therefore, even if a suitable navigation support device is installed, the dependence on the crew is not alleviated. The crew needs to manually extract the collected multimodal maritime safety information to judge the uncertainty of the ship's navigation situation and finally make appropriate decisions to ensure the safety of navigation.

[0003] Patent Document 1 discloses a method for automatic semantic classification of NAVTEX messages based on BLS, which performs low-granularity extraction on collected text data using only a single model in feature extraction of multimodal maritime safety information, but does not perform more serious information extraction using a parallel network based on an appropriate model in combination with the feature points of each piece of information, and does not perform more effective feature fusion using attention mechanism. Moreover, the result output mode is single and does not form a multi-mode output, so that sailors cannot easily understand and make decisions quickly. Therefore, a method is required to fully extract and effectively fuse multimodal maritime safety information to provide sailors with intelligent navigation decision-making support, which can improve work efficiency and further improve the navigation safety of ships and diversify navigation support functions. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] China Patent Application Publication No. 116719933 Summary of the Invention [Problem to be solved by the invention]

[0005] In order to solve the problems existing in the prior art, the present invention aims to propose an intelligent decision-making support method for maritime safety information based on a multi-modal fusion network, which is used to solve problems adapted to decision-making support, such as complete extraction of multi-modal data features, effective fusion, diversification of output forms, etc. The method can provide intelligent navigation decision-making support for sailors, so as to effectively reduce the work intensity of sailors and improve their work efficiency, further improving the navigation safety of ships and diversifying their navigation support functions. [Means for solving the problem]

[0006] In order to achieve the above object, the solution of the present invention is as follows.

[0007] A method for supporting intelligent decision-making of maritime safety information based on a multimodal fusion network according to one embodiment of the present invention includes the following steps S1 to S3: S1, constructing a multimodal fusion network that receives multimodal data input; S2, displaying the multimodal results of the multiple modes; and S3, decision support process.

[0008] S1, S11, constructing a multimodal network; S12, pre-processing the multi-modal data; S13, inputting multimodal data; S14, performing alignment and fusion on the features of the multi-modal parallel network.

[0009] S11, S111, a sub-step of constructing a time series network TSN; S112, a sub-step of constructing a text convolutional network T-CNN; S113, a substep of constructing an image convolutional network I-CNN; S114, a sub-step of constructing a time-series image convolutional network TICN.

[0010] In S111, the AIS data I with time series characteristics is generated using a gated neural network with long short-term memory according to the following formula: AIS and Electronic Chart Display and Information System, or ECDIS Data I ECDIS Vector data I Vector Process O TSN =TSN(I AIS ,I Vector ) Here, O TSN represents the output of TSN, and I AIS and I Vectorrepresents the input of TSN, and TSN represents a function based on a long short-term memory gated neural network structure.

[0011] In S112, the NAVTEX message data I with contextual dependencies is generated using a convolutional neural network for text vectorization according to the following formula: NAVTEX Process O T-CNN =T-CNN(I NAVTEX ) Here, O T-CNN represents the output of T-CNN, and I NAVTEX represents the input of T-CNN, and T-CNN represents a function based on the text convolutional network structure.

[0012] In S113, the video image, i.e., image data I is generated using a convolutional neural network for image digitization according to the following formula: Image RGB data of pixels in I RGB And, ECDIS data I ECDIS Raster data with image features I Raster Process O I-CNN =I-CNN(I RGB ,I Raster ) Here, O I-CNN represents the output of I-CNN, and I RGB and I Raster represents the input of I-CNN, and I-CNN represents a function based on the image convolutional network structure.

[0013] In S114, the image data I is generated using a space-time fusion convolutional neural network according to the following formula: Image Histogram of Oriented Gradient with Spatiotemporal Correlation in HOG Data I HOG Process O TICN =TICN(I HOG ) Here, O TICN represents the output of TICN, and I HOGrepresents the input of TICN, and TICN represents a function based on the time-series convolutional network structure.

[0014] In S12, the multimodal data is AIS data I AIS and NAVTEX Message Data I NAVTEX and Electronic Chart Data I ECDIS and video image data I Image Includes:

[0015] S12, S121, AIS data I AIS and a sub-step of preprocessing the S122, NAVTEX message data I NAVTEX and a sub-step of preprocessing the S123, Electronic Chart Data I ECDIS and a sub-step of preprocessing the S124, video image data I Image and a sub-step of preprocessing the image.

[0016] In S121, the AIS data I AIS contains the ship's longitude, latitude, speed, and direction of travel, and is a 4-group I AIS(k) =(longitude (k) ,latitude (k) ,speed (k) ,course (k) ) is constructed to obtain the input I AIS ={I AIS(1) ,I AIS(2) ,...,I AIS(k) ,...,I AIS(t)}, Here, I AIS(k) I AIS represents the AIS data corresponding to time k, and t is I AIS represents the total length of the data after preprocessing.

[0017] In S122, the NAVTEX message data I NAVTEX Store in text format and INAVTEX We adopt the text vectorization as utilized by T-CNN, and,I,according to the following formula: NAVTEX is converted to vectorized data as input to the text convolutional network T-CNN, I NAVTEX ={Word1,Word2,...,Word i ,...,Word L} where L is I NAVTEX The number of words after preprocessing is shown in Word i represents the one-hot encoding of the i-th word in the NAVTEX message.

[0018] In S123, the electronic chart data I ECDIS Vector data I Vector and raster data I Raster and the input I of the time series network TSN is stored in the format Vector and the input I of the image convolutional network I-CNN Raster and the formula is as follows: I Vector ={I Vector(1) ,I Vector(2) ,...,I Vector(k) ,...,I Vector(t)} I Raster ={I Raster(1) ,I Raster(2) ,...,I Raster(j) ,...,I Raster(m)} Here, I Vector(k) I Vector represents the vector data corresponding to time k in the Vector represents the total length of data after preprocessing, and I Raster(j) I Raster represents the j-th raster data in I Raster represents the number of raster images after preprocessing.

[0019] In S124, the video image data I Image RGB data I RGB and HOG data I HOGand the input I of the image convolution network I-CNN is stored in the format RGB and the input I of the time-series image network TICN HOG and the formula is as follows: I RGB ={I RGB(1) ,I RGB(2) ,...,I RGB(j) ,...,I RGB(m)} I HOG ={I HOG(1) ,I HOG(2) ,...,I HOG(q) ,...,I HOG(n)} Here, I RGB(j) I RGB represents the j-th RGB image data in I RGB represents the number of RGB images after preprocessing, and I HOG(q) I HOG represents the q-th HOG image data in I HOG represents the number of HOG images after preprocessing.

[0020] In S13, the preprocessed multi-modal data is input to the corresponding parallel networks TSN, T-CNN, I-CNN and TICN for training.

[0021] In S14, the parallel networks TSN, T-CNN, I-CNN and TICN are corresponding to the multiple attention mechanism MHA. TSN , O T―CNN , O I-CNN and O TICN Then, we perform Concat fusion on the multimodal fusion network to obtain the final output O Final =Contact(MHA(O TSN ), MHA(O T-CNN ), MHA(O I-CNN ), MHA(O TICN )) to get the

[0022] S2, S21, obtaining a generative pre-training model, i.e., a GPT model, through feedback training of large data; S22, outputting multiple modes; S23, displaying the result; In S21, the output of the GPT model GPT is expressed by the following formula: O GPT =GPT(O Final )={O GPT(TSN) ,O GPT(T-CNN) ,O GPT(I-CNN) ,O GPT(TICN)} Here, O GPT(TSN) , O GPT(T―CNN) , O GPT(I-CNN) and O GPT(TICN) are trained with GPT models. TSN , O T―CNN , O I-CNN and O TICN represents the output corresponding to

[0023] S22, S221, wake O Navi and a sub-step of outputting S222, Text O Text and a sub-step of outputting S223, Image O Image and outputting the sub-step of: In S221, based on the input multi-modal data, generate a trajectory description of the preset scene according to the following formula:

number

number

number

[0024] In S23, a scene display is provided in multiple modes for convenient reading, observation and monitoring.

[0025] In S3, the multi-modal outputs of navigation tracks, text and images obtained through multi-modal fusion are displayed, making it easier for mariners to understand multi-modal maritime safety information and make decisions quickly. Effect of the Invention

[0026] The present invention has the following beneficial effects compared to the prior art: 1. The present invention establishes TSN, thereby achieving alignment of AIS data and ECDIS data with respect to time characteristics. 2. The present invention constructs T-CNN, thereby achieving text feature coding of NAVTEX data. 3. The present invention establishes I-CNN, thereby achieving spatial feature fusion for ECDIS data and video image data. 4. The present invention constructs TICN, which achieves spatio-temporal feature fusion of video image data. 5. The present invention establishes a fusion structure based on MHA, which ensures that the parallel network can perform effective multi-modal fusion. 6. The present invention establishes a GPT model of multi-modal maritime safety information, thereby achieving multi-modal output to ensure a scene display that is convenient for reading, observing and monitoring, and to provide decision support. [Brief description of the drawings]

[0027] [Figure 1] 1 is a flow chart of an overall method for intelligent decision-making support of maritime safety information based on a multi-modal fusion network according to the present invention; [Diagram 2] FIG. 1 is a schematic diagram showing how the present invention aligns AIS data and ECDIS data with respect to time characteristics by constructing a TSN. [Diagram 3] FIG. 1 is a schematic diagram illustrating text feature coding of NAVTEX data by constructing a T-CNN in the present invention. [Figure 4] FIG. 1 is a schematic diagram showing spatial feature fusion for ECDIS data and video image data by constructing an I-CNN in the present invention; [Diagram 5] FIG. 1 is a schematic diagram showing spatio-temporal feature fusion of video image data by constructing a TICN in the present invention. [Figure 6] FIG. 1 is a schematic diagram showing how the present invention combines MHA and Concat to perform multimodal fusion for parallel networks. [Figure 7] FIG. 1 is a schematic diagram showing the generation of multi-modal outputs by constructing a GPT model of multi-modal maritime safety information in the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] In order to clarify the purpose, technical means and advantages of the embodiments of the present invention, the technical means of the embodiments of the present invention will be described clearly and completely below with reference to the drawings in the embodiments of the present invention, and it goes without saying that the described embodiments are not all the embodiments but only some of the embodiments of the present invention. Any other embodiments that a person skilled in the art can obtain based on the embodiments of the present invention without any creative effort shall be included in the scope of protection of the present invention.

[0029] As shown in FIG. 1 , the intelligent decision-making support method for maritime safety information based on multi-modal fusion network includes the following steps:

[0030] S1, construct a multimodal fusion network that receives multimodal data input.

[0031] S11, construct a multimodal network. As shown in Fig. 2, first, a time series network (TSN) is constructed to extract time series characteristics from the AIS track data (I AIS and Electronic Chart Display and Information System, or ECDIS Data I ECDIS Vector data I VectorProcessing the output of the TSN model TSN =TSN(I AIS ,I Vector As shown in Figure 3, we then construct a text convolutional network (T-CNN) to extract the NAVTEX message data I NAVTEX Processing the output of the T-CNN model, T-CNN =T-CNN(I NAVTEX As shown in Figure 4, the image convolution network I-CNN is then constructed to obtain the pixel RGB data I RGB and raster data I, which has image features in ECDIS data. Raster We process the I-CNN model output O I-CNN =I-CNN(I RGB ,I Raster As shown in Figure 5, finally, a time-series image convolution network (TICN) is constructed to obtain the histogram of oriented gradients (HOG) data I HOG Process the output of the TICN model TICN =TICN(I HOG ) to get the

[0032] S12, pre-processing the multi-modal data. First, AIS data I AIS and the four-group I AIS(k) =(longitude (k) ,latitude (k) ,speed (k) ,course (k) ) is constructed to obtain the input I AIS ={I AIS(1) ,I AIS(2) ,...,I AIS(k) ,...,I AIS(t)}. Next, get the NAVTEX message data I NAVTEX The input I of the text convolutional network T-CNN is preprocessed and vectorized. NAVTEX ={Word1,Word2,...,Word i ,...,Word L}. Next, get the electronic chart data I ECDIS and then input the vector data of the time series network TSN, I Vector ={I Vector(1) ,I Vector(2) ,...,I Vector(k) ,...,I Vector(t)} and the input I of the raster data for the image convolution network I-CNN Raster ={I Raster(1) ,I Raster(2) ,...,I Raster(j) ,...,I Raster(m) Finally, the video image data (I Image ) and then preprocess them into the RGB data input I RGB ={I RGB(1) ,I RGB(2) ,...,I RGB(j) ,...,I RGB(m)} and the HOG data input of the time-series image network TICN I HOG ={I HOG(1) ,I HOG(2) ,...,I HOG(q) ,...,I HOG(n)} to get

[0033] S13, input the multi-modal data. The pre-processed multi-modal data is input to the corresponding parallel networks TSN, T-CNN, I-CNN and TICN for training and learning.

[0034] S14, perform alignment and fusion of the features of the multimodal parallel networks. As shown in Fig. 6, the O-CNN, which corresponds to the parallel networks TSN, T-CNN, I-CNN, and TICN based on the multi-attention mechanism MHA, is TSN , O T―CNN , O I-CNN and O TICN Then, we perform Concat fusion on the multimodal fusion network to obtain the final output O Final =Contact(MHA(O TSN ), MHA(O T-CNN ), MHA(O I-CNN ), MHA(O TICN )) to get the

[0035] S2, displaying the multimodal results of multiple modes. As shown in Figure 7, based on the trained generative pre-trained model Transformer (GPT), the output O GPT =GPT(O Final )={O GPT(TSN) ,O GPT(T-CNN) ,O GPT(I-CNN) ,O GPT(TICN)}. Therefore, the output of the multiple modes is obtained by dividing the trajectory description of each generated preset scene.

number

number

number

[0036] S3, Support for decision making: The system displays multi-modal outputs such as ship tracks, text and images obtained through multi-modal fusion, making it easier for mariners to understand multi-modal maritime safety information and make faster decisions.

[0037] Finally, the following should be noted: The above embodiments are merely for explaining the technical means of the present invention, and are not intended to limit it, and the present invention has been described in detail with reference to the above embodiments, but it is possible to modify the technical means described in the above embodiments, or to implement equivalent replacements for part or all of the technical features, and it is obvious to those skilled in the art that such modifications and replacements do not depart from the essence of the corresponding technical means of the embodiments of the present invention.

[0038] (Additional Note) (Appendix 1) a step S1 of constructing a multimodal fusion network that receives multimodal data input; a step S2 of displaying a multimodal result of the plurality of modes; and a step S3 of supporting decision-making, S1, S11, constructing a multimodal network; S12, pre-processing the multi-modal data; S13, inputting multimodal data; S14, performing alignment and fusion on the features of the multi-modal parallel network; S11, S111, a sub-step of constructing a time series network TSN; S112, a sub-step of constructing a text convolutional network T-CNN; S113, a substep of constructing an image convolutional network I-CNN; S114, a sub-step of constructing a time-series image convolution network (TICN); In S111, the AIS data I with time series characteristics is generated using a gated neural network with long short-term memory according to the following formula: AIS and Electronic Chart Display and Information System, or ECDIS Data I ECDIS Vector data I Vector Process O TSN =TSN(I AIS ,I Vector ) Here, O TSN represents the output of TSN, and I AIS and I Vector represents the input of TSN, TSN represents a function based on the long short-term memory gate neural network structure, In S112, the NAVTEX message data I with contextual dependencies is generated using a convolutional neural network for text vectorization according to the following formula: NAVTEX Process O T-CNN =T-CNN(I NAVTEX ) Here, O T-CNN represents the output of T-CNN, and I NAVTEX represents the input of T-CNN, T-CNN represents the function based on the text convolutional network structure, In S113, the video image, i.e., image data I is generated using a convolutional neural network for image digitization according to the following formula: Image RGB data of pixels in I RGB And, ECDIS data I ECDIS Raster data with image features I Raster Process O I-CNN =I-CNN(I RGB ,I Raster ) Here, O I-CNN represents the output of I-CNN, and I RGB and I Raster represents the input of I-CNN, I-CNN represents a function based on the image convolutional network structure, In S114, the image data I is generated using a space-time fusion convolutional neural network according to the following formula: Image Histogram of Oriented Gradient with Spatiotemporal Correlation in HOG Data I HOG Process O TICN =TICN(I HOG ) Here, O TICN represents the output of TICN, and I HOG represents the input of TICN, TICN represents a function based on the time-series convolutional network structure, In S12, the multimodal data is AIS data I AIS and NAVTEX Message Data I NAVTEX and Electronic Chart Data I ECDIS and video image data I Image and S12, S121, AIS data I AIS and a sub-step of preprocessing the S122, NAVTEX message data I NAVTEX and a sub-step of preprocessing the S123, Electronic Chart Data I ECDIS and a sub-step of preprocessing the S124, video image data I Image and a sub-step of preprocessing the In S121, the AIS data I AIS contains the ship's longitude, latitude, speed, and direction of travel, and is a 4-group I AIS(k) =(longitude (k) ,latitude (k) ,speed (k) ,course (k) ) is constructed to obtain the input I AIS ={I AIS(1) ,I AIS(2) ,...,I AIS(k) ,...,I AIS(t)}, Here, I AIS(k) I AIS represents the AIS data corresponding to time k, and t is I AIS represents the total length of the data after preprocessing, In S122, the NAVTEX message data I NAVTEX Store in text format and I NAVTEX We adopt the text vectorization as utilized by T-CNN, and,I,according to the following formula: NAVTEX is converted to vectorized data as input to the text convolutional network T-CNN, I NAVTEX ={Word1,Word2,...,Word i ,...,Word L} where L is I NAVTEX The number of words after preprocessing is shown in Word i denotes the one-hot encoding of the i-th word in the NAVTEX message, In S123, the electronic chart data I ECDIS Vector data I Vectorand raster data I Raster and the input I of the time series network TSN is stored in the format Vector and the input I of the image convolutional network I-CNN Raster and the formula is as follows: I Vector ={I Vector(1) ,I Vector(2) ,...,I Vector(k) ,...,I Vector(t)} I Raster ={I Raster(1) ,I Raster(2) ,...,I Raster(j) ,...,I Raster(m)} Here, I Vector(k) I Vector represents the vector data corresponding to time k in the Vector represents the total length of data after preprocessing, and I Raster(j) I Raster represents the j-th raster data in I Raster represents the number of raster images after preprocessing, In S124, the video image data I Image RGB data I RGB and HOG data I HOG and the input I of the image convolution network I-CNN is stored in the format RGB and the input I of the time-series image network TICN HOG and the formula is as follows: I RGB ={I RGB(1) ,I RGB(2) ,...,I RGB(j) ,...,I RGB(m)} I HOG ={I HOG(1) ,I HOG(2) ,...,I HOG(q) ,...,I HOG(n)} Here, I RGB(j) I RGB represents the j-th RGB image data in I RGB represents the number of RGB images after preprocessing, and I HOG(q) I HOGrepresents the q-th HOG image data in I HOG represents the number of HOG images after preprocessing, In S13, the preprocessed multi-modal data is input to the corresponding parallel networks TSN, T-CNN, I-CNN and TICN for training; In S14, the parallel networks TSN, T-CNN, I-CNN and TICN are corresponding to the multiple attention mechanism MHA. TSN , O T―CNN , O I-CNN and O TICN Then, we perform Concat fusion on the multimodal fusion network to obtain the final output O Final =Contact(MHA(O TSN ), MHA(O T-CNN ), MHA(O I-CNN ), MHA(O TICN )) and S2, S21, obtaining a generative pre-training model, i.e., a GPT model, through feedback training of large data; S22, outputting multiple modes; S23, displaying the result; In S21, the output of the GPT model GPT is expressed by the following formula: O GPT =GPT(O Final )={O GPT(TSN) ,O GPT(T-CNN) ,O GPT(I-CNN) ,O GPT(TICN)} Here, O GPT(TSN) , O GPT(T―CNN) , O GPT(I-CNN) and O GPT(TICN) are trained with GPT models. TSN , O T―CNN , O I-CNN and O TICN represents the output corresponding to S22, S221, wake O Navi and a sub-step of outputting S222, Text O Textand a sub-step of outputting S223, Image O Image and outputting the sub-step of: In S221, based on the input multi-modal data, generate a trajectory description of the preset scene according to the following formula:

number

number

number

Claims

[Claim 1] S1 constructing a multimodal fusion network that receives multimodal data input; A step S2 of displaying a multimodal result of the plurality of modes; and a step S3 of supporting decision-making, S1, S11, constructing a multimodal network; S12, pre-processing the multi-modal data; S13, inputting multimodal data; S14. Performing alignment and fusion on features of the multi-modal parallel network; S11, S111, a sub-step of constructing a time series network TSN; S112, a sub-step of constructing a text convolutional network T-CNN; S113, a sub-step of constructing an image convolutional network I-CNN; S114, a sub-step of constructing a time-series image convolution network TICN; In S111, the AIS data I with time series characteristics is generated using a gated neural network with long short-term memory according to the following formula: ＡＩＳ and Electronic Chart Display and Information System, or ECDIS Data I. ＥＣＤＩＳ Vector information in Vector data I Ｖｅｃｔｏｒ Process O ＴＳＮ =TSN(I ＡＩＳ ,I Ｖｅｃｔｏｒ ) Here, O ＴＳＮ represents the output of TSN, and I ＡＩＳ and I Ｖｅｃｔｏｒ represents the input of the TSN, and the TSN represents a function based on a long short-term memory gated neural network structure; At S112, the NAVTEX message data I with contextual dependencies is extracted using a convolutional neural network for text vectorization according to the following formula: ＮＡＶＴＥＸ Process O Ｔ－ＣＮＮ =T-CNN(I ＮＡＶＴＥＸ ) Here, O Ｔ－ＣＮＮ represents the output of T-CNN, and I ＮＡＶＴＥＸ represents the input of T-CNN, T-CNN represents a function based on the text convolutional network structure, In S113, the video image, i.e., Image data I is converted into digital data using a convolutional neural network according to the following formula: Ｉｍａｇｅ Pixel RGB data in I ＲＧＢ And, ECDIS data I ＥＣＤＩＳ Raster data I having image features Ｒａｓｔｅｒ Process O Ｉ－ＣＮＮ =I-CNN(I ＲＧＢ ,I Ｒａｓｔｅｒ ) Here, O Ｉ－ＣＮＮ represents the output of I-CNN, and I ＲＧＢ and I Ｒａｓｔｅｒ represents the input of I-CNN, and I-CNN represents a function based on the image convolutional network structure. In S114, the image data I is calculated using a space-time fusion convolutional neural network according to the following formula: Ｉｍａｇｅ Histogram of Oriented Gradient (HOG) data I ＨＯＧ Process Oh ＴＩＣＮ =TICN(I ＨＯＧ ) Here, O ＴＩＣＮ represents the output of TICN, and I ＨＯＧ represents the input of TICN, TICN represents a function based on the time-series convolutional network structure, In S12, the multimodal data is AIS data I ＡＩＳ and NAVTEX Message Data I ＮＡＶＴＥＸ and Electronic Chart Data I ＥＣＤＩＳ and video image data I Ｉｍａｇｅ and S12, S121, AIS data I ＡＩＳ and a sub-step of preprocessing the S122, NAVTEX message data I ＮＡＶＴＥＸ and a sub-step of preprocessing the S123, Electronic Chart Data I ＥＣＤＩＳ and a sub-step of preprocessing the S124, video image data I Ｉｍａｇｅ and a sub-step of preprocessing the In S121, the AIS data I ＡＩＳ includes the longitude, latitude, speed, and direction of the ship's navigation, and the four-element group I ＡＩＳ（ｋ） =(longitude （ｋ） , Latitude （ｋ） , speed （ｋ） , course （ｋ） ) is constructed to obtain the input I ＡＩＳ = {I ＡＩＳ（１） , I ＡＩＳ（２） ,...,I ＡＩＳ（ｋ） , . . . , I ＡＩＳ（ｔ） }, Here, I ＡＩＳ（ｋ） I ＡＩＳ represents the AIS data corresponding to time k, and t is I ＡＩＳ represents the total length of the data after preprocessing, In S122, the NAVTEX message data I ＮＡＶＴＥＸ is stored in text format, and I ＮＡＶＴＥＸ We adopt text vectorization so that I can be utilized by T-CNN, according to the following formula: ＮＡＶＴＥＸ is converted into vectorized data as input to the text convolutional network T-CNN, I ＮＡＶＴＥＸ ={Word １ ,Word ２ ,...,Word ｉ ,...,Word Ｌ } Here, L is I ＮＡＶＴＥＸ indicates the number of words after preprocessing, and Word ｉ represents the one-hot encoding of the i-th word in the NAVTEX message, In S123, the electronic chart data I ＥＣＤＩＳ Vector data I Ｖｅｃｔｏｒ and Raster Data I Ｒａｓｔｅｒ and the input I of the time series network TSN is stored in the form Ｖｅｃｔｏｒ and the input I of the image convolutional network I-CNN Ｒａｓｔｅｒ and the formula is as follows: I Ｖｅｃｔｏｒ ={I Ｖｅｃｔｏｒ（１） ,I Ｖｅｃｔｏｒ（２） ,...,I Ｖｅｃｔｏｒ（ｋ） ,...,I Ｖｅｃｔｏｒ（ｔ） } I Ｒａｓｔｅｒ ={I Ｒａｓｔｅｒ（１） ,I Ｒａｓｔｅｒ（２） ,...,I Ｒａｓｔｅｒ（ｊ） ,...,I Ｒａｓｔｅｒ（ｍ） } Here, I Ｖｅｃｔｏｒ（ｋ） I Ｖｅｃｔｏｒ represents vector data corresponding to time k in the Ｖｅｃｔｏｒ represents the total length of data after preprocessing, and I Ｒａｓｔｅｒ（ｊ） I Ｒａｓｔｅｒ represents the j-th raster data in I Ｒａｓｔｅｒ represents the number of raster images after preprocessing, In S124, the video image data I Ｉｍａｇｅ RGB data I ＲＧＢ and HOG Data I ＨＯＧ and the input I of the image convolution network I-CNN is stored in the form ＲＧＢ and the input I of the time-series image network TICN ＨＯＧ and the formula is as follows: I ＲＧＢ ={I ＲＧＢ（１） ,I ＲＧＢ（２） ,...,I ＲＧＢ（ｊ） ,...,I ＲＧＢ（ｍ） } I ＨＯＧ ={I ＨＯＧ（１） ,I ＨＯＧ（２） ,...,I ＨＯＧ（ｑ） ,...,I ＨＯＧ（ｎ） } Here, I ＲＧＢ（ｊ） I ＲＧＢ represents the j-th RGB image data in I ＲＧＢ represents the number of RGB images after preprocessing, and I ＨＯＧ（ｑ） I ＨＯＧ represents the q-th HOG image data in I ＨＯＧ represents the number of HOG images after preprocessing, In S13, the preprocessed multi-modal data is input to the corresponding parallel networks TSN, T-CNN, I-CNN and TICN for training and learning; In S14, the O corresponding to the parallel networks TSN, T-CNN, I-CNN and TICN based on the multi-attention mechanism MHA is ＴＳＮ , O Ｔ―ＣＮＮ , O Ｉ－ＣＮＮ and O ＴＩＣＮ Then, we perform Concat fusion on the multimodal fusion network to obtain the final output O Ｆｉｎａｌ =Contact(MHA(O ＴＳＮ ), MHA(O Ｔ－ＣＮＮ ), MHA(O Ｉ－ＣＮＮ ), MHA(O ＴＩＣＮ )) and S2, S21: Obtaining a generative pre-training model, i.e., a GPT model, through feedback training of large data; S22, outputting multiple modes; S23, displaying the result; In S21, the output O of the GPT model ＧＰＴ is expressed by the following formula: O ＧＰＴ =GPT(O Ｆｉｎａｌ )={O ＧＰＴ（ＴＳＮ） ,O ＧＰＴ（Ｔ－ＣＮＮ） ,O ＧＰＴ（Ｉ－ＣＮＮ） ,O ＧＰＴ（ＴＩＣＮ） } Here, O ＧＰＴ（ＴＳＮ） , O ＧＰＴ（Ｔ―ＣＮＮ） , O ＧＰＴ（Ｉ－ＣＮＮ） and O ＧＰＴ（ＴＩＣＮ） are O trained with the GPT model, ＴＳＮ , O Ｔ―ＣＮＮ , O Ｉ－ＣＮＮ and O ＴＩＣＮ represents the output corresponding to S22, S221, wake O Ｎａｖｉ and a sub-step of outputting S222, Text O Ｔｅｘｔ and a sub-step of outputting S223, Image O Ｉｍａｇｅ and outputting the sub-step of: In S221, based on the input multi-modal data, generate a trajectory description of the preset scene according to the following formula: [0010] At S222, generate a textual description of the preset scene based on the input multimodal data according to the following formula: [0025] At S223, generate an image description of the preset scene based on the input multimodal data according to the following formula: [0030] In S23, providing a scene display in multiple modes for convenient reading, observation and monitoring; In S3, displaying the multi-modal outputs of the ship track, text and image obtained by multi-modal fusion makes it easier for mariners to understand the multi-modal maritime safety information and make decisions quickly; The present invention relates to an intelligent decision support method for maritime safety information based on a multimodal fusion network.

Images