Intelligence decision support method for marine safety information based on multimodal fusion network

Abstract

To provide an intelligent navigation decision support to ship staff by fully extracting and effectively fusing multimodal marine safety information.SOLUTION: In order to propose a multimodal fusion network capable of training various input information including navigation AIS latitude and longitude information, NAVTEX text information, electronic nautical chart vector information, raster information, video image RGB information, and HOG information by using various networks, a Contact fusion structure based on a multiple attention mechanism MHA is introduced into a parallel structure of the multimodal network so that features of the multimodal information are sufficiently extracted and effectively fused. A multimodal decision support method is proposed for a result display end by suggesting a Transformer model based on generative preliminary training and understanding a context of the multimodal information of fusion features.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to the technical field of maritime safety information, and in particular to an intelligent decision-making support method for maritime safety information based on multimodal fusion network. [Background technology]

[0002] To ensure safe navigation during maritime transport, ships are equipped with numerous sensors that collect multimodal maritime safety information, or navigational aid information, including longitude and latitude information from AIS, text information from NAVTEX, vector and raster information from electronic charts, and RGB and HOG information from video images. Therefore, it is crucial to fully extract and effectively fuse multimodal maritime safety information to provide intelligent decision-making support for navigation. However, current methods of feature fusion, which extract multimodal information features using a single model and then simply merge them, have issues with reliability and data consistency, preventing the full and accurate use of real-time navigational aid information. Therefore, even with the appropriate navigational aid devices installed, the dependence on crew members remains low. Crew members must manually extract the collected multimodal maritime safety information to determine the uncertainty of the ship's navigation situation and ultimately make appropriate decisions to ensure safe navigation.

[0003] Patent Document 1 discloses a method for automatically classifying the semantics of NAVTEX messages based on BLS. In this method, the feature extraction of multimodal maritime safety information is performed using only a single model, resulting in low-granularity extraction of collected text data. However, the method does not combine the feature points of each piece of information with an appropriate model to perform more in-depth information extraction using a parallel network, nor does it perform more effective feature fusion using an attention mechanism. Furthermore, the resulting output mode is single, rather than forming multimodal outputs, which makes it difficult for mariners to easily understand and make decisions quickly. Therefore, a method is needed that fully extracts and effectively fuses multimodal maritime safety information to provide mariners with intelligent navigation decision-making support. This method can improve work efficiency, further enhance ship navigation safety, and diversify navigation support functions. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Chinese Patent Application Publication No. 116719933 Summary of the Invention [Problem to be solved by the invention]

[0005] In order to solve the problems existing in the prior art, the present invention aims to propose an intelligent decision-making support method for maritime safety information based on a multimodal fusion network, which is used to solve problems related to decision-making support, such as complete extraction of multimodal data features, effective fusion, diversification of output forms, etc. This method can provide intelligent navigation decision-making support for mariners, thereby effectively reducing the workload of mariners and improving their work efficiency, further improving the navigation safety of ships and diversifying their navigation support functions. [Means for solving the problem]

[0006] In order to achieve the above object, the solution of the present invention is as follows.

[0007] A method for supporting intelligent decision-making of maritime safety information based on a multimodal fusion network, which is one embodiment of the present invention, includes the following steps S1 to S3: S1, constructing a multimodal fusion network that receives multimodal data input; S2, displaying the multimodal results of the multiple modes; and S3, decision support process.

[0008] S1, S11, constructing a multimodal network; S12, preprocessing the multimodal data; S13, inputting multimodal data; S14, performing alignment and fusion on the features of the multimodal parallel network.

[0009] S11, S111, a sub-step of constructing a time-series network (TSN); S112, the substep of constructing a text convolutional network T-CNN; S113, a substep of constructing an image convolutional network I-CNN; S114, a substep of constructing a time-series image convolution network (TICN).

[0010] In S111, the AIS data I with time series characteristics is generated using a gated neural network with long short-term memory according to the following formula: AIS and Electronic Chart Display and Information System, i.e. ECDIS Data I ECDIS Vector data I Vector Process O TSN =TSN(I AIS ,I Vector ) where O TSN represents the output of the TSN, and I AIS and I Vectorrepresents the input of the TSN, and TSN represents a function based on the long short-term memory gated neural network structure.

[0011] In S112, the NAVTEX message data I with contextual dependencies is generated using a convolutional neural network for text vectorization according to the following formula: NAVTEX Process O T-CNN =T-CNN(I NAVTEX ) where O T-CNN represents the output of T-CNN, and I NAVTEX represents the input of T-CNN, and T-CNN represents a function based on the text convolutional network structure.

[0012] In S113, the video image, i.e., image data I, is calculated using a convolutional neural network for image digitization according to the following formula: Image RGB data of pixels in I RGB and ECDIS data I ECDIS Raster data with image features in I Raster Process O I-CNN =I-CNN(I RGB ,I Raster ) where O I-CNN represents the output of I-CNN, and I RGB and I Raster represents the input of I-CNN, and I-CNN represents a function based on the image convolutional network structure.

[0013] In S114, the image data I is computed using a spatiotemporal convolutional neural network according to the following formula: Image Histogram of Oriented Gradient with Spatiotemporal Correlation in HOG Data I HOG Process O TICN =TICN(I HOG ) where O TICN represents the output of TICN, and I HOGrepresents the input of TICN, and TICN represents a function based on the time-series convolutional network structure.

[0014] In S12, the multimodal data is AIS data I AIS and NAVTEX message data I NAVTEX and Electronic Chart Data I ECDIS and video image data I Image Includes:

[0015] S12, S121, AIS data I AIS a sub-step of preprocessing the S122, NAVTEX message data I NAVTEX a sub-step of preprocessing the S123, Electronic Chart Data I ECDIS a sub-step of preprocessing the S124, video image data I Image and a sub-step of preprocessing the

[0016] In S121, the AIS data I AIS contains the longitude, latitude, speed, and direction of a ship's navigation, and the four-element group I AIS(k) =(longitude (k) ,latitude (k) ,speed (k) ,course (k) ) is constructed as the input I of the time-series network TSN. AIS ={I AIS(1) ,I AIS(2) ,...,I AIS(k) ,...,I AIS(t)}, where I AIS(k) I AIS represents the AIS data corresponding to time k, and t is I AIS represents the total length of the data after preprocessing.

[0017] In S122, the NAVTEX message data I NAVTEX Store in text format, INAVTEX We adopt text vectorization as used by T-CNN, and calculate I according to the following formula: NAVTEX is converted into vectorized data as input to the text convolutional network T-CNN, I NAVTEX ={Word1,Word2,...,Word i ,...,Word L} where L is I NAVTEX indicates the number of words after preprocessing, and Word i represents the one-hot encoding of the i-th word in the NAVTEX message.

[0018] In S123, the electronic chart data I ECDIS Vector data I Vector and raster data I Raster and store them in the form of the input I of the time series network TSN. Vector and the input I of the image convolutional network I-CNN Raster and the formula is as follows: I Vector ={I Vector(1) ,I Vector(2) ,...,I Vector(k) ,...,I Vector(t)} I Raster ={I Raster(1) ,I Raster(2) ,...,I Raster(j) ,...,I Raster(m)} Here, I Vector(k) I Vector represents the vector data corresponding to time k in the Vector represents the total length of data after preprocessing, and I Raster(j) I Raster represents the j-th raster data in I Raster represents the number of raster images after preprocessing.

[0019] In S124, the video image data I Image RGB data I RGB and HOG Data I HOGand the input I of the image convolutional network I-CNN is stored in the format RGB and the input I of the time series image network TICN HOG and the formula is as follows: I RGB ={I RGB(1) ,I RGB(2) ,...,I RGB(j) ,...,I RGB(m)} I HOG ={I HOG(1) ,I HOG(2) ,...,I HOG(q) ,...,I HOG(n)} where I RGB(j) I RGB represents the j-th RGB image data in I RGB represents the number of RGB images after preprocessing, and I HOG(q) I HOG represents the q-th HOG image data in I HOG represents the number of HOG images after preprocessing.

[0020] In S13, the preprocessed multimodal data is input to the corresponding parallel networks TSN, T-CNN, I-CNN and TICN for training.

[0021] In S14, the parallel networks TSN, T-CNN, I-CNN and TICN are implemented based on the multiple attention mechanism MHA. TSN , O T―CNN , O I-CNN and O TICN Concat fusion is performed on the multimodal fusion network to obtain the final output O Final =Contact(MHA(O TSN ), MHA(O T-CNN ), MHA(O I-CNN ), MHA(O TICN )) to get the

[0022] S2, S21, obtain a generative pre-training model, i.e., a GPT model, through feedback training of large data; S22, outputting multiple modes; S23, displaying the results; In S21, the output of the GPT model GPT is expressed by the following formula: O GPT =GPT(O Final )={O GPT(TSN) ,O GPT(T-CNN) ,O GPT(I-CNN) ,O GPT(TICN)} where O GPT(TSN) , O GPT(T―CNN) , O GPT(I-CNN) and O GPT(TICN) are O trained with GPT models, respectively. TSN , O T―CNN , O I-CNN and O TICN represents the output corresponding to

[0023] S22, S221, wake O Navi a sub-step that outputs S222, Text O Text a sub-step that outputs S223, Image O Image and outputting: In S221, based on the input multimodal data, a track description of the preset scene is generated according to the following formula:

number

number

number

[0024] In S23, a scene display is provided in multiple modes for convenient reading, observation and monitoring.

[0025] In S3, the multi-modal output of the track, text and images obtained through multi-modal fusion is displayed, making it easier for mariners to understand the multi-modal maritime safety information and make decisions quickly. [Effects of the Invention]

[0026] The present invention has the following beneficial effects compared to the prior art. 1. The present invention establishes TSN, thereby achieving alignment of AIS data and ECDIS data with respect to time characteristics. 2. The present invention builds T-CNN to achieve text feature coding of NAVTEX data. 3. The present invention builds I-CNN, which achieves spatial feature fusion for ECDIS data and video image data. 4. The present invention builds TICN, which achieves spatiotemporal feature fusion of video image data. 5. The present invention builds a fusion structure based on MHA, which ensures that the parallel network can perform effective multimodal fusion. 6. The present invention builds a GPT model of multi-modal maritime safety information, thereby achieving multi-modal output to ensure a scene display that is convenient for reading, observing and monitoring, and to provide decision support. [Brief explanation of the drawings]

[0027] [Figure 1] 1 is a flowchart of the entire method for intelligent decision-making support of maritime safety information based on multimodal fusion network according to the present invention; [Figure 2] FIG. 1 is a schematic diagram showing how the TSN is constructed in the present invention to align AIS data and ECDIS data with respect to time characteristics. [Figure 3] FIG. 1 is a schematic diagram illustrating text feature coding of NAVTEX data by constructing a T-CNN in the present invention. [Figure 4] FIG. 1 is a schematic diagram illustrating spatial feature fusion for ECDIS data and video image data by constructing an I-CNN in the present invention. [Figure 5] FIG. 1 is a schematic diagram illustrating spatiotemporal feature fusion of video image data by constructing a TICN in the present invention. [Figure 6] FIG. 1 is a schematic diagram illustrating the combination of MHA and Concat to perform multimodal fusion on parallel networks in the present invention. [Figure 7] FIG. 1 is a schematic diagram illustrating generating multi-modal outputs by constructing a GPT model of multi-modal maritime safety information in the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0028] In order to clarify the purpose, technical means and advantages of the embodiments of the present invention, the technical means of the embodiments of the present invention will be described clearly and completely below with reference to the drawings in the embodiments of the present invention, and it goes without saying that the described embodiments are not all embodiments but only some embodiments of the present invention. Any other embodiments that a person skilled in the art can obtain based on the embodiments of the present invention without any creative effort shall all be included in the scope of protection of the present invention.

[0029] As shown in Figure 1, the intelligent decision-making support method for maritime safety information based on multimodal fusion network includes the following steps:

[0030] S1, construct a multimodal fusion network that receives multimodal data input.

[0031] S11: Construct a multimodal network. As shown in Figure 2, first, a time series network (TSN) is constructed to extract time series characteristics from the AIS track data. AIS and Electronic Chart Display and Information System, i.e. ECDIS Data I ECDIS Vector data I Vectorand process the output of the TSN model. TSN =TSN(I AIS ,I Vector ) is obtained. Next, as shown in Figure 3, a text convolutional network (T-CNN) is constructed to extract the NAVTEX message data I NAVTEX and process the output of the T-CNN model. T-CNN =T-CNN(I NAVTEX ) is obtained. Next, as shown in Figure 4, an image convolutional network (I-CNN) is constructed to obtain the pixel RGB data I RGB and raster data I, which has image features in ECDIS data. Raster and process the output of the I-CNN model. I-CNN =I-CNN(I RGB ,I Raster As shown in Figure 5, finally, a time-series image convolution network (TICN) is constructed to obtain the histogram of oriented gradients (HOG) data I HOG Process the output of the TICN model TICN =TICN(I HOG ) to get the

[0032] S12, preprocessing multimodal data. First, AIS data I AIS and the four-group I AIS(k) =(longitude (k) ,latitude (k) ,speed (k) ,course (k) ) is constructed as the input I of the time-series network TSN. AIS ={I AIS(1) ,I AIS(2) ,...,I AIS(k) ,...,I AIS(t)}. Next, get the NAVTEX message data I NAVTEX The text is preprocessed and vectorized, and then input to the text convolutional network T-CNN. NAVTEX ={Word1,Word2,...,Word i ,...,Word LNext, get the electronic chart data I ECDIS and then preprocess the vector data input I of the time series network TSN. Vector ={I Vector(1) ,I Vector(2) ,...,I Vector(k) ,...,I Vector(t)} and the input I of raster data for the image convolutional network I-CNN Raster ={I Raster(1) ,I Raster(2) ,...,I Raster(j) ,...,I Raster(m) Finally, the video image data (I Image ) and then preprocess them into the RGB data input I of the image convolutional network I-CNN. RGB ={I RGB(1) ,I RGB(2) ,...,I RGB(j) ,...,I RGB(m)} and the HOG data input of the time series image network TICN I HOG ={I HOG(1) ,I HOG(2) ,...,I HOG(q) ,...,I HOG(n)} to get

[0033] S13: Input the multimodal data. The preprocessed multimodal data is input to the corresponding parallel networks TSN, T-CNN, I-CNN, and TICN for training.

[0034] S14: Align and merge the features of the multimodal parallel network. As shown in Figure 6, the O corresponding to the parallel networks TSN, T-CNN, I-CNN, and TICN based on the multi-attention mechanism MHA. TSN , O T―CNN , O I-CNN and O TICN Concat fusion is performed on the multimodal fusion network to obtain the final output O Final =Contact(MHA(O TSN ), MHA(O T-CNN ), MHA(O I-CNN ), MHA(O TICN )) to get the

[0035] S2, multimodal results are displayed. As shown in Figure 7, based on the trained generative pre-trained Transformer (GPT), the output O GPT =GPT(O Final )={O GPT(TSN) ,O GPT(T-CNN) ,O GPT(I-CNN) ,O GPT(TICN) Therefore, the output of multiple modes is obtained by the track description of each generated preset scene.

number

number

number

[0036] S3, Decision Support: This system displays multimodal outputs such as ship tracks, text, and images obtained through multimodal fusion, making it easier for mariners to understand multimodal maritime safety information and make decisions quickly.

[0037] Finally, it should be noted that the above embodiments are merely for explaining the technical means of the present invention and are not intended to limit the same, and the present invention has been described in detail with reference to the above embodiments, but it is also possible to modify the technical means described in the above embodiments or to replace part or all of the technical features with equivalents, and it is obvious to those skilled in the art that such modifications and replacements do not deviate from the scope of the technical means of the embodiments of the present invention.

[0038] (Addendum) (Appendix 1) S1 constructing a multimodal fusion network that receives multimodal data input; a step S2 of displaying a multimodal result of the plurality of modes; a step S3 of supporting decision-making, S1, S11, constructing a multimodal network; S12, preprocessing the multimodal data; S13, inputting multimodal data; S14, performing alignment and fusion on the features of the multimodal parallel network; S11, S111, a sub-step of constructing a time-series network (TSN); S112, the substep of constructing a text convolutional network T-CNN; S113, a substep of constructing an image convolutional network I-CNN; S114, a sub-step of constructing a time-series image convolution network (TICN); In S111, the AIS data I with time series characteristics is generated using a gated neural network with long short-term memory according to the following formula: AIS and Electronic Chart Display and Information System, i.e. ECDIS Data I ECDIS Vector data I Vector Process O TSN =TSN(I AIS ,I Vector ) where O TSN represents the output of the TSN, and I AIS and I Vector represents the input of TSN, TSN represents a function based on the long short-term memory gated neural network structure, In S112, the NAVTEX message data I with contextual dependencies is generated using a convolutional neural network for text vectorization according to the following formula: NAVTEX Process O T-CNN =T-CNN(I NAVTEX ) where O T-CNN represents the output of T-CNN, and I NAVTEX represents the input of T-CNN, T-CNN represents a function based on the text convolutional network structure, In S113, the video image, i.e., image data I, is calculated using a convolutional neural network for image digitization according to the following formula: Image RGB data of pixels in I RGB and ECDIS data I ECDIS Raster data with image features in I Raster Process O I-CNN =I-CNN(I RGB ,I Raster ) where O I-CNN represents the output of I-CNN, and I RGB and I Raster represents the input of I-CNN, and I-CNN represents a function based on the image convolutional network structure. In S114, the image data I is computed using a spatiotemporal convolutional neural network according to the following formula: Image Histogram of Oriented Gradient with Spatiotemporal Correlation in HOG Data I HOG Process O TICN =TICN(I HOG ) where O TICN represents the output of TICN, and I HOG represents the input of TICN, TICN represents a function based on the time-series convolutional network structure, In S12, the multimodal data is AIS data I AIS and NAVTEX message data I NAVTEX and Electronic Chart Data I ECDIS and video image data I Image and S12, S121, AIS data I AIS a sub-step of preprocessing the S122, NAVTEX message data I NAVTEX a sub-step of preprocessing the S123, Electronic Chart Data I ECDIS a sub-step of preprocessing the S124, video image data I Image and a sub-step of preprocessing the In S121, the AIS data I AIS contains the longitude, latitude, speed, and direction of a ship's navigation, and the four-element group I AIS(k) =(longitude (k) ,latitude (k) ,speed (k) ,course (k) ) is constructed as the input I of the time-series network TSN. AIS ={I AIS(1) ,I AIS(2) ,...,I AIS(k) ,...,I AIS(t)}, Here, I AIS(k) I AIS represents the AIS data corresponding to time k, and t is I AIS represents the total length of the data after preprocessing, In S122, the NAVTEX message data I NAVTEX Store in text format, I NAVTEX We adopt text vectorization as used by T-CNN, and calculate I according to the following formula: NAVTEX is converted into vectorized data as input to the text convolutional network T-CNN, I NAVTEX ={Word1,Word2,...,Word i ,...,Word L} where L is I NAVTEX indicates the number of words after preprocessing, and Word i denotes the one-hot encoding of the i-th word in the NAVTEX message, In S123, the electronic chart data I ECDIS Vector data I Vectorand raster data I Raster and store them in the form of the input I of the time series network TSN. Vector and the input I of the image convolutional network I-CNN Raster and the formula is as follows: I Vector ={I Vector(1) ,I Vector(2) ,...,I Vector(k) ,...,I Vector(t)} I Raster ={I Raster(1) ,I Raster(2) ,...,I Raster(j) ,...,I Raster(m)} Here, I Vector(k) I Vector represents the vector data corresponding to time k in the Vector represents the total length of data after preprocessing, and I Raster(j) I Raster represents the j-th raster data in I Raster represents the number of raster images after preprocessing, In S124, the video image data I Image RGB data I RGB and HOG Data I HOG and the input I of the image convolutional network I-CNN is stored in the format RGB and the input I of the time series image network TICN HOG and the formula is as follows: I RGB ={I RGB(1) ,I RGB(2) ,...,I RGB(j) ,...,I RGB(m)} I HOG ={I HOG(1) ,I HOG(2) ,...,I HOG(q) ,...,I HOG(n)} Here, I RGB(j) I RGB represents the j-th RGB image data in I RGB represents the number of RGB images after preprocessing, and I HOG(q) I HOGrepresents the q-th HOG image data in I HOG represents the number of HOG images after preprocessing, In S13, the preprocessed multi-modal data is input to the corresponding parallel networks TSN, T-CNN, I-CNN and TICN for training; In S14, the parallel networks TSN, T-CNN, I-CNN and TICN are implemented based on the multiple attention mechanism MHA. TSN , O T―CNN , O I-CNN and O TICN Concat fusion is performed on the multimodal fusion network to obtain the final output O Final =Contact(MHA(O TSN ), MHA(O T-CNN ), MHA(O I-CNN ), MHA(O TICN )) and S2, S21, obtain a generative pre-training model, i.e., a GPT model, through feedback training of large data; S22, outputting multiple modes; S23, displaying the results; In S21, the output of the GPT model GPT is expressed by the following formula: O GPT =GPT(O Final )={O GPT(TSN) ,O GPT(T-CNN) ,O GPT(I-CNN) ,O GPT(TICN)} where O GPT(TSN) , O GPT(T―CNN) , O GPT(I-CNN) and O GPT(TICN) are O trained with GPT models, respectively. TSN , O T―CNN , O I-CNN and O TICN represents the output corresponding to S22, S221, wake O Navi a sub-step that outputs S222, Text O Texta sub-step that outputs S223, Image O Image and outputting: In S221, based on the input multimodal data, a track description of the preset scene is generated according to the following formula:

number

number

number

Claims

[Claim 1] S1 constructing a multimodal fusion network that receives multimodal data input; a step S2 of displaying a multimodal result of the plurality of modes; and a step S3 of supporting decision-making, S1 is, S11, constructing a multimodal network; S12, preprocessing the multimodal data; S13, inputting multimodal data; S14, performing alignment and fusion on features of the multimodal parallel network; S11 is, S111, a sub-step of constructing a time series network TSN; S112, a substep of constructing a text convolutional network T-CNN; S113, a substep of constructing an image convolutional network I-CNN; S114, a sub-step of constructing a time-series image convolutional network TICN; In S111, the AIS data I with time series characteristics is generated using a gated neural network with long short-term memory according to the following equation: ＡＩＳ and Electronic Chart Display and Information System, or ECDIS Data I. ＥＣＤＩＳ Vector information in Vector data I Ｖｅｃｔｏｒ Process O ＴＳＮ =TSN(I ＡＩＳ ,I Ｖｅｃｔｏｒ ) Here, O ＴＳＮ represents the output of the TSN, and I ＡＩＳ and I Ｖｅｃｔｏｒ represents the input of the TSN, and TSN represents a function based on a long short-term memory gated neural network structure; At S112, the NAVTEX message data I with contextual dependencies is extracted using a convolutional neural network for text vectorization according to the following equation: ＮＡＶＴＥＸ Process O Ｔ－ＣＮＮ =T-CNN(I ＮＡＶＴＥＸ ) Here, O Ｔ－ＣＮＮ represents the output of the T-CNN, and I ＮＡＶＴＥＸ represents the input of T-CNN, T-CNN represents a function based on the text convolutional network structure, In S113, the video image, i.e., image data I, is calculated using a convolutional neural network for image digitization according to the following formula: Ｉｍａｇｅ Pixel RGB data I ＲＧＢ and ECDIS data I ＥＣＤＩＳ Raster data I with image features Ｒａｓｔｅｒ Process O Ｉ－ＣＮＮ =I-CNN(I ＲＧＢ ,I Ｒａｓｔｅｒ ) Here, O Ｉ－ＣＮＮ represents the output of I-CNN, and I ＲＧＢ and I Ｒａｓｔｅｒ represents the input of I-CNN, and I-CNN represents a function based on the image convolutional network structure. In S114, the image data I is calculated using a space-time fusion convolutional neural network according to the following formula: Ｉｍａｇｅ Histogram of Oriented Gradient (HOG) data I with spatiotemporal correlation ＨＯＧ Process Oh ＴＩＣＮ =TICN(I ＨＯＧ ) Here, O ＴＩＣＮ represents the output of TICN, and I ＨＯＧ represents the input of TICN, TICN represents a function based on the time-series convolutional network structure, In S12, the multimodal data is AIS data I ＡＩＳ and NAVTEX message data I ＮＡＶＴＥＸ and Electronic Chart Data I ＥＣＤＩＳ and video image data I Ｉｍａｇｅ and S12 is, S121, AIS data I ＡＩＳ a sub-step of preprocessing the S122, NAVTEX message data I ＮＡＶＴＥＸ a sub-step of preprocessing the S123, Electronic Chart Data I ＥＣＤＩＳ a sub-step of preprocessing the S124, video image data I Ｉｍａｇｅ and a sub-step of preprocessing the In S121, the AIS data I ＡＩＳ includes the longitude, latitude, speed, and direction of the ship's navigation, and the four-element group I ＡＩＳ（ｋ） =(longitude （ｋ） , Latitude （ｋ） , speed （ｋ） , course （ｋ） ) is constructed as the input I of the time series network TSN. ＡＩＳ = {I ＡＩＳ（１） , I ＡＩＳ（２） ,...,I ＡＩＳ（ｋ） ,... ,I ＡＩＳ（ｔ） }, Here, I ＡＩＳ（ｋ） I ＡＩＳ represents the AIS data corresponding to time k, and t is I ＡＩＳ represents the total length of the data after preprocessing, At S122, the NAVTEX message data I ＮＡＶＴＥＸ is stored in text format, and I ＮＡＶＴＥＸ We adopt text vectorization so that I is utilized by T-CNN, according to the following formula: ＮＡＶＴＥＸ is converted into vectorized data as input to the text convolutional network T-CNN, I ＮＡＶＴＥＸ ={Word １ ,Word ２ ,...,Word ｉ ,...,Word Ｌ } where L is I ＮＡＶＴＥＸ indicates the number of words after preprocessing, and Word ｉ represents the one-hot encoding of the i-th word in the NAVTEX message, In S123, the electronic chart data I ＥＣＤＩＳ Vector data I Ｖｅｃｔｏｒ and Raster Data I Ｒａｓｔｅｒ and the input I of the time series network TSN is stored in the format Ｖｅｃｔｏｒ and the input I of the image convolutional network I-CNN Ｒａｓｔｅｒ and the formula is as follows: I Ｖｅｃｔｏｒ ={I Ｖｅｃｔｏｒ（１） ,I Ｖｅｃｔｏｒ（２） ,...,I Ｖｅｃｔｏｒ（ｋ） ,...,I Ｖｅｃｔｏｒ（ｔ） } I Ｒａｓｔｅｒ ={I Ｒａｓｔｅｒ（１） ,I Ｒａｓｔｅｒ（２） ,...,I Ｒａｓｔｅｒ（ｊ） ,...,I Ｒａｓｔｅｒ（ｍ） } Here, I Ｖｅｃｔｏｒ（ｋ） I Ｖｅｃｔｏｒ represents vector data corresponding to time k in Ｖｅｃｔｏｒ represents the total length of data after preprocessing, and I Ｒａｓｔｅｒ（ｊ） I Ｒａｓｔｅｒ represents the j-th raster data in I Ｒａｓｔｅｒ represents the number of raster images after preprocessing, In S124, the video image data I Ｉｍａｇｅ RGB data I ＲＧＢ and HOG Data I ＨＯＧ and store them in the form of I ＲＧＢ and the input I of the time-series image network TICN ＨＯＧ and the formula is as follows: I ＲＧＢ ={I ＲＧＢ（１） ,I ＲＧＢ（２） ,...,I ＲＧＢ（ｊ） ,...,I ＲＧＢ（ｍ） } I ＨＯＧ ={I ＨＯＧ（１） ,I ＨＯＧ（２） ,...,I ＨＯＧ（ｑ） ,...,I ＨＯＧ（ｎ） } Here, I ＲＧＢ（ｊ） I ＲＧＢ represents the j-th RGB image data in I ＲＧＢ represents the number of preprocessed RGB images, and I ＨＯＧ（ｑ） I ＨＯＧ represents the q-th HOG image data in ＨＯＧ represents the number of preprocessed HOG images, In S13, the preprocessed multimodal data is input to the corresponding parallel networks TSN, T-CNN, I-CNN, and TICN for training and learning; In S14, O corresponding to the parallel networks TSN, T-CNN, I-CNN and TICN based on the multiple attention mechanism MHA is ＴＳＮ , O Ｔ―ＣＮＮ , O Ｉ－ＣＮＮ and O ＴＩＣＮ Concat fusion is performed on the multimodal fusion network to obtain the final output O Ｆｉｎａｌ =Contact(MHA(O ＴＳＮ ), MHA (O Ｔ－ＣＮＮ ), MHA (O Ｉ－ＣＮＮ ), MHA (O ＴＩＣＮ )) and S2, S21: Obtain a generative pre-training model, i.e., a GPT model, through feedback training of large amounts of data; S22, outputting multiple modes; S23, displaying the results; In S21, the output O of the GPT model ＧＰＴ is expressed by the following formula: O ＧＰＴ =GPT(O Ｆｉｎａｌ )={O ＧＰＴ（ＴＳＮ） ,O ＧＰＴ（Ｔ－ＣＮＮ） ,O ＧＰＴ（Ｉ－ＣＮＮ） ,O ＧＰＴ（ＴＩＣＮ） } Here, O ＧＰＴ（ＴＳＮ） , O ＧＰＴ（Ｔ―ＣＮＮ） , O ＧＰＴ（Ｉ－ＣＮＮ） and O ＧＰＴ（ＴＩＣＮ） are O trained with the GPT model, respectively. ＴＳＮ , O Ｔ―ＣＮＮ , O Ｉ－ＣＮＮ and O ＴＩＣＮ represents the output corresponding to S22 is, S221, wake O Ｎａｖｉ a sub-step that outputs S222, Text O Ｔｅｘｔ a sub-step that outputs S223, Image O Ｉｍａｇｅ and outputting: At S221, based on the input multimodal data, a track description of the preset scene is generated according to the following formula: [Equation 1] At S222, generate a textual description of the preset scene based on the input multimodal data according to the following formula: [Equation 2] At S223, an image description of the preset scene is generated based on the input multimodal data according to the following formula: [Equation 3] In S23, providing a scene display in multiple modes for convenient reading, observation and monitoring; In S3, displaying the multi-modal output of the track, text and image obtained by multi-modal fusion makes it easier for mariners to understand the multi-modal maritime safety information and make decisions quickly; The present invention relates to an intelligent decision support method for maritime safety information based on a multimodal fusion network.

Images