Offshore multi-modal situation cognition method and device

Abstract

The invention relates to the technical field of situation cognition, in particular to a maritime multi-mode situation cognition method and device, and the method comprises the steps: training a multi-mode large model, and predicting a maritime situation through employing the trained multi-mode large model; acquiring image data and text data in the same time period; the image encoder performs feature extraction on the image data to obtain image features, and the text encoder performs feature extraction on the text data to obtain text features; and the multi-modal fusion device performs pairing processing on the image features and the text features to obtain cross-modal features, and analyzes the marine situation according to the cross-modal features. Compared with an existing multi-mode large model of a double-tower structure which only performs interaction and fusion of image features and text features in the last layer, the model structure and the feature fusion method adopted by the invention can more fully capture a deep relationship between the image features and the text features; and the fusion effect of the image features and the text features is better.

Description

Technical Field

[0001] The present invention relates to the technical field of situation awareness, and in particular to a method and device for multi-modal situation awareness at sea. Background Art

[0002] In modern society, with the increasing frequency of marine activities, the maritime security environment has become more complex and uncertain factors have gradually increased. Maritime situation awareness, that is, the grasp of the current status of the marine environment, the distribution of various types of ships and their development trends, is an important basis for ensuring the smooth progress of marine activities. At present, there are many channels for obtaining maritime situation information, the amount of information is huge and changes rapidly, and there are problems such as information fragmentation, localization and instability. Therefore, how to construct a complex mapping relationship from the current sea conditions to future trend predictions has become the key to accurately understand the changes in the maritime situation and assist decision makers such as shipping companies and marine management departments to make effective decisions.

[0003] The current mainstream multimodal large model architectures are mainly divided into single-tower and dual-tower architectures. Among them, the single-tower architecture only uses a deep neural network structure to complete the interactive fusion between images and texts, which is a pre-information fusion solution; the dual-tower architecture uses different neural networks to complete information extraction of different modalities, and then only performs information interaction and fusion in the last layer, which is a post-information fusion solution. However, the current dual-tower structure of multimodal large models has low training efficiency, and image data and text data only interact and fuse in the last layer, which may not be able to fully capture the deep relationship between modalities. For processing complex situation tasks, its reasoning speed will be affected to a certain extent.

[0004] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0005] The technical problem to be solved by the present invention is that the training efficiency of the existing large multi-modal model with a dual-tower structure is low, and the fusion effect of image data and text data and the speed of processing complex situation tasks need to be improved.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for multimodal situation awareness at sea, comprising:

[0008] Training the multimodal large model to predict the maritime situation using the trained multimodal large model; wherein the multimodal large model includes an image encoder, a text encoder and a multimodal fusion device;

[0009] Acquire image data and text data in the same time period;

[0010] The image encoder extracts features from the image data to obtain image features, and the text encoder extracts features from the text data to obtain text features;

[0011] The image encoder performs pairing processing of the image features with text features, and the text encoder performs pairing processing of the text features with image features.

[0012] The multimodal fuser fuses the paired image features and text features to obtain cross-modal features, and analyzes the maritime situation based on the cross-modal features.

[0013] Preferably, the training of the multimodal large model includes:

[0014] Acquire positive samples and negative samples; wherein the positive samples include paired image data and text data, and the negative samples include random image data and text data;

[0015] The image encoder performs feature recognition on the image data to obtain an image embedding vector; the text encoder performs text recognition on the text data to obtain a text embedding vector;

[0016] The image encoder and the text encoder use a loss function to continuously iteratively optimize the parameters of the image encoder and the parameters of the text encoder through comparative learning, so that the distance between the image embedding vector and the text embedding vector corresponding to the positive sample is reduced, and the distance between the image embedding vector and the text embedding vector corresponding to the negative sample is increased, so as to reduce the loss value of the loss function.

[0017] Preferably, the image encoder and the text encoder use a loss function to continuously iteratively optimize the parameters of the image encoder and the parameters of the text encoder by contrastive learning, so that the distance between the image embedding vector and the text embedding vector corresponding to the positive sample is reduced, and the distance between the image embedding vector and the text embedding vector corresponding to the negative sample is increased, so as to reduce the loss value of the loss function, specifically including:

[0018] A similarity calculation module is provided between the image encoder and the text encoder, and the similarity calculation module calculates the cosine similarity between the image embedding vector and the text embedding vector;

[0019] The similarity calculation module compares the difference between the cosine similarity and the expected similarity, and if there is an error, the similarity calculation module transmits the error back to the image encoder and the text encoder;

[0020] The image encoder and the text encoder update parameters of the image encoder and the text encoder according to the gradient of the loss function.

[0021] Preferably, the loss function used by the image encoder is an image-text loss function, specifically:

[0022]

[0023] The loss function used by the text encoder is a text-image loss function, specifically:

[0024]

[0025] Among them, x i refers to the image embedding vector in the i-th image and text pair, y i refers to the text embedding vector in the i-th image and text pair; x j refers to the image embedding vector in the jth image and text pair, y j refers to the text embedding vector in the jth image and text pair, N is the batch size, and σ is the temperature parameter.

[0026] Preferably, the loss function used by the multimodal fuser during training is:

[0027] L CL =αL i2t +βL t2i ;

[0028] Among them, α, β refer to the weights of different loss functions, L i2t is the image-text loss function, L i2t is the text-image loss function.

[0029] Preferably, the image encoder uses an efficient network with global pooling to obtain an image embedding vector, and the text encoder uses BERT with [CLS] token embedding to obtain a text embedding vector.

[0030] Preferably, the multimodal fusion device performs pairing processing on the image features and the text features to obtain cross-modal features, and analyzes the maritime situation according to the cross-modal features, specifically including:

[0031] Build prompt templates;

[0032] The multimodal fusion device infers the cross-modal features according to the prompt template to obtain an analysis result of the maritime situation;

[0033] The analysis results are input into the situation database.

[0034] Preferably, when the analysis result is an analysis result of a complex situation, the multimodal fuser filters out relevant data from the situation library, performs joint reasoning on the prompt template and the relevant data, and obtains the analysis result of the complex situation.

[0035] In a second aspect, the present invention provides a device for a multimodal situation awareness method at sea, which is used to implement the multimodal situation awareness method at sea described in the first aspect, and the device comprises:

[0036] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the maritime multimodal situation awareness method described in the first aspect.

[0037] In a third aspect, the present invention provides a non-volatile computer storage medium for implementing the method for multimodal situation awareness at sea described in the first aspect, comprising: the computer storage medium stores computer executable instructions, which are executed by one or more processors for the method for multimodal situation awareness at sea described in the first aspect.

[0038] Compared with the prior art, the beneficial effect of the present invention lies in: the present invention uses an image encoder and a text encoder to extract image features and text features respectively, and after pairing the image features and the text features, the paired image features and text features are clustered together, while the unpaired image features and text features are pulled apart, and finally a multimodal fuser is used to fuse the paired image features and text features to obtain cross-modal features, and the maritime situation is analyzed based on the cross-modal features. Compared with the existing multimodal large model with a dual-tower structure that only interacts and fuses image features and text features in the last layer, the model structure and feature fusion method adopted by the present invention can more fully capture the deep relationship between image features and text features, and the fusion effect of image features and text features is better.

[0039] In a preferred embodiment, the present invention trains the image encoder and the text encoder by contrastive learning using a loss function and a back-propagation algorithm. The image encoder and the text encoder update the parameters of the image encoder and the text encoder according to the gradient of the loss function, so that the image encoder and the text encoder can more accurately match the text features that match the image features, as well as the image features that better match the text features.

[0040] In the preferred solution, when analyzing complex situation tasks, by combining relevant historical data in the situation library and jointly reasoning the historical data and cross-modal features, more accurate analysis results of complex situations can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 It is a schematic diagram of the overall process of a multi-modal situation awareness method at sea provided by an embodiment of the present invention;

[0043] Figure 2 It is a specific flow chart of step S1 of a method for multi-modal situation recognition at sea provided by an embodiment of the present invention;

[0044] Figure 3 1 is a specific flow chart of step S12 of a method for multi-modal situation awareness at sea provided by an embodiment of the present invention;

[0045] Figure 4 It is a schematic diagram of a model training of a marine multimodal situation awareness method provided by an embodiment of the present invention;

[0046] Figure 5 It is a specific flow chart of step S5 of a method for multi-modal situation awareness at sea provided by an embodiment of the present invention;

[0047] Figure 6 It is a schematic diagram of the overall structure of a device for a marine multimodal situation recognition method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0049] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0050] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0051] When describing some embodiments, the expressions "coupling", "coupling" and "connection" and their derivatives may be used. For example, when describing some embodiments, the term "connection" may be used to indicate that two or more components are in direct physical or electrical contact with each other. For another example, when describing some embodiments, the term "coupling" may be used to indicate that two or more components are in direct physical or electrical contact. However, the terms "connection" or "coupling" may also refer to two or more components that are not in direct contact with each other, but still cooperate or interact with each other, such as "optical path coupling", "wireless connection", etc. The embodiments disclosed herein are not necessarily limited to the contents of the present invention.

[0052] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0053] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0054] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0055] Embodiment 1:

[0056] Embodiment 1 of the present invention provides a method for multi-modal situation recognition at sea, such as Figure 1 As shown, including:

[0057] In step S1, the multimodal large model is trained to predict the maritime situation using the trained multimodal large model.

[0058] The multimodal large model includes an image encoder, a text encoder and a multimodal fuser.

[0059] In step S2, image data and text data of the same time period are acquired.

[0060] Specifically, after acquiring the image and text, the image and text are preprocessed to generate image data and text data.

[0061] In step S3, the image encoder extracts features from the image data to obtain image features, and the text encoder extracts features from the text data to obtain text features.

[0062] Specifically, before the image encoder performs a feature extraction operation on the image data, one or more preprocessing operations of scaling, cropping, rotating, graying and normalizing are performed on the image data to generate an input image; before the text encoder performs a feature extraction operation on the text data, one or more preprocessing operations of removing punctuation marks, converting uppercase and lowercase, removing stop words, deleting spaces and special characters are performed on the text data to generate input text; the image encoder performs feature extraction on the input image to generate image features, and the text encoder performs feature extraction on the input text to generate text features.

[0063] In step S4, the image encoder performs pairing processing on the image features with text features, and the text encoder performs pairing processing on the text features with image features.

[0064] Among them, the image encoder and the text encoder process image data and text data of the same time period respectively to generate multiple image features and multiple text features, and the image encoder performs text feature pairing processing on the image features, specifically referring to that the image encoder extracts text features that match the currently processed image features from multiple text features output by the text encoder, and then outputs an image embedding vector similar to the matched text features; the text encoder performs image feature pairing processing on the text features, specifically referring to that the text encoder extracts image features that match the currently processed text features from multiple image features output by the image encoder, and then outputs a text embedding vector similar to the matched image features.

[0065] In step S5, the multimodal fuser fuses the paired image features and text features to obtain cross-modal features, and analyzes the maritime situation based on the cross-modal features.

[0066] In this embodiment, the cross-modal feature refers to a comprehensive feature representation formed by fusing image features and text features. When the multimodal fuser pairs image features and text features, it is actually looking for the correlation and complementarity between the two modal data, thereby generating a cross-modal feature that can simultaneously reflect image and text information. Cross-modal features are of great significance for complex tasks such as analyzing maritime situations. By fusing image features (such as ships at sea, waves, weather conditions, etc.) and text features (such as weather reports, navigation logs, maritime traffic information, etc.), the multimodal fuser can more comprehensively understand various aspects of the maritime environment and make more accurate situation analysis.

[0067] The embodiment of the present invention uses an image encoder and a text encoder to extract image features and text features respectively. After pairing the image features and text features, the paired image features and text features are clustered together, while the unpaired image features and text features are pulled apart. Finally, a multimodal fuser is used to fuse the paired image features and text features to obtain cross-modal features, and the maritime situation is analyzed based on the cross-modal features. Compared with the existing multimodal large model with a dual-tower structure that only interacts and fuses image features and text features in the last layer, the model structure and feature fusion method adopted by the present invention can more fully capture the deep relationship between image features and text features, and has a better fusion effect on image features and text features.

[0068] In the above step S1, if Figure 2 As shown, the training of the multimodal large model includes:

[0069] In step S10, positive samples and negative samples are obtained; wherein the positive samples include paired image data and text data, and the negative samples include random image data and text data.

[0070] Among them, in the positive samples and negative samples of this embodiment, the annotations of the image data and text data of the positive samples include text annotations, image annotations and image-text pairing annotations. Different from the traditional label-based classification target, the annotations applied to the text data in the embodiment of the present invention are "label" weights generated by the text encoder, rather than discrete labels like in ImageNet. The acquisition methods of the image data and text data in the positive samples and the negative samples include but are not limited to public data set downloads, self-owned data set expansions, simulation system simulations, keyword crawling of various search engines, etc. The image data and text data described in the embodiment of the present invention mainly include various data related to the maritime situation, such as rules and regulations, situation image sequences, etc., wherein the rules and regulations refer to the rules and regulations related to the sea, which are derived from various publicly published publications and are of text type; the situation image sequence refers to the sequence after the situation is visualized, which is derived from various simulation systems and is of image type.

[0071] After obtaining the images and texts of the positive and negative samples, the images and texts are preprocessed, and then the images and texts are paired. The image preprocessing includes screening low-resolution, low-exposure, over-exposure and simple background low-quality images in a batch of images, inputting the screened images into the visual model, and the visual model judges and removes duplicate images, and the remaining images generate image data; the text preprocessing includes filtering, deduplication, privacy removal or word segmentation of low-quality data in a batch of texts to generate text data. The low-quality data filtering refers to filtering out data with hyperlinks, templates and offensive words, filtering out texts inconsistent with the target language, etc.; deduplication is to remove duplicate data in the corpus at different granularities such as sentence level, document level and data set level; privacy removal refers to removing user-generated content containing sensitive or personal information in the corpus to reduce the risk of privacy leakage; word segmentation refers to dividing the original text into word sequences; pairing images and texts refers to matching images and text descriptions associated with them to form an image-text pairing pool, so as to better train the model.

[0072] In step S11, the image encoder performs feature recognition on the image data to obtain an image embedding vector; the text encoder performs text recognition on the text data to obtain a text embedding vector.

[0073] Specifically, the image encoder uses an efficient network with global pooling to obtain an image embedding vector, and the text encoder uses a Transformer-based bidirectional encoder representation (Bidirectional Encoder Representations from Transformers, abbreviated as BERT) with [CLS] token embedding to obtain a text embedding vector.

[0074] In step S12, the image encoder and the text encoder use a loss function to continuously iteratively optimize the parameters of the image encoder and the parameters of the text encoder by contrastive learning, so that the distance between the image embedding vector and the text embedding vector corresponding to the positive sample is reduced, and the distance between the image embedding vector and the text embedding vector corresponding to the negative sample is increased, so as to reduce the loss value of the loss function.

[0075] Specifically, in the above step S12, it is mentioned that the image encoder and the text encoder continuously iteratively optimize the parameters of the image encoder and the parameters of the text encoder by using a loss function through comparative learning, so that the distance between the image embedding vector and the text embedding vector corresponding to the positive sample is reduced, and the distance between the image embedding vector and the text embedding vector corresponding to the negative sample is increased, so as to reduce the loss value of the loss function, such as Figure 3 As shown, it specifically includes: a similarity calculation module is arranged between the image encoder and the text encoder, and the similarity calculation module calculates the cosine similarity between the image embedding vector and the text embedding vector; specifically, the loss function measures the ability of the image encoder and the text encoder to distinguish the positive sample and the negative sample according to the cosine similarity; the stronger the distinguishing ability is, the lower the loss value of the loss function is, and the weaker the distinguishing ability is, the greater the loss value of the loss function is.

[0076] In step S120, the similarity calculation module compares the difference between the cosine similarity and the expected similarity. If there is an error, the similarity calculation module transmits the error back to the image encoder and the text encoder.

[0077] In step S121, the image encoder and the text encoder update parameters of the image encoder and the text encoder according to the gradient of the loss function.

[0078] This process is iterated continuously until the loss function reaches a lower value, that is, the model can better distinguish between positive and negative samples.

[0079] In the above steps, the loss function used by the image encoder is the image-text loss function, specifically: The loss function used by the text encoder is a text-image loss function, specifically: Among them, x i refers to the image embedding vector in the i-th image and text pair, y i refers to the text embedding vector in the i-th image and text pair; x j refers to the image embedding vector in the jth image and text pair, y j refers to the text embedding vector in the jth image and text pair, N is the batch size, and σ is the temperature parameter. In the above formula, exp(x) represents an exponential function with the natural constant e as the base, that is, e to the power of x. refers to the image embedding vector x i is transposed, Refers to the text embedding vector y i is transposed.

[0080] In order to evaluate the performance of the multimodal fuser in fusing the image embedding vector output by the image encoder and the text embedding vector output by the text encoder, the loss function used by the multimodal fuser during training is: L CL =αL i2t +βL t2i ; Among them, α, β refer to the weights of different loss functions, L i2t is the image-text loss function, L i2t is the text-image loss function. By continuously optimizing the loss function and fine-tuning the model weights for different specific tasks, it can be applied to a variety of downstream tasks, such as target detection and tracking, maritime traffic monitoring, target behavior prediction, etc.

[0081] See also Figure 4 As shown, it is a structural diagram of the model training of the method described in the above steps S10 to S12. In the above preferred embodiment, the present invention trains the image encoder and the text encoder by contrastive learning using a loss function and a back-propagation algorithm. The image encoder and the text encoder update the parameters of the image encoder and the text encoder according to the gradient of the loss function, so that the image encoder and the text encoder can more accurately match the text features that match the image features, as well as the image features that better match the text features.

[0082] In the above step S5, the multimodal fuser fuses the paired image features and text features to obtain cross-modal features, and analyzes the maritime situation based on the cross-modal features, such as Figure 5 As shown, specifically including:

[0083] In step S50, a prompt template is constructed.

[0084] For simple situation tasks, such as target detection and tracking (for example, identifying and tracking specific objects in a video) or maritime traffic monitoring (monitoring the activities of ships at sea), the prompt template can be a predefined input format to guide the multimodal fuser on how to understand and process these tasks; for complex situation tasks, it is necessary to construct a more complex prompt template, which can contain multiple subtasks or analysis steps to guide the multimodal fuser on how to comprehensively process the filtered situation data.

[0085] In step S51, the multimodal fuser infers the cross-modal features according to the prompt template to obtain an analysis result of the maritime situation.

[0086] In step S52, the analysis result is input into the situation library.

[0087] Specifically, when the analysis result is an analysis result of a complex situation, the multimodal fuser filters out relevant data from the situation library, performs joint reasoning on the prompt template and the relevant data, and obtains the analysis result of the complex situation.

[0088] For simple situation tasks, the multimodal fuser is used to reason the prompt template and generate the task results. These results may include information such as the location, speed, and type of the target; for complex situation tasks, the multimodal fuser is used to jointly reason the prompt template and the filtered situation data to obtain the results of the comprehensive situation analysis, which can be a comprehensive analysis of the current situation, including potential risks, trend predictions, etc.

[0089] Embodiment 2:

[0090] Based on Example 1, the embodiment of the present invention provides a device for a method for multi-modal situation recognition at sea, which is used to implement the method for multi-modal situation recognition at sea described in Example 1, such as Figure 6 As shown, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the maritime multimodal situation awareness method described in Example 1.

[0091] like Figure 6 , which is a schematic diagram of the architecture of a device according to an embodiment of the present invention. The device according to this embodiment includes one or more processors 1 and a memory 2. Figure 6 A processor 1 is taken as an example.

[0092] The processor 1 and the memory 2 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0093] The memory 2 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the method in Example 1. The processor 1 executes the method by running the non-volatile software programs and instructions stored in the memory 2.

[0094] The memory 2 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 2 may optionally include a memory remotely arranged relative to the processor 1, and these remote memories may be connected to the processor 1 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0095] The program instructions / modules are stored in the memory 2, and when executed by the one or more processors 1, the method in the above embodiment 1 is executed, for example, the method described above is executed. Figure 1-Figure 3 ,as well as Figure 5 The steps shown.

[0096] In addition to the above-mentioned device, an embodiment of the present invention also provides a non-volatile computer storage medium for implementing the multimodal situation awareness method at sea described in the aforementioned embodiment 1, including: the computer storage medium stores computer executable instructions, which are executed by one or more processors for the multimodal situation awareness method at sea described in embodiment 1.

[0097] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0098] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for multimodal situation awareness at sea, characterized in that: include: Training the multimodal large model to predict the maritime situation using the trained multimodal large model; wherein the multimodal large model includes an image encoder, a text encoder and a multimodal fusion device; Acquire image data and text data in the same time period; The image encoder extracts features from the image data to obtain image features, and the text encoder extracts features from the text data to obtain text features; The image encoder performs a pairing process of text features on the image features, and the text encoder performs a pairing process of image features on the text features; The multimodal fuser fuses the paired image features and text features to obtain cross-modal features, and analyzes the maritime situation based on the cross-modal features.

2. The method for multimodal situation awareness at sea according to claim 1, characterized in that: The training of the multimodal large model includes: Acquire positive samples and negative samples; wherein the positive samples include paired image data and text data, and the negative samples include random image data and text data; The image encoder performs feature recognition on the image data to obtain an image embedding vector; the text encoder performs text recognition on the text data to obtain a text embedding vector; The image encoder and the text encoder use a loss function to continuously iteratively optimize the parameters of the image encoder and the parameters of the text encoder through comparative learning, so that the distance between the image embedding vector and the text embedding vector corresponding to the positive sample is reduced, and the distance between the image embedding vector and the text embedding vector corresponding to the negative sample is increased, so as to reduce the loss value of the loss function.

3. The method for multimodal situation awareness at sea according to claim 2, characterized in that: The image encoder and the text encoder continuously iteratively optimize the parameters of the image encoder and the parameters of the text encoder by using a loss function in a comparative learning manner, so that the distance between the image embedding vector and the text embedding vector corresponding to the positive sample is reduced, and the distance between the image embedding vector and the text embedding vector corresponding to the negative sample is increased, so as to reduce the loss value of the loss function, specifically including: A similarity calculation module is provided between the image encoder and the text encoder, and the similarity calculation module calculates the cosine similarity between the image embedding vector and the text embedding vector; The similarity calculation module compares the difference between the cosine similarity and the expected similarity, and if there is an error, the similarity calculation module transmits the error back to the image encoder and the text encoder; The image encoder and the text encoder update parameters of the image encoder and the text encoder according to the gradient of the loss function.

4. The method for multimodal situation awareness at sea according to claim 3, characterized in that: The loss function used by the image encoder is an image-text loss function, specifically: The loss function used by the text encoder is a text-image loss function, specifically: Among them, x i refers to the image embedding vector in the i-th image and text pair, y i refers to the text embedding vector in the i-th image and text pair; x j refers to the image embedding vector in the jth image and text pair, y j refers to the text embedding vector in the jth image and text pair, N is the batch size, and σ is the temperature parameter.

5. The method for multimodal situation awareness at sea according to claim 4, characterized in that: The training of the multimodal large model includes: The multimodal fusion device is trained, and the loss function used in the training process is: L CL =αL i2t +βL t2i ; Among them, α, β refer to the weights of different loss functions, L i2t is the image-text loss function, L i2t is the text-image loss function.

6. The method for multimodal situation awareness at sea according to claim 2, characterized in that: The image encoder uses an efficient network with global pooling to obtain an image embedding vector, and the text encoder uses BERT with [CLS] token embedding to obtain a text embedding vector.

7. The method for multimodal situation awareness at sea according to claim 1, characterized in that: The multimodal fuser fuses the paired image features and text features to obtain cross-modal features, and analyzes the maritime situation according to the cross-modal features, specifically including: Build prompt templates; The multimodal fusion device infers the cross-modal features according to the prompt template to obtain an analysis result of the maritime situation; The analysis results are input into the situation database.

8. The method for multimodal situation awareness at sea according to claim 7, characterized in that: When the analysis result is an analysis result of a complex situation, the multimodal fusion device filters out relevant data from the situation library, performs joint reasoning on the prompt template and the relevant data, and obtains the analysis result of the complex situation.

9. A device for a multi-modal situation recognition method at sea, used to implement the multi-modal situation recognition method at sea according to any one of claims 1 to 8, characterized in that: The device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the maritime multimodal situation awareness method as described in any one of claims 1-8.

10. A non-volatile computer storage medium, used to implement the marine multimodal situation awareness method according to any one of claims 1 to 8, characterized in that: include: The computer storage medium stores computer executable instructions, which are executed by one or more processors and are used in the maritime multimodal situation awareness method as described in any one of claims 1-8.

Images