Self-adaptive alignment cross-modal vision-language ship intelligent man-machine interaction method

Abstract

The embodiment of the invention discloses an adaptive alignment cross-modal vision-language ship intelligent man-machine interaction method. The method comprises the following steps: extracting target image objects semantic representation and position information corresponding to each object from a visual image acquired by a ship-borne visual sensor; extracting a text token semantic representation from a corresponding language instruction of the visual image, and calculating a text summary representation; performing in-mold alignment operation on the extracted text token semantic representation; carrying out intra-mode and inter-mode interactive alignment on the extracted image token semantic representation; performing compression and integration on the text token semantic representation information of the fine granularity, and fusing the text semantic representation and the cross-modal image objects semantic representation; and projecting the fusion features, constructing a loss function, and obtaining a target related to a language instruction and a position thereof, so that a sailor can conveniently perform real-time man-machine interaction with an intelligent identification scene in a cruising process, the intelligence and operation efficiency of a ship are improved, and the next intelligent decision can be better performed.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction of ships, and in particular to an adaptively aligned cross-modal vision-language ship intelligent human-computer interaction method. Background Art

[0002] Human-computer interaction technology has been widely used in many fields, from the early command line interface to today's graphical user interface, touch technology, and voice interaction technology. The existing human-computer interaction methods have won the favor of users for their intuitiveness and efficiency, especially in the fields of personal computing devices, smart phones, smart homes, etc. Users can interact with devices through visual elements and natural language, which greatly reduces the threshold of use. However, although the existing technology has made significant progress in operational convenience and user experience, there are still some shortcomings. On the one hand, most systems rely on a single modal interaction method and perform poorly in complex scenarios. On the other hand, the existing interactive system lacks sufficient intelligence and mainly relies on preset rules and simple pattern matching, which makes it difficult to cope with complex and changing user needs.

[0003] With the continuous development of artificial intelligence technology, multimodal human-computer interaction provides a more natural and flexible interaction method by combining information from multiple different modalities. By integrating data from multiple sensors such as cameras, microphones, and touch sensors, multimodal interaction systems have been widely used in fields such as smart assistants, augmented reality, virtual reality, and autonomous driving. Compared with traditional single-modal interaction methods, multimodal technology can more accurately capture user intentions and provide personalized services through comprehensive analysis of multimodal data. This interaction method not only improves the naturalness of the user experience, but also enhances the robustness of the system in complex environments, and can compensate through other modalities when a single channel is insufficient or fails. However, multimodal human-computer interaction technology still faces a series of challenges in terms of how to obtain fine-grained feature information and align and fuse data information from different sources.

[0004] Driven by the national marine power strategy, the demand for marine development and utilization continues to increase, prompting the rapid development of ship technology towards intelligence. By integrating modern information technology and automation technology, intelligent ships aim to achieve more efficient autonomous navigation, intelligent management and optimal resource allocation to enhance my country's competitiveness in international marine affairs. With the gradual promotion of intelligent ships, research on the intelligence of ships has become a top priority, especially in complex sea conditions and ocean navigation environments. How to improve the safety and operational efficiency of ships through intelligent means has become the key to the development of the industry. However, through in-depth research, it is found that existing intelligent ships still have certain deficiencies in human-computer interaction guided by visual-language information. There is an urgent need for a human-computer interaction system that can provide dynamic adaptive modulation and multimodal visual-language information alignment to improve the convenience of operation and provide higher robustness and safety for the next decision in complex environments. Summary of the invention

[0005] The purpose of the present invention is to solve the problems raised in the above-mentioned background technology. The present invention provides an adaptive active learning method under limited label of ship multimodal data. When the labels of massive ship data are limited, the method can strategically select data with diversity and uncertainty as sub-sample sets based on a limited labeling budget, optimize the labeling process of unlabeled data, and solve the problems of high cost and poor effect of labeling massive ship unlabeled data.

[0006] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0007] Adaptively aligned cross-modal vision-language ship intelligent human-computer interaction method, including:

[0008] S1, extracting the semantic representation of target image objects and the corresponding position information of each object from the visual images collected by the ship-borne visual sensor;

[0009] S2, extracting the semantic representation of text tokens from the corresponding language instructions of the visual image and calculating the text summary representation;

[0010] S3, performing an intra-modal alignment operation on the extracted text tokens semantic representation to obtain a fine-grained text tokens semantic representation;

[0011] S4, perform intra-modal and inter-modal interactive alignment on the extracted semantic representations of image tokens to obtain fine-grained semantic representations of image objects;

[0012] S5. Compress and integrate the fine-grained text tokens semantic representation information to obtain the text semantic representation in vector form, and fuse the text semantic representation with the cross-modal image objects semantic representation to obtain the fusion feature;

[0013] S6. Project the fused features, and build a loss function based on the classification ranking and regression strategy to obtain the targets and their positions related to the language instructions.

[0014] Beneficial effects:

[0015] By realizing cross-modal information alignment operations dominated by visual information and language information, it is convenient for crew members to conduct real-time human-computer interaction with intelligent recognition scenarios during cruising, improve the intelligence and operational efficiency of the ship, and make better intelligent decisions for the next step.

[0016] The method proposed in this invention considers the redundant information contained in the text and image content, designs an adaptive attention unit to filter out irrelevant information, and further ensures the accuracy of information during the interaction process, so that the model can accurately capture the key information in the language instructions, and focus on the image targets that match the key information of the language instructions, significantly improving the accuracy of intelligent human-computer interaction. This advantage is different from the problem of some existing cross-modal research - the failure to effectively solve the interference caused by redundant noise information.

[0017] The present invention introduces the latest technical method KAN in the process of intra-modal interaction and cross-modal alignment of vision and text, which effectively replaces the traditional multi-layer perceptron operation, enhances the robustness of the model and greatly reduces the model parameters of this part, so that it can further meet the requirements of model reasoning speed and model update time in complex marine environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is a model structure diagram of the adaptively aligned cross-modal vision-language ship intelligent human-computer interaction method proposed in the present invention.

[0020] Figure 2 This is a flow chart of the preprocessing and feature extraction of language instructions in the present invention.

[0021] Figure 3 It is a structural diagram of the text encoding module in the present invention.

[0022] Figure 4 It is a structural diagram of the image encoding module in the present invention.

[0023] Figure 5 It is the work flow chart of the present invention. DETAILED DESCRIPTION

[0024] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0025] like Figure 1 As shown, the present invention provides an adaptively aligned cross-modal vision-language ship intelligent human-computer interaction method, comprising:

[0026] S1, extracting the semantic representation of target image objects and the corresponding position information of each object from the visual images collected by the ship-borne visual sensor;

[0027] S2, extracting the semantic representation of text tokens from the corresponding language instructions of the visual image and calculating the text summary representation;

[0028] S3, performing an intra-modal alignment operation on the extracted text tokens semantic representation to obtain a fine-grained text tokens semantic representation;

[0029] S4, perform intra-modal and inter-modal interactive alignment on the extracted semantic representations of image tokens to obtain fine-grained semantic representations of image objects;

[0030] S5. Compress and integrate the fine-grained text tokens semantic representation information to obtain the text semantic representation in vector form, and fuse the text semantic representation with the cross-modal image objects semantic representation to obtain the fusion feature;

[0031] S6. Project the fused features, and build a loss function based on the classification ranking and regression strategy to obtain the targets and their positions related to the language instructions.

[0032] The specific steps of step S1 are as follows:

[0033] S11, use the Faster R-CNN network with ResNet152 as the backbone to extract the semantic representation of objects in the target image and the coordinates corresponding to each object;

[0034] S12, performing absolute positioning and relative positioning of the image object according to the coordinate information, and obtaining absolute positioning features and relative positioning features respectively through coding design;

[0035] For step S11, an image Img collected by the ship's visual sensor is used as input. Considering that the grid type and patch type features may not be very effective in locating and tracking objective objects and recognizing objects of different sizes in complex environments, and a lot of redundant information will appear, the present invention uses the Faster R-CNN network with ResNet152 as the backbone to extract the objects semantic representation of the target image Img. The coordinate position Pos corresponding to objects, so that the model can accurately capture objects of different sizes:

[0036]

[0037] in, represents the semantic features of extracted image objects, n represents the number of objects contained in the image, which is set to 100. Represents the feature dimension of the object, set to 768; Pos = [pos1, pos2, ..., pos n ,]∈R n×4 Represents the coordinate information of the corresponding objects. Each object has 4 dimensions, namely is obj i The coordinates of and Refers to obj respectively i The lower left and upper right corner coordinates;

[0038] For step S12, since the directly acquired objects coordinate information is not very conducive to the inference of the model, the present invention effectively processes and encodes the image object according to the original coordinate information, and obtains effective absolute positioning features and relative positioning features, respectively, to enhance the model's positioning recognition and reasoning capabilities for the target object. The specific implementation process is as follows:

[0039] Convert the coordinates of each object into pos i ={xpos i ,ypos i ,wpos i ,hpos i};

[0040]

[0041] Among them, {xpos i ,ypos i} means obj i The center point coordinates, wpos i and hpos i Refers to obj respectivelyi The width and height of

[0042] The absolute position relationship of each object relative to the image Img is used for scaling to enhance the model's learning ability and target positioning ability, and then the absolute coordinate feature Pos of objects abs It is expressed as:

[0043]

[0044] Among them, Img w and Img h are the width and height of the image Img respectively; Linear(·) is the linear conversion function;

[0045] The relative coordinate feature is constructed using the relative relationship between each pair of objects:

[0046]

[0047] sti,j∈[1,…,n]

[0048] in, Refers to high-dimensional embedding algorithms.

[0049] like Figure 2 As shown, the specific steps of step S2 are as follows:

[0050] S21, cleaning, word segmentation and encoding the text in the form of language instructions to obtain a text token list;

[0051] S22. Extract semantic feature representation of text token list using pre-trained language model RoBERTa;

[0052] S23, calculating and obtaining a global representation of the text context according to the semantic feature representation of the text tokens;

[0053] For step S21, for the image Img, the language instruction Que in text form is used as input. First, the text Que is cleaned to remove unnecessary noise interference, including but not limited to stop word removal, special symbol removal, and capitalization unification; secondly, the BPE word segmenter is used to segment Que; further, to ensure the uniform length of the Que sequence and facilitate training, the maximum length of the Que sequence is set to 30. If it is not enough, it is padded, and if it is too long, it is truncated; finally, the segmented Que sequence is converted into an ID sequence, and an attention mask is generated to obtain the input tensor Que ts ;

[0054] For step S22, ensure that the text Que tsAfter the input format is obtained, the pre-trained language model RoBERTa is used to extract features and obtain the semantic representation of text tokens:

[0055]

[0056] in, is the semantic feature of text tokens extracted by RoBERTa, m represents the number of tokens in the text sequence, which is set to 30. Indicates the feature dimension of the token, set to 768;

[0057] For step S23, based on the obtained semantic feature representation of text tokens, the average pooling method is used to calculate and obtain the text summary feature representation, and broadcast it to the same dimension as the semantic feature of text tokens, in order to use this feature representation to optimize the intra-module interaction and filter out irrelevant noise information in the subsequent process. The specific implementation process is as follows:

[0058]

[0059] Among them, Avgpooling (·) refers to average pooling; is the text summary representation obtained by the average pooling method; 1 m is a vector of all 1s of length m; Indicates the matrix outer product calculation method; It is the text summary feature after the broadcast.

[0060] The specific steps of step S3 are:

[0061] S31. Considering that the self-attention mechanism in the prior art methods usually only involves the correlation calculation between token pairs, and lacks the redundant information filtering operation on the original tokens sequence, the present invention designs a textual query-key filter self-attention (Textural Query-Key Filter Self-Attention, QKFSA) method to capture the long-distance dependencies of text tokens, as well as local context information and global context information, and optimize the intra-modal interaction modeling of text tokens information.

[0062] S32. Introduce Kolmogorov-Arnold Networks (KAN) to obtain more complex dependencies within the text, and realize nonlinear modeling mode and interpretability;

[0063] For step S32, in view of the fact that the feed-forward network (FFN) in the standard Transformer has problems such as high inference calculation overhead, low parameter efficiency, and easy overfitting during the model training process, the present invention introduces Kolmogorov-Arnold Networks (KAN) as an alternative to FFN to handle the complex nonlinear mapping and high-dimensional processing problems of the semantic features of text tokens, obtain more complex dependencies within the text, realize nonlinear modeling mode and interpretability, and enhance the advantages of the model in terms of representation ability, parameter efficiency, and training complexity.

[0064] Semantic features of text tokens Taking input as an example, the implementation process is as follows:

[0065]

[0066] in, It is the language feature of text tokens after KAN processing.

[0067] S33, design a text encoding unit based on text query-key filtered self-attention (QKFSA) and Kolmogorov-Arnold network (KAN), such as Figure 3 As shown in the figure, taking the encoding part of Transformer as a reference, multiple layers of cascaded text encoding units are stacked to build a text encoding module, which can achieve deep redundant information filtering and interaction modeling to obtain fine-grained semantic representation of text tokens.

[0068] S34. Based on the global representation of the text context obtained by each layer of text encoding units, xLSTM is introduced to obtain the global summary representation of the text context, which will be used as the filtering benchmark for image objects in the subsequent process.

[0069] The implementation process of the self-attention method for text query-key filtering in step S31 is as follows:

[0070] Semantic features of text tokens As input, we get the tokens Query matrix of the same dimension. tokens Key matrix and the tokens Value matrix Summarize semantic features with text As input, we get the summary Query matrix of the same dimension. And summarize the Key matrix

[0071]

[0072] Among them, Linear 3 (·) and Linear 2 (·) refers to performing three and two different linear transformation calculations on the contents in brackets respectively;

[0073] Query matrix for original tokens and Key matrix The feature information in the query matrix is ​​summarized using and Key matrix Perform filter modulation and build a gate control unit based on this Filter and sort tokens according to the importance of the entire text Que to obtain the optimized tokens Query matrix With Key Matrix

[0074]

[0075] Computation using multi-head attention units and The relevance score and the tokens Value matrix Perform operations to obtain the semantic features of text tokens after the interaction in the modal

[0076]

[0077] Among them, Linear i (·) represents the linear transformation layer of the i-th head; d scale Indicates scale scaling and defines i∈[1,2,…,8], which means that the multi-head attention unit is set to 8 heads.

[0078] Since each text encoding unit layer is based on the semantic features of the output text tokens of this layer Get text summary feature representation Considering that the text summary features of different layers have their own emphasis on information presentation, step S34 of the present invention introduces an xLSTM network to effectively capture and integrate the complex relationship between the high-level and low-level text summary features, and generate a jointly represented text summary feature. Aims to achieve efficient complementarity of key information to serve as a filtering benchmark for image objects in subsequent processes:

[0079]

[0080] in, is the joint matrix representation of all layers of text summary features; ht is the hidden state variable generated by xLSTM at time step t.

[0081] The specific implementation process of step S4 includes:

[0082] S41. Design a Visual Optimized Self-Attention (VOSA) method for fine-grained optimization of semantic features of image objects extracted by Faster R-CNN. In the process, the text summary features of the joint representation are used. And the absolute positioning feature Pos abs Relative positioning feature Pos rel , accurately and efficiently model the dependency relationship within image objects. The specific implementation process is as follows:

[0083] Input image objects semantic features Get the objects Query matrix of the same dimension respectively objectsKey Matrix objects Value matrix

[0084] Utilizing the overall text features of joint representation Design the gated filter θ for and Information screening is performed to encourage the model to give more weight to key objects:

[0085]

[0086] Fusion of absolute positioning feature Pos abs , the objects Query matrix Q after information screening θ and Key matrix K θ Perform secondary information screening and modulation to make the model more focused on the positioning and selection of key objects:

[0087]

[0088] Where dot(·) represents the dot product calculation of the matrix vector;

[0089] Based on the improved multi-head attention unit, the relative positioning feature Pos of image objects is used rel As prior information, obtain the fine-grained feature representation of the image objects after intra-model interaction:

[0090]

[0091] S42. In order to obtain the correlation between image objects and language instruction tokens, Visual-Textual Alignment Attention (VTAA) is designed to learn the alignment relationship between text tokens and image objects, and further refine the semantic features of image objects. The specific implementation process is as follows:

[0092] Semantic features of image objects based on intra-model interaction Get the Query matrix Based on fine-grained semantic features of text tokens Get the Key matrix respectively and Value matrix

[0093]

[0094] Optimizing image objects features using absolute positioning features

[0095]

[0096] Calculate the correlation score between the image objects and the text tokens pairs, and based on this, obtain the semantic features of the image objects after alignment interaction.

[0097]

[0098] S43. Thanks to KAN's powerful function approximation capability and its ability to optimize model performance and computational efficiency, the present invention uses KAN to process complex nonlinear mapping and high-dimensional processing problems of semantic features of image objects, aiming to capture the complex dependencies of image objects in more depth:

[0099]

[0100] S44, such as Figure 4 Based on the Transformer decoding architecture, we design an image encoding unit and further construct an image encoding module to achieve in-depth redundant information filtering and interactive alignment between image objects and text tokens to obtain fine-grained semantic representation of image objects.

[0101] The specific implementation process of step S5 includes:

[0102] S51. Based on KAN, an attention compression unit is designed to integrate the semantic feature information of text tokens after optimized interaction modeling to obtain text semantic features in vector form. Assume that the output text tokens semantic features after the text encoding module are expressed as Considering the inconsistent contribution of each text token, in order to focus on the key tokens with higher scores, we first use KAN to calculate the attention weights of the text tokens:

[0103]

[0104] Secondly, the weighted summation strategy is used to calculate the text semantic features in vector form. It is worth noting that this feature is very representative after multiple layers of screening and filtering:

[0105]

[0106] S52, further, the text semantic features in vector form Expanded to semantic features of image objects At the same time, in order to remove the difference between text semantic features and image objects features and enhance the correlation, the present invention adopts a method based on a gating mechanism to fuse the two to obtain a fused feature

[0107]

[0108] Among them, 1 n is a vector of all 1s of length n, and ρ is the designed gating mechanism.

[0109] The specific implementation process of step S6 includes:

[0110] S61: Use linear transformation operation to fusion features Projection as a vector And a 4-dimensional matrix The former is used to rank and classify all image objects, while the latter is used to regress the coordinates of all image objects:

[0111]

[0112] S62: First, obtain the true scores of n image objects based on the IoU scores between the region box of the only real image object and the bounding boxes of all objects And based on the ranking score vector C score Constructing KL divergence loss Aims to measure the KL divergence between the predicted scores and the true scores of n image objects:

[0113]

[0114] Finally, the total target loss function is set based on the classification and regression strategy:

[0115]

[0116] Wherein, ξ is a balance parameter, which is set to 0.5 in this embodiment. The model parameters are updated by calculating the loss and implementing back propagation, and finally the object target and its coordinate position related to the language instruction are obtained.

[0117] Of course, the present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in the previous example of the present invention can be implemented.

[0118] Since the computer-readable storage medium introduced in the second embodiment of the present invention is a computer-readable storage medium used to implement the cross-modal visual language-ship intelligent human-computer interaction method based on adaptive alignment in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the computer-readable storage medium, so it is not repeated here. All computer-readable storage media used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0119] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the method described in Embodiment 1 when executing the program.

[0120] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. Adaptive alignment cross-modal vision-language ship intelligent human-computer interaction method, characterized in that: include: S1, extracting the semantic representation of target image objects and the corresponding position information of each object from the visual images collected by the ship-borne visual sensor; S2, extracting the semantic representation of text tokens from the corresponding language instructions of the visual image and calculating the text summary representation; S3, performing an intra-modal alignment operation on the extracted text tokens semantic representation to obtain a fine-grained text tokens semantic representation; S4, perform intra-modal and inter-modal interactive alignment on the extracted semantic representations of image tokens to obtain fine-grained semantic representations of image objects; S5. Compress and integrate the fine-grained text tokens semantic representation information to obtain the text semantic representation in vector form, and fuse the text semantic representation with the cross-modal image objects semantic representation to obtain the fusion feature; S6. Project the fused features, and build a loss function based on the classification ranking and regression strategy to obtain the targets and their positions related to the language instructions.

2. The adaptively aligned cross-modal vision-language ship intelligent human-computer interaction method according to claim 1 is characterized in that: The specific steps of step S1 are as follows: S11, use the Faster R-CNN network with ResNet152 as the backbone to extract the semantic representation of objects in the target image and the coordinates corresponding to each object; S12, performing absolute positioning and relative positioning of the image object according to the coordinate information, and obtaining absolute positioning features and relative positioning features respectively through coding design; For step S11, the Faster R-CNN network with ResNet152 as the backbone extracts the objects semantic representation of the target image Img. The coordinate position Pos corresponding to objects: in, represents the semantic features of extracted image objects, n represents the number of objects contained in the image, which is set to 100. Represents the feature dimension of the object, set to 768; Pos = [pos1, pos2, ..., pos n ,]∈R n×4 Represents the coordinate information of the corresponding objects. Each object has 4 dimensions, namely is obj i The coordinates of and Refers to obj respectively i The lower left and upper right corner coordinates; For step S12, the image object is effectively processed and encoded according to the original coordinate information, and effective absolute positioning features and relative positioning features are obtained respectively, so as to enhance the model's positioning recognition and reasoning capabilities for the target object. The specific implementation process is as follows: Convert the coordinates of each object into pos i ={xpos i ,ypos i ,wpos i ,hpos i }; Among them, {xpos i ,ypos i } means obj i The center point coordinates, wpos i and hpos i Refers to obj respectively i The width and height of The absolute position relationship of each object relative to the image Img is used for scaling to enhance the model's learning ability and target positioning ability, and then the absolute coordinate feature Pos of objects abs It is expressed as: Among them, Img w and Img h are the width and height of the image Img respectively; Linear(·) is the linear conversion function; The relative coordinate feature is constructed using the relative relationship between each pair of objects: sti,j∈[1,…,n] in, Refers to high-dimensional embedding algorithms.

3. The adaptively aligned cross-modal vision-language ship intelligent human-computer interaction method according to claim 1 is characterized in that: The specific steps of step S2 are as follows: S21, cleaning, word segmentation and encoding the text in the form of language instructions to obtain a text token list; S22. Extract semantic feature representation of text token list using pre-trained language model RoBERTa; S23, calculating and obtaining a global representation of the text context according to the semantic feature representation of the text tokens; For step S21, for the image Img, the language instruction Que in text form is used as input, and the text Que is cleaned to remove noise interference; the BPE word segmenter is used to segment Que; the maximum length of the Que sequence is set to 30, if it is less than that, it is padded, and if it is too long, it is truncated; the segmented Que sequence is converted into an ID sequence, and an attention mask is generated to obtain the input tensor Que ts ; For step S22, ensure that the text Que ts After the input format is obtained, the pre-trained language model RoBERTa is used to extract features and obtain the semantic representation of text tokens: in, is the semantic feature of text tokens extracted by RoBERTa, m represents the number of tokens in the text sequence, which is set to 30. Indicates the feature dimension of the token, set to 768; For step S23, according to the obtained text tokens semantic feature representation, the text summary feature representation is calculated by using the average pooling method, and broadcasted to the same dimension as the text tokens semantic feature. The specific implementation process is as follows: Among them, Avgpooling (·) refers to average pooling; is the text summary representation obtained by the average pooling method; 1 m is a vector of all 1s of length m; Indicates the matrix outer product calculation method; It is the text summary feature after the broadcast.

4. The adaptively aligned cross-modal vision-language ship intelligent human-computer interaction method according to claim 1 is characterized in that: The specific steps of step S3 are: S31. Design a self-attention method for text query-key filtering to capture the long-range dependencies of text tokens, as well as local and global context information, to achieve intra-modal interaction modeling. S32, introduce Kolmogorov-Arnold network to obtain more complex dependencies within the text, realize nonlinear modeling mode and interpretability; S33. Design a text encoding unit based on the self-attention of text query-key filtering and the Kolmogorov-Arnold network. Take the encoding part of Transformer as a reference, cascade and superimpose text encoding units in multiple layers to build a text encoding module, realize deep redundant information filtering and interaction modeling, and obtain fine-grained semantic representation of text tokens. S34. Based on the global representation of the text context obtained by each layer of text encoding units, xLSTM is introduced to obtain the global summary representation of the text context, which will be used as the filtering benchmark for image objects in the subsequent process.

5. The adaptively aligned cross-modal vision-language ship intelligent human-computer interaction method according to claim 4 is characterized in that: The implementation process of the self-attention method for text query-key filtering in step S31 is as follows: Semantic features of text tokens As input, we get the tokens Query matrix of the same dimension. tokensKey Matrix and the tokens Value matrix Summarize semantic features with text As input, we get the summary Query matrix of the same dimension. And summarize the Key matrix Among them, Linear 3 (·) and Linear 2 (·) refers to performing three and two different linear transformation calculations on the contents in brackets respectively; Query matrix for original tokens and Key matrix The feature information in the query matrix is ​​summarized using and Key matrix Perform filter modulation and build a gate control unit based on this Filter and sort tokens according to the importance of the entire text Que to obtain the optimized tokens Query matrix With Key Matrix Computation using multi-head attention units and The relevance score and the tokens Value matrix Perform operations to obtain the semantic features of text tokens after the interaction in the modal Among them, Linear i (·) represents the linear transformation layer of the i-th head; d scale Indicates scale scaling and defines i∈[1,2,…,8], which means that the multi-head attention unit is set to 8 heads.

6. The adaptively aligned cross-modal vision-language ship intelligent human-computer interaction method according to claim 4 is characterized in that: Step S34 introduces the xLSTM network to capture and integrate the complex relationship between high-level and low-level text summary features, and generates a jointly represented text summary feature. in, is the joint matrix representation of all layers of text summary features; h t is the hidden state variable generated by xLSTM at time step t.

7. The adaptively aligned cross-modal vision-language ship intelligent human-computer interaction method according to claim 6 is characterized in that: The specific implementation process of step S4 includes: S41. Design a visually optimized self-attention method for fine-grained optimization based on the semantic features of image objects extracted by Faster R-CNN, and use the text summary features of the joint representation And the absolute positioning feature Pos abs Relative positioning feature Pos rel , modeling the dependency relationship within image objects. The specific implementation process is as follows: Input image objects semantic features Get the objects Query matrix of the same dimension respectively objects Key matrix objects Value matrix Utilizing the overall text features of joint representation Design the gated filter θ for and Filter information: Fusion of absolute positioning features Pos abs , the objects Query matrix Q after information screening θ and Key matrix K θ Perform secondary information screening and modulation: Where dot(·) represents the dot product calculation of the matrix vector; Based on the improved multi-head attention unit, the relative positioning feature Pos of image objects is used rel As prior information, obtain the fine-grained feature representation of the image objects after intra-model interaction: S42. In order to obtain the correlation between image objects and language instruction tokens, a visual-text alignment attention is designed to learn the alignment relationship between text tokens and image objects and refine the semantic features of image objects. The specific implementation process is as follows: Semantic features of image objects based on intra-model interaction Get the Query matrix Based on fine-grained semantic features of text tokens Get the Key matrix respectively and Value matrix Optimizing image objects features using absolute positioning features Calculate the correlation score between the image objects and the text tokens pairs, and based on this, obtain the semantic features of the image objects after alignment interaction. S43. Using KAN to process complex nonlinear mapping and high-dimensional processing of semantic features of image objects: S44. Based on the Transformer decoding architecture, we design an image encoding unit and further construct an image encoding module to implement in-depth redundant information filtering and interactive alignment between image objects and text tokens to obtain fine-grained semantic representation of image objects.

8. The adaptively aligned cross-modal vision-language ship intelligent human-computer interaction method according to claim 7 is characterized in that: The specific implementation process of step S5 includes: S51. Integrate the semantic feature information of text tokens after optimized interaction modeling to obtain text semantic features in vector form. Assume that the output text tokens semantic features after the text encoding module are expressed as Use KAN to calculate the attention weights of text tokens: Secondly, the weighted summation strategy is used to calculate the text semantic features in vector form: S52, further, the text semantic features in vector form Expanded to semantic features of image objects The same dimension. A gating mechanism-based method is used to fuse the two to obtain the fusion feature Among them, 1 n is a vector of all 1s of length n, and ρ is the designed gating mechanism.

9. The adaptively aligned cross-modal vision-language ship intelligent human-computer interaction method according to claim 8, characterized in that: The specific implementation process of step S6 includes: S61: Use linear transformation operation to fusion features Projection as a vector And a 4-dimensional matrix The former is used to rank and classify all image objects, while the latter is used to regress the coordinates of all image objects: S62: First, obtain the true scores of n image objects based on the IoU scores between the region box of the only real image object and the bounding boxes of all objects And based on the ranking score vector C score Constructing KL divergence loss Aims to measure the KL divergence between the predicted scores and the true scores of n image objects: Finally, the total target loss function is set based on the classification and regression strategy: Wherein, ξ is a balance parameter, which is set to 0.5 in this embodiment. The model parameters are updated by calculating the loss and implementing back propagation, and finally the object target and its coordinate position related to the language instruction are obtained.

Images