Sonar image visual question answering method and system

Abstract

This invention belongs to the field of image visual question answering technology, and relates to a sonar image visual question answering method and system. The method includes constructing and training a sonar image visual question answering model, inputting the sonar image to be identified and the question into the model to obtain the answer; the model includes a linear projection module for processing the sonar image to obtain visual embeddings, a word embedding module for processing the question to obtain text embeddings, a Vision Transformer encoder module for processing the co-embedding of visual and text embeddings to generate cross-modal features, a contrast enhancement module for processing visual embeddings to generate enhanced visual features, a feature inter-referencing fusion module for processing cross-modal features and enhanced visual features to generate fused features, and a classifier for predicting the answer based on the fused features. This invention can deeply mine and fully identify the unique information of sonar images, providing pioneering ideas and methods for sonar image visual question answering tasks.

Description

Technical Field

[0001] This invention belongs to the field of image visual question answering technology, specifically relating to a sonar image visual question answering method and system. Background Technology

[0002] Currently, some progress has been made in visual question answering tasks for natural and remotely sensed images, but there is no research specifically on visual question answering for sonar images. Furthermore, visual question answering methods for natural images cannot be directly applied to visual question answering tasks for sonar images because sonar images and natural images have significant differences in visual characteristics, as explained below:

[0003] (1) In natural images, targets are usually prominent while the background is relatively simple; in sonar images, the background area often occupies a large proportion, and the targets are small, irregular in shape, and sparsely distributed. This characteristic of "large background and small target" makes it difficult to establish a clear cross-modal semantic relationship between targets in sonar images and textual questions. Moreover, unlike natural image visual question answering tasks that focus on target color or human expression, sonar image visual question answering tasks focus more on the presence, type, and quantity of targets. If existing natural image visual question answering methods are directly transferred, the target features are easily submerged by the cluttered background, making it difficult to effectively establish cross-modal relationships.

[0004] (2) When exploring the cross-modal relationship between visual and textual features, existing natural image visual question answering methods usually rely on cross-attention mechanisms for multimodal feature fusion and treat visual and textual features as equally important. However, in sonar images, visual features contain far more information than text and can provide more discriminative and interpretable semantics. Therefore, relying solely on cross-attention for fusion lacks a focus on modeling visual features and makes it difficult to fully explore the unique information of sonar images. Summary of the Invention

[0005] To address the shortcomings of related technologies, this invention provides a sonar image visual question answering method and system, aiming to deeply explore and fully identify the unique information of sonar images, thereby providing pioneering ideas and methods for sonar image visual question answering tasks.

[0006] This invention provides a sonar image visual question answering method, comprising the following steps:

[0007] S1. Construct a sonar image visual question answering model; the sonar image visual question answering model includes a linear projection module, a word embedding module, a Vision Transformer encoder module, a contrast enhancement module, a feature inter-feedback fusion module, and a classifier; the linear projection module is used to process the sonar image to obtain visual embeddings; the word embedding module is used to process the question to obtain text embeddings; the Vision Transformer encoder module is used to process the co-embedding after concatenating the visual embeddings and text embeddings to generate cross-modal features; the contrast enhancement module is used to process the visual embeddings to generate enhanced visual features; the feature inter-feedback fusion module is used to process the cross-modal features and enhanced visual features through self-attention and guided attention mechanisms and interactive dynamic fusion to generate fused features; the classifier is used to predict the answer based on the fused features;

[0008] S2. Train a visual question-answering model for sonar images;

[0009] S3. Input the sonar image to be identified and the question into the trained sonar image visual question answering model to obtain the answer.

[0010] In some embodiments, in step S2, the training process of the sonar image visual question-answering model includes using multiple sets of sample data consisting of sonar images, questions, and answers as the training set:

[0011] S21. Using a linear projection module to process sonar images The process involves extracting visual features using a lightweight linear projection method to obtain the visual embedding. Using the word embedding module to address the problem The text is processed using a pre-trained BERT word segmenter to extract text features to obtain text embeddings. Embedding visuals and text embedding splicing into co-embedded ;in, This refers to the group number of the sample data. ;Using the first VisionTransformer encoder in the VisionTransformer encoder module for co-embedding Processing is performed to generate cross-modal features. ;

[0012] S22. Utilize the second Vision Transformer encoder and deep convolutional block in the contrast enhancement module to perform visual embedding. Processing is performed to produce enhanced visual features. ;

[0013] S23. Utilize the feature cross-modal feature fusion module to perform cross-modal feature analysis. and enhanced visual features The system employs self-attention and guided attention mechanisms, and then dynamically fuses these mechanisms to generate fused features. ;

[0014] S24, Integrating Features The input is fed into a classifier to predict the answer;

[0015] S25. Based on the comparison between the predicted answer and the answer in the training set sample data, adjust the sonar image visual question answering model and continue training until the training result meets the preset requirements.

[0016] In some embodiments, in step S21, visual embedding Obtaining it involves the following steps:

[0017] Sonar images Flatten and slice Image blocks, , and These represent the height, width, and number of channels of a sonar image, respectively. The height, width, and number of channels of an image patch are represented by... , and This indicates the total number of image patches. Calculate according to formula (1);

[0018] (1);

[0019] Each image patch is mapped to a matrix. Dimensional visual markers ,in, Indicates the first Visual markers corresponding to each image patch; image patch embedding Represented as equation (2);

[0020] (2);

[0021] Then classify and label the images. With image patch embedding Perform concatenation and introduce a position embedding matrix. arrive In the end, visual embedding Defined as Equation (3);

[0022] (3).

[0023] In some embodiments, in step S21, text embedding Obtaining it involves the following steps:

[0024] Assuming it includes The problem of one word for Each word Transform using the embedding matrix Dimensional text markup Word embedding Represented as equation (4);

[0025] (4);

[0026] Then, use fixed text classification tags. Add to word embeddings In this context, a position embedding matrix is ​​introduced. arrive In the end, text embedding Defined as Equation (5);

[0027] (5).

[0028] In some embodiments, in step S21, the visual embedding is performed according to equation (6). and text embedding splicing into co-embedded ;

[0029] (6).

[0030] In some embodiments, in step S21, the first Vision Transformer encoder includes an MSA layer and an MLP layer, wherein the MSA layer includes... Parallel self-attention heads; cross-modal features The generation includes the following steps:

[0031] The first A self-attention head Represented as equation (7), where, , and These are the query matrix, key matrix, and value matrix, respectively. It is the transpose symbol. The feature dimensions of the key matrix and the query matrix, Indicates input data, It is the Softmax function;

[0032] (7);

[0033] The MSA layer is represented by equation (8), where, , The output of the normalized layer is represented. The normalization operation of the representation layer, This indicates a splicing operation. Indicates the output mapping matrix;

[0034] (8);

[0035] Ultimately, cross-modal features According to equation (9), where, This represents the output of the first VisionTransformer encoder;

[0036] (9).

[0037] In some embodiments, in step S22, the enhanced visual features The generation includes the following steps:

[0038] The second Vision Transformer encoder performs visual embedding according to equation (10). Processing is carried out, among which, The output of the normalized layer is represented. The normalization operation of the representation layer, This represents the output of the MSA layer. This represents the output of the second Vision Transformer encoder;

[0039] (10);

[0040] The deep convolutional blocks capture local structural features of the sonar image according to Equation (11), making the target in the sonar image a salient region and the broad background a non-salient region; where, Indicates to Perform a convolution operation. Indicates the kernel size. This represents the output of a depthwise convolution;

[0041] (11);

[0042] The depthwise convolutional block further enhances the salient regions according to equation (12), where, and This represents the convolution output of different convolution kernels;

[0043] (12);

[0044] The depthwise convolutional block uses Gaussian error linear units to suppress insignificant regions, as expressed in equation (13), where, express The output;

[0045] (13);

[0046] Then, to and Element-wise multiplication is performed to obtain fused features, which are then convolved to ultimately enhance the visual features. Obtained according to formula (14);

[0047] (14).

[0048] In some embodiments, in step S23, the features are fused. The generation includes the following steps:

[0049] Using a self-attention mechanism for cross-modal features and enhanced visual features The processing is expressed as equation (15), where, and They represent self-attention. The output; implemented using a guided attention mechanism. and Mutual guidance and mutual guidance characteristics between them and Represented as equation (16), where, Representation of features In features Guided mutual attention algorithm;

[0050] (15);

[0051] (16);

[0052] Then, based on equation (17), the mutual conduction characteristics are analyzed. and To interact, among which, This indicates element-wise multiplication. and These represent the interaction results between two mutually guided features;

[0053] (17);

[0054] Then through the dynamic weight matrix and To achieve dynamic feature fusion, it is expressed as equation (18), where, and These are features and The initial weight matrix is ​​generated through a fully connected layer. Represents the Sigmoid function;

[0055] (18);

[0056] Ultimately, fusion features Obtained according to formula (19);

[0057] (19).

[0058] In some embodiments, the trained sonar image visual question answering model needs to be verified and tested before performing step S3;

[0059] The verification process includes using multiple sets of sample data consisting of sonar images, questions, and answers as the verification set. The sample data in the verification set does not overlap with the sample data in the training set. Using the verification set, the trained sonar image visual question answering model is verified according to steps S21 to S24. Based on the comparison between the predicted answer and the answer in the verification set sample data, the optimal sonar image visual question answering model is selected.

[0060] The testing process includes using multiple sets of sample data consisting of sonar images, questions, and answers as the test set. The sample data in the test set does not overlap with the sample data in the training set and validation set. Using the test set, the validated sonar image visual question answering model is tested according to steps S21 to S24. The performance of the sonar image visual question answering model is evaluated based on the comparison between the predicted answer and the answer in the test set sample data, thereby obtaining the final sonar image visual question answering model.

[0061] The present invention also provides a sonar image visual question answering system, comprising:

[0062] The information acquisition module is used to acquire sonar images and information.

[0063] The model building module is used to build a sonar image visual question answering model with sonar images and questions as input and answers as output.

[0064] The model training module is used to train the constructed sonar image visual question answering model using multiple sets of sample data consisting of sonar images, questions, and answers as the training set.

[0065] The prediction module is used to input sonar images and questions into a sonar image visual question answering model to obtain answers.

[0066] Based on the above technical solutions, the sonar image visual question answering method and system in the embodiments of the present invention solves the problem that small targets in sonar images are easily affected by large background interference and it is difficult to effectively establish cross-modal relationships. It enhances the visual features of small targets and realizes the interactive dynamic fusion of enhanced visual features and cross-modal features, thereby enabling in-depth mining and full identification of information unique to sonar images, providing pioneering ideas and methods for visual question answering tasks of sonar images. Attached Figure Description

[0067] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0068] Figure 1 This is a diagram illustrating the overall architecture of the present invention;

[0069] Figure 2 This is an architecture diagram of the Vision Transformer encoder module in this invention;

[0070] Figure 3 This is an architecture diagram of the contrast enhancement module in this invention;

[0071] Figure 4 This is an architecture diagram of the feature cross-referencing and fusion module in this invention;

[0072] Figure 5 This is an architecture diagram of the self-attention mechanism in the feature mutual guidance and fusion module of this invention;

[0073] Figure 6 This is an architecture diagram of the guided attention mechanism in the feature mutual guidance and fusion module of this invention;

[0074] Figure 7 This is an example diagram illustrating the use of the present invention for visual question answering of sonar images. Detailed Implementation

[0075] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0076] In the description of this invention, it should be understood that the terms "center", "lateral", "longitudinal", "upper", "lower", "top", "bottom", "inner", "outer", "left", "right", "front", "rear", "vertical", "horizontal", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0077] The terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature.

[0078] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0079] refer to Figures 1-6 As shown, the present invention also provides a sonar image visual question answering method for performing visual question answering tasks on sonar images. This sonar image visual question answering method includes the following steps S1 to S3.

[0080] Step S1: Construct a sonar image visual question answering model. The sonar image visual question answering model includes a linear projection module, a word embedding module, a Vision Transformer encoder module, a Contrast enhancement module (CEM), a Mutual guidance feature fusion module (MGFF), and a classifier. The linear projection module processes the sonar image input to the sonar image visual question answering model, using a lightweight linear projection method to extract visual features to obtain visual embeddings. The word embedding module processes the question input to the sonar image visual question answering model, using a pre-trained BERT word segmenter to extract text features and generate word embeddings to obtain text embeddings. The Vision Transformer encoder module explores the cross-modal semantic relationship between the visual features of the sonar image and the textual features of the question, achieving cross-modal feature modeling; the Vision Transformer encoder module includes a first Vision Transformer encoder, used to process the co-embedding after concatenating the visual embedding and the textual embedding to generate cross-modal features. The contrast enhancement module includes a second Vision Transformer encoder and a deep convolutional block, which are used to process the visual embedding, enrich the information therein, and enhance the visual features of small targets to generate enhanced visual features; the feature intermodal fusion module is used to process cross-modal features and enhanced visual features using self-attention and guided attention mechanisms, and to perform interactive dynamic fusion to generate fused features; the classifier is used to predict the answer based on the fused features.

[0081] Step S2: Train the sonar image visual question answering model.

[0082] Step S3: Input the sonar image to be identified and the question into the trained sonar image visual question answering model to obtain the answer to the question.

[0083] The above illustrative embodiments propose a novel visual question answering method for sonar images that enhances visual-textual features. This method addresses the challenge of establishing effective cross-modal relationships for small targets in sonar images due to the susceptibility of small targets to interference from large backgrounds. By setting and jointly applying the Vision Transformer encoder module, contrast enhancement module, and feature fusion module in the sonar image visual question answering model, it can generate cross-modal features, enhanced visual features, and their interactive dynamic fusion features. This enables in-depth mining and full identification of information unique to sonar images, providing pioneering ideas and methods for visual question answering tasks in sonar images.

[0084] refer to Figures 1-6As shown, in some embodiments, in step S2, multiple sets of sample data consisting of sonar images, questions, and answers are used as a training set to train the sonar image visual question-answering model. The training process includes first inputting the sonar images and corresponding questions in the training set into the linear projection module and word embedding module of the sonar image visual question-answering model, respectively, and then performing the following steps S21 to S25.

[0085] Step S21: Use the linear projection module to process the sonar image. The process involves extracting visual features using a lightweight linear projection method to obtain the visual embedding. Using the word embedding module to address the problem The text is processed by using a pre-trained BERT word segmenter to extract text features and generate word embeddings to obtain the text embeddings. Embedding visuals and text embedding splicing into co-embedded ;in, This refers to the group number of the sample data. ; Co-embedded The input is fed into the VisionTransformer encoder module, and the first VisionTransformer encoder in the VisionTransformer encoder module is used to perform co-embedding. Processing is performed to generate cross-modal features. .

[0086] To further explain, visual embedding Obtaining it involves the following steps:

[0087] Sonar images Flatten and slice Image blocks, , and These represent the height, width, and number of channels of a sonar image, respectively. The height, width, and number of channels of an image patch are represented by... , and express, Total number of image blocks Calculate according to formula (1);

[0088] (1);

[0089] The linear projection module uses a matrix to map each image patch to... Dimensional visual markers ,in, Indicates the first Visual markers corresponding to each image patch; image patch embedding Represented as equation (2);

[0090] (2);

[0091] Then, the images are categorized and labeled. With image patch embedding Perform concatenation and introduce a position embedding matrix. arrive In the end, visual embedding Defined as Equation (3);

[0092] (3).

[0093] To further explain, the text embedding process for this problem is similar to the visual embedding process for sonar images; text embedding Obtaining it involves the following steps:

[0094] Assuming it includes The problem of one word for Each word Transform using the embedding matrix Dimensional text markup Word embedding Represented as equation (4);

[0095] (4);

[0096] Then, the fixed text classification tags are... Add to word embeddings It should be noted that fixed classification labels are introduced when processing sonar images and question text, following the design of the pre-trained ViLT model. Each of the sonar image and text modalities contains a classification label, providing a foundation for the subsequent Vision Transformer to model the cross-modal relationship between the sonar image and text modalities. This design helps to aggregate information from both modalities, ultimately improving the performance of the method. Furthermore, to avoid the potential impact of the same words appearing in different positions within the question sentence on text features, a positional embedding matrix is ​​introduced. arrive In the end, text embedding Defined as Equation (5);

[0097] (5).

[0098] To further explain, visual embedding is performed according to equation (6). and text embedding splicing into co-embedded ;

[0099] (6).

[0100] For further explanation, please refer to Figure 2 As shown, the first Vision Transformer encoder includes an MSA layer (multi-head self-attention layer), an MLP layer (multilayer perceptron layer), and two layer normalization layers; the MSA layer includes Several parallel self-attention heads, each independently using the query matrix. Key matrix Sum matrix Calculate attention weights. Cross-modal features. The generation includes the following steps:

[0101] The first A self-attention head Represented as equation (7), where, , and These are the query matrix, key matrix, and value matrix, respectively. It is the transpose symbol. The feature dimensions of the key matrix and the query matrix, Indicates input data, It is the Softmax function;

[0102] (7);

[0103] The MSA layer is represented by equation (8), where, , The output of the normalized layer is represented. The normalization operation of the representation layer, This indicates a splicing operation. Indicates the output mapping matrix;

[0104] (8);

[0105] Ultimately, cross-modal features According to equation (9), where, This represents the output of the first VisionTransformer encoder;

[0106] (9).

[0107] Step S22: Utilize the second Vision Transformer encoder and depth convolution block in the contrast enhancement module to perform visual embedding. This process is used to analyze deep relationships within visual features, generating enhanced visual features. It should be noted that the Vision Transformer encoder typically uses an attention mechanism to process all extracted visual embeddings; however, for sonar images, relying solely on the Vision Transformer encoder may not effectively capture specific visual details in the sonar images, especially for small targets, which may lead to the loss of visual features. Therefore, to address this issue and mitigate the interference of a broad background in sonar images, this step introduces a deep convolutional block to construct a contrast enhancement module. The deep convolutional block utilizes spatially neighboring pixel information to capture local structural features, identify salient regions, and enrich the contextual content of the features.

[0108] For further explanation, please refer to Figure 3 As shown, enhanced visual features The generation includes the following steps:

[0109] Embedding the vision The input is fed into the second Vision Transformer encoder, which performs visual embedding according to equation (10). Processing is carried out, among which, The output of the normalized layer is represented. The normalization operation of the representation layer, This represents the output of the MSA layer. This represents the output of the second VisionTransformer encoder;

[0110] (10);

[0111] Furthermore, the introduction of deep convolutional blocks is to enhance the learning and representation capabilities of this part for visual features. Using convolution to expand the feature channels helps to learn more complex feature representations. In addition, deep convolution helps to capture the local structural features of sonar images. This step is expressed as Equation (11), where, Indicates to Perform a convolution operation. Indicates the kernel size. This represents the output of the depthwise convolution; after this step, the target in the sonar image becomes a salient region, while the broad background becomes a non-salient region.

[0112] (11);

[0113] To minimize the interference from a broad background, the following steps are performed to further suppress unimportant background information and enhance the salience of targets in the sonar image.

[0114] Depth convolutional blocks are based on equation (12) Processing was performed to further enhance the prominent areas, among which, and This represents the convolution output of different convolution kernels;

[0115] (12);

[0116] The deep convolutional block uses a Gaussian Error Linear Unit (GeLU) to suppress insignificant regions, as expressed in equation (13), where, express The output;

[0117] (13);

[0118] Then, through the and Element-wise multiplication is performed to obtain fused features containing rich information; convolution is then performed again to reduce the number of channels to the original input dimension; finally, the enhanced visual features are obtained. Obtained according to formula (14);

[0119] (14).

[0120] As can be seen, the contrast enhancement module, by integrating the Vision Transformer encoder and the depth convolution block, enhances the extraction of visual features of small targets in sonar images, while suppressing the interference of background features, thereby making small targets more prominent and improving the ability to capture small targets in sonar images.

[0121] Step S23: Utilize the feature cross-modal feature fusion module to process cross-modal features. and enhanced visual features The system employs self-attention and guided attention mechanisms, and then dynamically fuses these mechanisms to generate fused features. It should be noted that, in the aforementioned steps, cross-modal features were obtained through the Vision Transformer encoder module. Enhanced visual features were obtained through the contrast enhancement module. ; The visual features in The richness of it, and It contains rich visual features but lacks textual features; therefore, it is necessary to use... To supplement visual features The missing visual features are also used To supplement The missing text features are identified; therefore, a feature cross-referencing and fusion module is used to effectively integrate them. and These two characteristics.

[0122] For further explanation, please refer to Figures 4-6 As shown, fusion features The generation includes the following steps:

[0123] To better integrate these two features, a self-attention mechanism is used for cross-modal features. and enhanced visual features The processing is expressed as equation (15), where, and They represent self-attention. The output of the self-attention mechanism is to optimize the internal information of features by calculating attention scores, thereby capturing global contextual relationships more effectively.

[0124] (15);

[0125] In addition, attention can be guided based on characteristics. Guided by calculating two features and Output features based on similarity Therefore, this step uses a guided attention mechanism to introduce... To achieve and Mutual guidance and mutual guidance characteristics between them and Represented as equation (16), where, Representation of features In features Guided mutual attention algorithm;

[0126] (16);

[0127] In obtaining cross-conduction characteristics and Next, the two are fused; to further enhance the effect of feature fusion, two operations are performed on the two mutually derived features. The first operation is to perform the two mutually derived features according to equation (17). and To interact, among which, This indicates element-wise multiplication. and These represent the interaction results between two mutually derived features; the first operation makes the feature It obtains effective visual feature enhancement attributes and directly complements the features. Missing textual feature information;

[0128] (17);

[0129] The second operation is to use a dynamic weight matrix. and To achieve dynamic feature fusion, it is expressed as equation (18), where, and These are features and The initial weight matrix is ​​generated through a fully connected layer. Represents the Sigmoid function;

[0130] (18);

[0131] The second operation uses the Sigmoid activation function to process the features, introducing non-linearity to help the model learn more complex feature maps; at the same time, it dynamically adjusts the contribution of features in the dynamic fusion process, improving the performance of the module.

[0132] In obtaining features , and dynamic weight matrix , Then, finally, the fusion features Obtained according to formula (19);

[0133] (19).

[0134] It can be seen that the feature mutual guidance and fusion module combines self-attention and guided attention mechanisms to achieve cross-modal feature fusion through mutual guidance, thereby effectively fusing enhanced visual features and cross-modal features; and through the participation of dynamic weight matrix, dynamic feature fusion is achieved while maintaining the dominance of visual features, thus improving the performance of the feature mutual guidance and fusion module.

[0135] Step S24: Fuse features The input is fed into the classifier for answer prediction. Step S25: Based on the comparison between the predicted answer and the answers in the training set sample data, adjust the sonar image visual question answering model and continue training until the training result meets the preset requirements. Steps S24 and S25 are routine operations for model training in machine learning and will not be elaborated here.

[0136] In some embodiments, before executing step S3, the trained sonar image visual question-answering model needs to be validated and tested. The validation process includes using multiple sets of sample data consisting of sonar images, questions, and answers as a validation set. The sample data in the validation set does not overlap with the sample data in the training set. Using the validation set, the trained sonar image visual question-answering model is validated according to steps S21-S24. Based on the comparison between the predicted answers and the answers in the validation set sample data, the optimal sonar image visual question-answering model is selected. The testing process includes using multiple sets of sample data consisting of sonar images, questions, and answers as a test set. The sample data in the test set does not overlap with the sample data in the training and validation sets. Using the test set, the validated sonar image visual question-answering model is tested according to steps S21-S24. Based on the comparison between the predicted answers and the answers in the test set sample data, the performance of the sonar image visual question-answering model is evaluated, thereby obtaining the final sonar image visual question-answering model. Model validation and model testing are routine operations in machine learning and will not be elaborated upon here.

[0137] This invention also provides a sonar image visual question-answering system, including an information acquisition module, a model building module, a model training module, and a prediction module. The information acquisition module acquires sonar images and questions; the model building module constructs a sonar image visual question-answering model using the sonar images and questions as input and the answers as output; the model training module trains the constructed sonar image visual question-answering model using multiple sets of sample data consisting of sonar images, questions, and answers as a training set; and the prediction module responds to sonar image visual question-answering commands by inputting the sonar images and questions into the sonar image visual question-answering model to obtain the answers.

[0138] refer to Figure 7As shown, the present invention will be illustrated through experimental examples below. The sonar image visual question-answering dataset used in the experiment was constructed from an underwater acoustic target detection dataset, containing 4000 sonar images. Due to the different pixel sizes of the sonar images, their pixel size was standardized to 256x256 for the convenience of network training. The entire dataset consists of 20,000 sets of sonar image-question-answer triplets, involving multiple target categories such as cube, sphere, cylinder, human body, tire, circle cage, square cage, and metal bucket. Each sonar image corresponds to 5 sets of question-answer pairs. The questions in this dataset can be divided into three categories: "target count", "yes / no", and "target presence". 16,000 (80%) sonar image-question-answer triples from the entire dataset were used as the training set, 2,000 (10%) sonar image-question-answer triples were used as the validation set, and the remaining 2,000 (10%) sonar image-question-answer triples were used as the test set. Figure 7 Examples of the sonar image visual question answering method of the present invention accurately answering questions are shown. These examples highlight the method's ability to accurately answer three types of questions: "target count", "yes / no", and "target presence".

[0139] Accuracy was used as the evaluation index in the experiment, and was evaluated from three dimensions: accuracy of various types of questions, average accuracy, and overall accuracy.

[0140] Accuracy (Accuracy for short): Calculates the percentage of correct answers given by the method when answering a certain type of question in the total sample. This index is used to evaluate the performance of the method when answering a certain type of question. The calculation formula is shown in Equation (20): where is True Positive, indicating that the actual example is positive and the prediction is positive; is True Negative, indicating that the result is negative and the prediction is negative; is False Positive, indicating that the result is positive and the prediction is negative; is False Negative, indicating that the result is negative and the prediction is positive.

[0141] (20).

[0142] Average accuracy (AA): Calculates the average accuracy of the method across all categories in the dataset. This metric is used to evaluate the method's performance when answering multi-category questions, ensuring that the method is not artificially inflated or understated due to the influence of a particular category of questions. The calculation formula is shown in Equation (21).

[0143] (twenty one).

[0144] Overall accuracy (OA): This measures the percentage of correct answers given by the method when answering all questions in the dataset. This metric evaluates the method's performance on the dataset, and the calculation formula is shown in equation (22).

[0145] (twenty two).

[0146] Table 1 provides a detailed performance comparison of the sonar image visual question answering method of the present invention and the existing baseline method RSVQA on the sonar image visual question answering test set; the baseline method RSVQA can be found in the literature—Lobry, S., Marcos, D., Murray, J., Tuia, D., 2020. Rsvqa: Visual question answering for remote sensing data. IEEE Transactions on Geoscience and Remote Sensing 58, 8555–8566.

[0147] Table 1. Comparison of the proposed method with the baseline method on the sonar image visual question-and-answer test set.

[0148]

[0149] As can be seen from the data in Table 1, the sonar image visual question answering method of the present invention has an average accuracy of 84.08% and an overall accuracy of 87.02%, which is significantly better than the baseline method RSVQA.

[0150] In summary, this invention, through the setting and application of a contrast enhancement module, solves the problem that small targets in sonar images are easily affected by large background interference, making it difficult to effectively establish cross-modal relationships. It enhances the visual features of small targets while suppressing background interference, thus making small targets more prominent and improving the ability to capture small targets in sonar images. Through the setting and application of a feature cross-modal fusion module, it achieves cross-modal fusion of enhanced visual features and cross-modal features, and achieves dynamic fusion while maintaining the dominance of visual features, allowing visual features to have a more significant impact on the final inference, thereby enabling a deeper extraction of rich information from sonar images. Therefore, this invention can deeply mine and fully identify the unique information of sonar images, and thus efficiently and accurately retrieve content of interest to users from sonar images, providing pioneering ideas and methods for visual question answering tasks using sonar images.

[0151] Finally, it should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0152] The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of the present invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in the present invention.

Claims

1. A sonar image visual question answering method, characterized in that, Includes the following steps: S1. Construct a sonar image visual question-answering model; the sonar image visual question-answering model includes a linear projection module, a word embedding module, a Vision Transformer encoder module, a contrast enhancement module, a feature cross-referencing and fusion module, and a classifier; the linear projection module is used to process the sonar image to obtain visual embeddings; the word embedding module is used to process the question to obtain text embeddings; the Vision Transformer encoder module is used to process the co-embedding after concatenating the visual embeddings and text embeddings to generate cross-modal features. The contrast enhancement module is used to process the visual embedding to generate enhanced visual features. The contrast enhancement module includes a second Vision Transformer encoder and a deep convolutional block, the enhanced visual features... The generation process includes using the second Vision Transformer encoder to perform layer normalization on the visual embedding, a multi-head self-attention layer (MSA layer), and MLP processing to output the result. Using the depth convolution block to Multiple convolution processes are performed to obtain enhanced visual features. The feature cross-modal feature fusion module is used to process and dynamically fuse cross-modal features and enhanced visual features using self-attention and guided attention mechanisms to generate fused features. The generation of the fused features includes using a self-attention mechanism for cross-modal features. and enhanced visual features Processing is performed to obtain self-attention output. and Achieve this using a guided attention mechanism. and Mutual guidance between them to obtain mutual guidance characteristics and According to equation (17) and Perform interactive processing to obtain features and ,in, This indicates element-wise multiplication. and These represent the interaction results between two mutually dependent features, and are expressed through a dynamic weight matrix. and Dynamically fuse features, and then... , , and The fusion features are obtained by combining the fusion features; the classifier is used to predict the answer based on the fusion features. (17)； S2. Train the sonar image visual question answering model; S3. Input the sonar image to be identified and the question into the trained sonar image visual question answering model to obtain the answer.

2. The sonar image visual question-answering method according to claim 1, characterized in that, In step S2, the training process of the sonar image visual question-answering model includes using multiple sets of sample data consisting of sonar images, questions, and answers as the training set: S21. Using the linear projection module to process the sonar image. The process involves extracting visual features using a lightweight linear projection method to obtain the visual embedding. Using the word embedding module to address the problem The text is processed using a pre-trained BERT word segmenter to extract text features to obtain text embeddings. Embed the vision and text embedding splicing into co-embedded ;in, This refers to the group number of the sample data. ;Using the first Vision Transformer encoder in the Vision Transformer encoder module for co-embedding Processing is performed to generate cross-modal features. ; S22, Utilize the second Vision Transformer encoder and deep convolutional block in the contrast enhancement module to perform visual embedding. Processing is performed to produce enhanced visual features. ; S23. Utilize the feature cross-modal feature fusion module to process cross-modal features. and enhanced visual features The system employs self-attention and guided attention mechanisms, and then dynamically fuses these mechanisms to generate fused features. ; S24. The fusion feature The input is fed into a classifier to predict the answer; S25. Based on the comparison between the predicted answer and the answer in the training set sample data, adjust the sonar image visual question answering model and continue training until the training result meets the preset requirements.

3. The sonar image visual question-answering method according to claim 2, characterized in that, In step S21, the visual embedding Obtaining it involves the following steps: Sonar images Flatten and slice Image blocks, , and These represent the height, width, and number of channels of a sonar image, respectively. The height, width, and number of channels of an image patch are represented by... , and This indicates the total number of image patches. Calculate according to formula (1); (1)； Each image patch is mapped to a matrix. Dimensional visual markers ,in, Indicates the first Visual markers corresponding to each image patch; image patch embedding Represented as equation (2); (2)； Then classify and label the images. With image patch embedding Perform concatenation and introduce a position embedding matrix. arrive In the end, visual embedding Defined as Equation (3); (3)。 4. The sonar image visual question-answering method according to claim 3, characterized in that, In step S21, the text embedding Obtaining it involves the following steps: Assuming it includes The problem of one word for Each word Transform using the embedding matrix Dimensional text markup Word embedding Represented as equation (4); (4)； Then, use fixed text classification tags. Add to word embeddings In this context, a position embedding matrix is ​​introduced. arrive In the end, text embedding Defined as Equation (5); (5)。 5. The sonar image visual question-answering method according to claim 4, characterized in that, In step S21, the vision is embedded according to equation (6). and text embedding splicing into co-embedded ; (6)。 6. The sonar image visual question-answering method according to claim 5, characterized in that, In step S21, the first Vision Transformer encoder includes an MSA layer and an MLP layer, wherein the MSA layer includes... A parallel self-attention head; the cross-modal feature The generation includes the following steps: The first A self-attention head Represented as equation (7), where, , and These are the query matrix, key matrix, and value matrix, respectively. It is the transpose symbol. The feature dimensions of the key matrix and the query matrix, Indicates input data, It is the Softmax function; (7)； The MSA layer is represented by equation (8), where, , The output of the normalized layer is represented. The normalization operation of the representation layer, This indicates a splicing operation. Indicates the output mapping matrix; (8)； Ultimately, cross-modal features According to equation (9), where, This represents the output of the first VisionTransformer encoder; (9)。 7. The sonar image visual question-answering method according to claim 2 or 3, characterized in that, In step S22, the enhanced visual features The generation includes the following steps: The second Vision Transformer encoder performs visual embedding according to equation (10). Processing is carried out, among which, The output of the normalized layer is represented. The normalization operation of the representation layer, This represents the output of the MSA layer. This represents the output of the second Vision Transformer encoder; (10)； The deep convolutional block captures local structural features of the sonar image according to equation (11), so that the target in the sonar image becomes a salient region, while the broad background becomes a non-salient region; wherein, Indicates to Perform a convolution operation. Indicates the kernel size. This represents the output of a depthwise convolution; (11)； The depthwise convolutional block further enhances the salient regions according to equation (12), wherein, and This represents the convolution output of different convolution kernels; (12)； The depthwise convolutional block uses Gaussian error linear units to suppress insignificant regions, as expressed in equation (13), where, express The output; (13)； Then, to and Element-wise multiplication is performed to obtain fused features, which are then convolved to ultimately enhance the visual features. Obtained according to formula (14); (14)。 8. The sonar image visual question-answering method according to claim 2, characterized in that, In step S23, the fusion feature The generation includes the following steps: Using a self-attention mechanism for cross-modal features and enhanced visual features The processing is expressed as equation (15), where, and They represent self-attention. The output; implemented using a guided attention mechanism. and Mutual guidance and mutual guidance characteristics between them and Represented as equation (16), where, Representation of features In features Guided mutual attention algorithm; (15)； (16)； Then, based on equation (17), the mutual conduction characteristics are analyzed. and Interact; Then through the dynamic weight matrix and To achieve dynamic feature fusion, it is expressed as equation (18), where, and These are features and The initial weight matrix is ​​generated through a fully connected layer. Represents the Sigmoid function; (18)； Ultimately, fusion features Obtained according to equation (19); (19)。 9. The sonar image visual question-answering method according to claim 2, characterized in that, Before performing step S3, the trained sonar image visual question answering model needs to be verified and tested. The verification process includes using multiple sets of sample data consisting of sonar images, questions, and answers as a verification set, wherein the sample data of the verification set does not overlap with the sample data of the training set. Using the verification set, the trained sonar image visual question answering model is verified in steps S21 to S24. Based on the comparison between the predicted answer and the answer in the verification set sample data, the optimal sonar image visual question answering model is selected. The testing process includes using multiple sets of sample data consisting of sonar images, questions, and answers as a test set. The sample data in the test set does not overlap with the sample data in the training set and validation set. Using the test set, the validated sonar image visual question answering model is tested according to steps S21 to S24. The performance of the sonar image visual question answering model is evaluated based on the comparison between the predicted answer and the answer in the test set sample data, thereby obtaining the final sonar image visual question answering model.

10. A sonar image visual question-and-answer system, characterized in that, For performing the sonar image visual question answering method as described in claim 1, the sonar image visual question answering system comprises: The information acquisition module is used to acquire sonar images and information. The model building module is used to construct the sonar image visual question answering model by taking sonar images and questions as inputs and answers as outputs. The model training module is used to train the constructed sonar image visual question answering model using multiple sets of sample data consisting of sonar images, questions, and answers as the training set. The prediction module is used to input sonar images and questions into the sonar image visual question answering model to obtain answers.

Images