A three-dimensional model quantitative retrieval method and system of a multi-modal underwater image

Abstract

The application discloses a kind of three-dimensional model quantitative retrieval method and system of multi-modal underwater image, including acquisition underwater image, input graph attention network model, output three-dimensional model retrieval result;Graph attention network model includes the two-dimensional image dataset of constructing underwater image and three-dimensional model library;Establish image feature extraction network and model feature extraction network to obtain the feature vector of two-dimensional image and three-dimensional model, mapping to the same high-dimensional space to establish adjacency matrix and establish graph structure;Using self-attention mechanism, graph attention network obtains the feature vector after graph embedding, and the model parameters are updated by loss function back propagation;Underwater image to be measured obtains three-dimensional model retrieval result by trained graph attention network;The present application can also realize three-dimensional model retrieval to underwater optical image or sonar image and to underwater image there is partial effective feature's occlusion diagram, it is helpful to the application of three-dimensional model quantitative retrieval in seabed environment detection and underwater search and rescue field.

Description

Technical Field

[0001] This invention relates to the field of 3D model retrieval technology, and more specifically to a 3D model quantization retrieval method and system for multimodal underwater images. Background Technology

[0002] Currently, marine resource development technology is developing rapidly, and the advancement of various equipment and technologies has greatly promoted the continuous expansion of marine development activities. However, there is still much room for improvement in the development of visual imaging technology in the ocean. Due to the complex underwater environment, the use of optical imaging technology is greatly limited, and it is often difficult to obtain complete and clear three-dimensional image features. Sonar imaging technology is widely used, but image features are often severely lost, and it is difficult to identify the shape features of targets in the image. A multimodal image processing technology that can process both optical and sonar images is of great significance and will make cross-modal data retrieval more intuitive and convenient.

[0003] Furthermore, existing methods only perform retrieval on surface images and do not perform model retrieval on underwater optical images and underwater sonar images. In existing 3D model retrieval methods based on 2D images, most methods perform a general category retrieval of the model based on the image, but cannot determine the specific 3D model corresponding to the image. This greatly limits the application of visual technology in fields such as ocean exploration and personnel search and rescue.

[0004] Therefore, how to provide a method and system for quantizing and retrieving three-dimensional models of multimodal underwater images is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a method and system for quantizing and retrieving three-dimensional models from multimodal underwater images. It utilizes multi-angle rendered views of three-dimensional models to characterize the feature information of the three-dimensional models, and constructs a network structure that can process both underwater optical images and sonar images to retrieve three-dimensional models. Furthermore, it relies on self-attention mechanisms and graph attention networks to reduce the similarity gap between three-dimensional models and underwater optical and sonar images, thereby improving the accuracy of three-dimensional model retrieval.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for quantizing and retrieving 3D models from multimodal underwater images includes: acquiring underwater images, inputting the underwater images into a graph attention network model, and outputting 3D model retrieval results.

[0008] The specific content of the graph attention network model includes:

[0009] S1. Construct a two-dimensional image dataset and a three-dimensional model library for underwater images;

[0010] S2. Establish an image feature extraction network and a model feature extraction network. Input the two-dimensional image and the three-dimensional model into the image feature extraction network and the model feature extraction network respectively to obtain the feature vectors of the two-dimensional image and the three-dimensional model. Map them to the same high-dimensional space and establish an adjacency matrix to establish a graph structure.

[0011] S3. Update the eigenvectors V and adjacency matrix A of the graph structure nodes using a self-attention mechanism, and update the eigenvectors of all nodes. Adjacency Matrix Combined into a new graph structure The data is then input into the graph attention network to obtain the feature vector after graph embedding. Based on the feature vector after graph embedding, the similarity loss between the two-dimensional image and the three-dimensional model is calculated through a loss function. The model parameters are then updated through backpropagation to obtain the trained graph attention network model.

[0012] Preferably, the specific content of the graph attention network model also includes:

[0013] S4. Select the underwater image to be tested, obtain the image feature vector through the image feature extraction network in S2, and compare it with the graph structure. The similarity of the feature vectors of all nodes is calculated and a threshold is set. When the similarity is less than the threshold, the node of the image to be tested is added to the graph structure. The graph structure after adding the node of the image to be tested is then input into the trained graph attention network model for graph embedding. The feature vector of the image to be tested updated after graph embedding is used to calculate the loss function with the nodes of the 3D model and arranged in order to obtain the 3D model retrieval result.

[0014] Preferably, the underwater images include underwater optical images or underwater sonar images.

[0015] Preferably, the specific content of S2 includes:

[0016] S21. The two-dimensional image is downsampled, and the VGG network is selected as the image feature extraction network. ImageNet pre-trained weights are used as the initial parameters of the feature extraction network. The downsampled image is input into the VGG network, and the image feature vector is output through a multi-layer convolutional neural network.

[0017] S22. Convert the 3D model into a rendered view from different perspectives, downsample the rendered views of the 3D model from multiple perspectives, and input them into the MVCNN network to obtain the model feature vector that integrates multi-view information.

[0018] S23. Treat the two-dimensional image and the three-dimensional model as nodes and establish the adjacency matrix A in the graph structure. Establish the graph structure G(V,E) based on the image feature vector, the model feature vector and the adjacency matrix, where V represents the feature vectors of the two-dimensional image and the three-dimensional model, and E represents the adjacency matrix A of the relationship between the nodes.

[0019] Preferably, the updated feature vector V of S3 includes the following:

[0020] The feature vectors of nodes in a graph structure are updated using a self-attention mechanism. Nodes with similar features are clustered to obtain updated feature vectors. Specifically

[0021]

[0022] Where, Q∈R n×d and K∈R n×d It is an initialization matrix of the same size as the eigenvectors, V∈R n×d The input feature vector is denoted as n, where n is the total number of images and models, and d is the set vector dimension.

[0023] Preferably, the updated adjacency matrix A of S3 includes the following:

[0024] A self-attention mechanism is used to update the values ​​of the adjacency matrix, replacing the binary values ​​with weight values ​​in the range [0,1]. Specifically:

[0025]

[0026] in, For the updated adjacency matrix, Q A ∈R n×n and K A ∈R n×n It is an initialization matrix of the same size as the adjacency matrix, A∈R n×n d is the adjacency matrix of the input feature vectors, where n is the total number of images and models, and d is the set vector dimension.

[0027] Preferably, the specific content of the feature vector obtained by S3 through the graph attention network after graph embedding includes:

[0028] (1) The new graph structure The input is fed into a graph attention network. For each input node i, the graph attention network computes the relationship between node i and its neighboring nodes j∈N. i correlation coefficient e ij :

[0029] e ij =a([Wh i ||Whj ])

[0030] Where W is the model initialization parameter, used to enhance the feature vectors of node i and node j, and e is a real number. ij Let i be the correlation coefficient between node i and node j;

[0031] (2) Normalize the correlation coefficients of a node to all its neighboring nodes in the adjacency matrix, and then activate the obtained correlation coefficients using the LeakyReLU activation function to obtain the final attention coefficients:

[0032]

[0033] (3) The attention coefficients are weighted and summed with the feature vectors after feature enhancement, and the feature vectors are concatenated using a multi-head mechanism to finally obtain the feature vectors after graph embedding:

[0034]

[0035] Where, α ij W is the attention coefficient. K h is the model initialization parameter for the Kth layer in the graph attention network. i K Let be the feature vector of the i-th node in the K-th layer, and σ be the activation function.

[0036] Preferably, the loss function for S3 is:

[0037] Loss=max(d(a,p)-d(a,Random(n))+margin,0)

[0038] Where d(·) is the Euclidean distance function, a is the feature vector of the target node after embedding through the model graph, p is the model feature vector of the target label, n is the model feature vector of the non-target label, and margin is a hyperparameter used to measure the maximum distance between positive and negative samples.

[0039] Preferably, the similarity calculation in S4 includes:

[0040] Image I to be tested test and image node I image Similarity S1 calculation:

[0041] S1(I test ,I image )=||v test -v image ||2

[0042] Their eigenvectors are v test With v image ;

[0043] Image I to be tested test Similarity S2 between node M and the 3D model is calculated as follows:

[0044] S2(I test M)=||v test -v M ||2

[0045] Its eigenvector is v test and v Model .

[0046] A 3D model quantization retrieval system for multimodal underwater images includes an image acquisition module, a graph attention network model, a 3D model output module, and a graph attention network model building module; the graph attention network model building module includes a dataset construction unit, a graph structure building unit, and a training unit.

[0047] The dataset construction unit is used to build a two-dimensional image dataset and a three-dimensional model library for underwater images, including underwater optical images or underwater sonar images.

[0048] The graph structure building unit is used to build an image feature extraction network and a model feature extraction network. It inputs two-dimensional images and three-dimensional models into the image feature extraction network and the model feature extraction network respectively to obtain the feature vectors of the two-dimensional images and the three-dimensional models, which are then mapped to the same high-dimensional space and an adjacency matrix is ​​established to build the graph structure.

[0049] The training unit is used to update the feature vector V and adjacency matrix A of the graph structure nodes using a self-attention mechanism, and to update the feature vectors of all nodes. Adjacency Matrix Combined into a new graph structure The input is then fed into the graph attention network to obtain the graph-embedded feature vector. The graph-embedded feature vector is then processed by a loss function to calculate the similarity loss between the two-dimensional image and the three-dimensional model. The model parameters are then updated via backpropagation to obtain the trained graph attention network model.

[0050] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a method and system for quantifying and retrieving three-dimensional models from multimodal underwater images. Using multi-view renderings of three-dimensional models as an intermediary, and relying on self-attention mechanisms and graph attention networks, the similarity gap between three-dimensional models and underwater optical and sonar images is reduced. A network structure for retrieving three-dimensional models from underwater optical or sonar images is constructed. Three-dimensional model retrieval can be achieved for both underwater optical and sonar images. Three-dimensional model retrieval can also be achieved for occluded images with some effective features in the input underwater images. This provides a technical solution for the application of three-dimensional model quantification retrieval in the fields of seabed environmental detection and underwater search and rescue. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1 The attached figure is a schematic diagram of a three-dimensional model quantization retrieval method for multimodal underwater images provided by the present invention;

[0053] Figure 2 The attached figure is a schematic diagram of the method for obtaining rendering views of a 3D model from different perspectives provided by the present invention;

[0054] Figure 3 The attached figure is a schematic diagram of the graph structure creation method provided by the present invention;

[0055] Figure 4 The attached figure is a schematic diagram of the graph attention network training provided by the present invention;

[0056] Figure 5 The attached figure is a schematic diagram of the retrieval of a three-dimensional model of an underwater image to be tested provided by the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] This invention discloses a method for quantizing and retrieving three-dimensional models from multimodal underwater images, such as... Figure 1 This includes: acquiring underwater images, inputting the underwater images into a graph attention network model, and outputting the 3D model retrieval results;

[0059] The specific content of the graph attention network model includes:

[0060] S1. Construct a two-dimensional image dataset and a three-dimensional model library for underwater images;

[0061] S2. Establish an image feature extraction network and a model feature extraction network. Input the 2D image and the 3D model into the image feature extraction network and the model feature extraction network respectively to obtain the feature vectors of the 2D image and the 3D model. Map them to the same high-dimensional space and establish an adjacency matrix to build a graph structure, such as... Figure 3 ;

[0062] S3. Update the eigenvectors V and adjacency matrix A of the graph structure nodes using a self-attention mechanism, and update the eigenvectors of all nodes. Adjacency Matrix Combined into a new graph structure The input is then fed into a graph attention network to obtain the graph-embedded feature vector. Based on the graph-embedded feature vector, the similarity loss between the 2D image and the 3D model is calculated using a loss function. Backpropagation is then used to update the model parameters, resulting in the trained graph attention network model, as shown below. Figure 4 .

[0063] To further implement the above technical solutions, the graph attention network model also includes the following specific features:

[0064] S4. Select the underwater image to be tested, obtain the image feature vector through the image feature extraction network in S2, and compare it with the graph structure. The similarity of the feature vectors of all nodes is calculated, and a threshold is set. When the similarity is less than the threshold, a node of the image to be tested is added to the graph structure. The graph structure after adding the node of the image to be tested is then input into the trained graph attention network model for graph embedding. The updated feature vector of the image to be tested after graph embedding is compared with the loss function of the 3D model nodes and arranged in order to obtain the 3D model retrieval result, such as... Figure 5 .

[0065] In this embodiment, the specific content of S1 includes: S11 uses a simulation model to establish an underwater target environment, acquires optical images containing targets in different scenes through an underwater binocular camera, and combines them with the three-dimensional models in the Pascol3D+ dataset, divides the data into training set and test set, calibrates the data, matches it with the three-dimensional models in the library, and completes the data acquisition work.

[0066] In this example, there are 5326 underwater optical images, which are divided into 4 categories: airplanes, ships, bottles and cars; and there are 32 3D models.

[0067] S12 uses simulated sonar imaging software to obtain simulated sonar images of different targets, and combines them with the three-dimensional models in the Pascol3D+ dataset. The data is divided into training and testing sets, and the data is calibrated and matched with the three-dimensional models in the library to complete the data acquisition work.

[0068] In this example, there are 989 underwater sonar images, divided into two categories: aircraft and ships, and 14 3D models.

[0069] To further implement the above technical solutions, underwater images include underwater optical images or underwater sonar images.

[0070] To further implement the above technical solutions, such as Figure 3 The specific content of S2 includes:

[0071] S21 downsamples the two-dimensional image, selects the VGG network as the image feature extraction network, uses ImageNet pre-trained weights as the initial parameters of the feature extraction network, inputs the downsampled image into the VGG network, and outputs the image feature vector through a multi-layer convolutional neural network.

[0072] S22 converts the 3D model into rendered views from different perspectives, downsamples these multi-view rendered views, and inputs them into the MVCNN network to obtain model feature vectors that fuse multi-view information, such as... Figure 2 ;

[0073] S23 treats the two-dimensional image and the three-dimensional model as nodes and establishes the adjacency matrix A in the graph structure. Based on the image feature vector, the model feature vector and the adjacency matrix, a graph structure G(V,E) is established, where V represents the feature vectors of the two-dimensional image and the three-dimensional model, and E represents the adjacency matrix A of the relationship between the nodes.

[0074] In practical applications, such as Figure 2 In S22, the 3D model is imported into 3D graphics software. Based on the retrieval requirements, n virtual cameras are placed in a circular pattern. The virtual cameras can convert the 3D model into rendering views from different angles. Then, the multi-view rendering views of the model are downsampled and input into the MVCNN network. Each view first passes through a ResNet with ImageNet pre-trained weights for feature extraction, obtaining feature vectors from n views. Then, a view pooling layer is used to normalize the feature vectors obtained from the n views, resulting in a model feature vector that integrates multi-view information. In S23, the image and the model are treated as nodes, and an adjacency matrix A is established in the graph structure. In the training set, the value of the adjacency matrix A is set according to the class relationship between the image and the model. That is, when the class of the nodes is the same, A(i,j) = 1, indicating that node i and node j have an intrinsic relationship; otherwise, A(i,j) = 0, indicating that there is no relationship between node i and node j.

[0075] In this embodiment, in S21, the two-dimensional image size is downsampled to 224×224, and the vector dimension is set to 128; in S22, the three-dimensional graphics software is Blender, the number of virtual cameras n is 12, the angle interval of each virtual camera is 30°, the multi-view rendering image of the model is downsampled to 224×224, and the feature vector dimension of the model is set to 128.

[0076] To further implement the above technical solutions, such as Figure 4The specific content of the updated feature vector V of S3 includes:

[0077] The feature vectors of nodes in a graph structure are updated using a self-attention mechanism. Nodes with similar features are clustered to obtain updated feature vectors. Specifically

[0078]

[0079] Where, Q∈R n×d and K∈R n×d It is an initialization matrix of the same size as the eigenvectors, V∈R n×d The input feature vector is denoted as n, where n is the total number of images and models, and d is the set vector dimension.

[0080] To further implement the above technical solution, the specific content of S3's updated adjacency matrix A includes:

[0081] In step S2, the adjacency matrix A is binary, where A(i,j) = 1 represents a relationship between nodes, and A(i,j) = 0 represents no relationship. This is not accurate enough to describe the relationships between nodes, as the strength of the connections varies. Therefore, the model also uses a self-attention mechanism to update the values ​​of the adjacency matrix A, replacing the binary values ​​of A with weight values ​​in the range [0,1]. This allows the model to measure the strength of the relationships between nodes. Furthermore, through the self-attention mechanism, the model can identify potential relationships between different types of nodes, improving the model's retrieval accuracy and resulting in a more accurate adjacency matrix. It can more accurately reflect the connections between nodes, specifically:

[0082]

[0083] in, For the updated adjacency matrix, Q A ∈R n×n and K A ∈R n×n It is an initialization matrix of the same size as the adjacency matrix, A∈R n×n d is the adjacency matrix of the input feature vectors, where n is the total number of images and models, and d is the set vector dimension.

[0084] To further implement the above technical solution, the specific content of the feature vector obtained by S3 through the graph attention network after graph embedding includes:

[0085] (1) The new graph structure The input is fed into a graph attention network. For each input node i, the graph attention network computes the relationship between node i and its neighboring nodes j∈N. icorrelation coefficient e ij :

[0086] e ij =a([Wh i ||Wh j ])

[0087] Where W is the model initialization parameter, used to enhance the feature vectors of node i and node j, and e is a real number. ij Let i be the correlation coefficient between node i and node j;

[0088] (2) Normalize the correlation coefficients of a node to all its neighboring nodes in the adjacency matrix, and then activate the obtained correlation coefficients using the LeakyReLU activation function to obtain the final attention coefficients:

[0089]

[0090] (3) The attention coefficients are weighted and summed with the enhanced feature vectors, and a multi-head mechanism is used to concatenate the feature vectors to obtain the final feature vector after graph embedding. The graph attention network uses the adjacency matrix... The values ​​are used for graph embedding, and nodes with high correlation are clustered:

[0091]

[0092] Where, α ij W is the attention coefficient. K h is the model initialization parameter for the Kth layer in the graph attention network. i K Let be the feature vector of the i-th node in the K-th layer, and σ be the activation function.

[0093] In this embodiment, ReLU is used as the activation function, and the feature vectors of each node after final graph embedding are obtained from the connections between model layers.

[0094] In practical applications, the multi-head mechanism is used to integrate the features of neighboring nodes into the node itself to achieve clustering and improve retrieval accuracy. Even if the extracted target features are not obvious, they can be supplemented by the features of its neighboring nodes to achieve the retrieval purpose. This allows the model to provide retrieval model results and obtain good retrieval performance without distinguishing between optical images and sonar images.

[0095] To further implement the above technical solution, the specific content of the loss function of S3 is as follows: Calculate the ternary loss between the updated image feature vector and the model feature vector, bringing the retrieved image closer to the target model feature vector and increasing the distance between the retrieved image and non-target models. This makes similar nodes closer and dissimilar nodes farther apart, reducing the likelihood of similar nodes and improving retrieval accuracy. The retrieval results are then arranged in order. The ternary loss function is:

[0096] Loss=max(d(a,p)-d(a,Random(n))+margin,0)

[0097] Where d(·) is the Euclidean distance function, a is the feature vector of the target node after embedding through the model graph, p is the model feature vector of the target label, n is the model feature vector of the non-target label, and margin is a hyperparameter used to measure the maximum distance between positive and negative samples.

[0098] In this embodiment, the input is a triplet, including the target Anchor, positive sample, and negative sample. The model uses the feature vector of the input node to narrow the distance with the feature vector of the 3D model shown by the label, and widen the distance with the feature vector of the 3D model shown by the non-label. By optimizing the similarity distance between the positive sample and the target, the model reduces the difference between similar data of different modalities and improves the accuracy of model retrieval.

[0099] In practical applications, for complex underwater environments, based on the correlation between feature vectors between nodes, image features of easily identifiable targets of the same type can be integrated into the retrieval image in a complex environment or where the target is occluded. This eliminates the impact of complex environment or target occlusion on retrieval and ultimately improves the model's retrieval accuracy for underwater targets.

[0100] To further implement the above technical solution, the similarity calculation in S4 includes:

[0101] Image I to be tested test and image node I image Similarity S1 calculation:

[0102] S1(I test ,I image )=||v test -v image ||2

[0103] Their eigenvectors are v test With v image ;

[0104] Image I to be tested test Similarity S2 between node M and the 3D model is calculated as follows:

[0105] S2(I test M)=||v test -v M ||2

[0106] Its eigenvector is v test and v Model .

[0107] In this embodiment, the hardware environment is Intel Xeon W-2125 + GV100 32G + 32G RAM, and the software environment is Ubuntu 18.04 + CUDA 11.3 + PyTorch 1.9 + Python 3.9. Python is used to process data and complete image retrieval and retrieval index evaluation.

[0108] The median error and accuracy are used as evaluation metrics to measure the model's performance.

[0109] The average error is:

[0110]

[0111] Where v Image v is the feature vector of the image to be retrieved. label The feature vector of the 3D model corresponding to the image label to be retrieved, where N is the total number of images in a certain category;

[0112] The accuracy rate is:

[0113]

[0114] Where TP is the number of predicted positive classes, TN is the number of predicted negative classes, and N is the total number of images of a certain category, that is, the proportion of the number of images correctly predicted by the model out of the total number.

[0115] Evaluation of category search results:

[0116]

[0117] Quantitative evaluation of search results:

[0118]

[0119]

[0120] Evaluation results show that the method of the present invention can provide high-precision model retrieval results for both underwater optical images and sonar images as input images.

[0121] A 3D model quantization retrieval system for multimodal underwater images includes an image acquisition module, a graph attention network model, a 3D model output module, and a graph attention network model building module; the graph attention network model building module includes a dataset construction unit, a graph structure building unit, and a training unit.

[0122] The dataset construction unit is used to build a two-dimensional image dataset and a three-dimensional model library for underwater images, including underwater optical images or underwater sonar images.

[0123] The graph structure building unit is used to build an image feature extraction network and a model feature extraction network. It inputs two-dimensional images and three-dimensional models into the image feature extraction network and the model feature extraction network respectively to obtain the feature vectors of the two-dimensional images and the three-dimensional models, which are then mapped to the same high-dimensional space and an adjacency matrix is ​​established to build the graph structure.

[0124] The training unit is used to update the feature vector V and adjacency matrix A of the graph structure nodes using a self-attention mechanism, and to update the feature vectors of all nodes. Adjacency Matrix Combined into a new graph structure The graph is then input into the graph attention network to obtain the graph-embedded feature vector. The graph-embedded feature vector is then processed by a loss function to calculate the similarity loss between the two-dimensional image and the three-dimensional model. The model parameters are then updated by backpropagation to obtain the trained graph attention network model.

[0125] To further implement the above technical solution, the graph attention network model building module also includes a testing unit, used to select the underwater optical image or underwater sonar image to be tested, obtain the image feature vector through the S3 image feature extraction network, and then compare it with the graph structure. The similarity of the feature vectors of all nodes is calculated and a threshold is set. When the similarity is less than the threshold, the node of the image to be tested is added to the graph structure. The graph structure after adding the node of the image to be tested is then input into the trained graph attention network for graph embedding. The feature vector of the image to be tested updated after graph embedding is used to calculate the ternary loss function with the nodes of the 3D model and arranged in order to obtain the 3D model retrieval result.

[0126] The calculation results will be sorted according to similarity, and the output search results will be the models with the highest similarity.

[0127] To further implement the above technical solution, the graph structure establishment module includes an image feature vector acquisition unit, a model feature vector acquisition unit, a connection matrix acquisition unit, and a graph structure acquisition unit;

[0128] The image feature vector acquisition unit is used to downsample the two-dimensional image. The VGG network is selected as the image feature extraction network, and the ImageNet pre-trained weights are used as the initial parameters of the feature extraction network. The downsampled image is input into the VGG network, and the image feature vector is output through a multi-layer convolutional neural network.

[0129] The model feature vector acquisition unit is used to convert the 3D model into a rendered view from different perspectives. The rendered views of the 3D model from multiple perspectives are downsampled and input into the MVCNN network to obtain the model feature vector that integrates multi-view information.

[0130] The adjacency matrix acquisition unit is used to treat the two-dimensional image and the three-dimensional model as nodes and establish the adjacency matrix A in the graph structure;

[0131] The graph structure acquisition unit is used to establish a graph structure G(V,E) based on the image feature vector, the model feature vector, and the adjacency matrix, where V represents the feature vectors of the image and the model, and E represents the adjacency matrix A representing the relationship between nodes.

[0132] To further implement the above technical solution, the training module includes: a self-attention processing unit, a graph attention processing unit, and a similarity loss processing unit;

[0133] The self-attention processing unit updates the feature vectors of graph structure nodes using a self-attention mechanism, clusters the feature vectors of nodes with similar features, and obtains the updated feature vectors of the nodes. Specifically

[0134]

[0135] Where, Q∈R n×d and K∈R n×d It is an initialization matrix of the same size as the eigenvectors, V∈R n×d The input feature vector is denoted as n, where n is the total number of images and models, and d is the set vector dimension.

[0136] It is also used to update the values ​​of the adjacency matrix using a self-attention mechanism, replacing the binary values ​​of the adjacency matrix with weight values ​​in the range [0,1], specifically:

[0137]

[0138] in, For the updated adjacency matrix, Q A ∈R n×n and K A ∈R n×n It is an initialization matrix of the same size as the adjacency matrix, A∈R n×nThe input feature vector is given, where n is the total number of images and models, and d is the set vector dimension.

[0139] The graph attention processing unit is used to process new graph structures. The input is fed into a graph attention network. For each input node i, the graph attention network computes the relationship between node i and its neighboring nodes j∈N. i correlation coefficient e ij :

[0140] e ij =a([Wh i ||Wh j ])

[0141] Where W is the model initialization parameter, used to enhance the feature vectors of node i and node j, and e is a real number. ij Let i be the correlation coefficient between node i and node j;

[0142] It is also used to probabilistically normalize the correlation coefficients of a node to all its neighboring nodes in the adjacency matrix, and then activates the obtained correlation coefficients using the LeakyReLU activation function to obtain the final attention coefficients:

[0143]

[0144] It is also used to perform a weighted summation of the attention coefficients and the feature vectors after feature enhancement, and to concatenate the feature vectors using a multi-head mechanism, ultimately obtaining the feature vectors after graph embedding:

[0145]

[0146] The similarity loss processing unit calculates the ternary loss between the updated image feature vector and the model feature vector, and arranges the results in order to obtain the retrieval results. The ternary loss function is:

[0147] Loss=max(d(a,p)-d(a,Random(n))+margin,0)

[0148] The input is a triple, consisting of the target Anchor, a positive sample, and a negative sample.

[0149] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0150] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for 3D model-based quantitative retrieval of multimodal underwater images, characterized in that, The method comprises the following steps: Collecting an underwater image, inputting the underwater image into a graph attention network model, and outputting a three-dimensional model retrieval result; The underwater image comprises an underwater optical image or an underwater sonar image; The specific content of the graph attention network model comprises: S1. Constructing a two-dimensional image dataset of the underwater image and a three-dimensional model library; S2. Establishing an image feature extraction network and a model feature extraction network, inputting the two-dimensional image and the three-dimensional model into the image feature extraction network and the model feature extraction network to obtain feature vectors of the two-dimensional image and the three-dimensional model respectively, mapping them into the same high-dimensional space, and establishing an adjacency matrix to establish a graph structure; S3. Update the feature vector V of the nodes of the graph structure and the adjacency matrix A using the self-attention mechanism, and combine all the updated feature vectors of the nodes with the adjacency matrix into a new graph structure and input into the graph attention network to obtain the graph-embedded feature vector, calculate the similarity loss between the two-dimensional image and the three-dimensional model according to the graph-embedded feature vector through the loss function, update the model parameters through back propagation, and obtain the trained graph attention network model. 2.The method of claim 1, wherein, The specific content of the graph attention network model further comprises: S4. Select the underwater image to be tested, obtain the image feature vector through the image feature extraction network of S2, and calculate the similarity with the feature vectors of all nodes in the graph structure Set a threshold, and when the similarity is less than the threshold, add the image node to be tested in the graph structure, and input the graph structure after adding the image node to be tested into the trained graph attention network model for graph embedding. Update the feature vector of the image to be tested after graph embedding, and calculate the loss function with the three-dimensional model node, and arrange in order to obtain the three-dimensional model retrieval result. 3.The method of claim 1, wherein, The specific content of S2 comprises: S21. Down-sampling the two-dimensional image, selecting a VGG network as the image feature extraction network, using ImageNet pre-training weights as the initial parameters of the feature extraction network, inputting the down-sampled image into the VGG network, and outputting an image feature vector through a multi-layer convolutional neural network; S22. Converting the three-dimensional model into rendered views under different perspectives, down-sampling the multi-perspective rendered views of the three-dimensional model, and inputting them into an MVCNN network to obtain a model feature vector fused with multi-perspective information; S23. Treating the two-dimensional image and the three-dimensional model as nodes and establishing an adjacency matrix A in the graph structure, and establishing a graph structure G(V, E) according to the image feature vector, the model feature vector, and the adjacency matrix A, wherein V represents the feature vectors of the two-dimensional image and the three-dimensional model, and E represents the adjacency matrix A of the relationship between the nodes. 4.The method of claim 1, wherein, The specific content of updating the feature vector V of S3 comprises: The feature vectors of the nodes of the graph structure are updated by using a self-attention mechanism, and the feature vectors of nodes with similar features are clustered to obtain updated feature vectors of the nodes Specifically: where Q ∈ R n×d and K ∈ R n×d is an initialized matrix of the same size as the feature vector, V ∈ R n×d is the input feature vector, where n is the total number of images and models, and d is the dimension of the setting vector. 5.The method of claim 1, wherein, The specific content of updating the adjacency matrix A of S3 comprises: The self-attention mechanism is used to update the value of the adjacency matrix, and the value of the adjacency matrix is replaced from a binary value to a weight value in the range of [0, 1], specifically: wherein, is the updated adjacency matrix, Q A ∈R n×n and K A ∈R n×n is an initialized matrix of the same size as the adjacency matrix, A∈R n×n is the adjacency matrix of input feature vectors, where n is the total number of images and models, and d is the dimension of the setting vector. 6.The method of claim 1, wherein, The specific content of obtaining the feature vector after graph embedding of S3 through the graph attention network comprises: (1) a new graph structure is input to a graph attention network, which computes a relevance coefficient e for each node i and its neighbor nodes j e N i for each node i and its neighbor nodes j e N ij : e ij = a([Wh i || Wh j ]) wherein W is a model initialization parameter, used to perform feature enhancement on the feature vectors of node i and node j, real number e ij is a correlation coefficient of node i and node j; (2) The correlation coefficients of all adjacent nodes of the node in the adjacency matrix are normalized, and the obtained correlation coefficients are activated using a LeakyReLU activation function to obtain the final attention coefficient: (3) The attention coefficient and the feature vector after feature enhancement are weighted and summed, and a multi-head mechanism is used to splice the feature vectors to finally obtain the feature vector after graph embedding: wherein α ij is an attention coefficient, W K is a model initialization parameter of the Kth layer in the graph attention network, is a feature vector of the ith node of the Kth layer, and σ is an activation function. 7.The method of claim 1, wherein, The loss function of S3 is: Loss=max(d(a,p)-d(a,Random(n))+margin,0) Wherein, d(·) is an Euclidean distance function, a is a feature vector of a target node after model graph embedding, p is a model feature vector of a target label, n is a model feature vector of a non-target label, and margin is a hyperparameter for measuring the maximum distance between positive samples and negative samples. 8.The method of claim 1, wherein, The similarity calculation in S4 comprises: Image to be tested I test and image node I image Similarity S1 calculation: S1(I test ,I image ) = ||v test -v image ||2 characteristic vectors of which are v test and v image ; Image I to be tested test and similarity S2 computation of the three-dimensional model node M: S2(I test ,M) = ||v test -v M ||2 The eigenvector of which is v test and v M .

9. A three-dimensional model quantitative retrieval system of multi-modal underwater images based on the three-dimensional model quantitative retrieval method of multi-modal underwater images according to any one of claims 1-8, characterized in that, The method comprises the following steps: The method comprises the following steps: A data set construction unit is configured to construct a two-dimensional image data set of underwater images and a three-dimensional model library, the underwater images including underwater optical images or underwater sonar images; A graph structure establishment unit is configured to establish an image feature extraction network and a model feature extraction network, input the two-dimensional images and the three-dimensional models into the image feature extraction network and the model feature extraction network to obtain feature vectors of the two-dimensional images and the three-dimensional models respectively, map the feature vectors into a same high-dimensional space, and establish an adjacency matrix to establish a graph structure. The training unit is configured to update the feature vectors V of the nodes of the graph structure and the adjacency matrix A by using a self-attention mechanism, and combine all the updated feature vectors V of the nodes and the adjacency matrix A into a new graph structure with the adjacency matrix into a new graph structure and input into the graph attention network to obtain a graph-embedded feature vector, and the graph-embedded feature vector is used to calculate a similarity loss between a two-dimensional image and a three-dimensional model by using a loss function, and the model parameters are updated by back propagation, so as to obtain a trained graph attention network model.

Images