Ship image retrieval method based on deep hash

Abstract

The invention discloses a deep hash-based ship image retrieval method. The method comprises the following steps: establishing an initial ship image database D0; constructing a deep Hash network based on a feature extraction module and a dynamic Hash coding module; training a deep Hash network by using the D0 to obtain a ship image retrieval model M0, generating a binary Hash code for a sample in the D0 by using the M0, and storing the binary Hash code in a retrieval database; and updating a ship image retrieval model based on incremental learning, generating a binary hash code for a newly added sample by using the updated model, and merging the binary hash code into a retrieval database for ship image retrieval. The invention provides a scheme combining a ViT network and a dynamic Hash code generation mechanism, and the Hash code can be dynamically adjusted when the data stream changes, so that the flexibility and adaptability of the system when facing large-scale and complex environment data are ensured, the Hash conflict is reduced, the retrieval efficiency is improved, and the training cost is reduced. The method is suitable for application scenes of small proportion, complex background and the like of ship images in complex marine environments.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a ship image retrieval method based on deep hashing. Background Art

[0002] Ships can be divided into cargo ships, tankers, fishing boats, etc. They are diverse in the marine environment, which increases the complexity of image recognition and retrieval. Existing image retrieval technologies mainly include the following. Method 1: Methods based on manual feature extraction, such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features) and other feature detection algorithms. These methods extract key point features for image matching; Method 2: Convolutional neural network (CNN) method based on deep learning, which uses deep learning models to automatically extract deep features of images to achieve higher-precision image retrieval; Method 3: Image retrieval technology based on hash coding, which compresses image features into low-dimensional hash codes to achieve fast and efficient retrieval. However, these methods all have some defects.

[0003] Regarding method 1: Feature extraction is highly subjective and relies on the experience of designers, making it difficult to extract features with universal adaptability. Manually extracted features are difficult to cope with complex backgrounds and environmental interference, resulting in low retrieval accuracy. The computational efficiency is low, and the manual feature extraction process is cumbersome and time-consuming, making it difficult to meet the real-time retrieval requirements of large-scale data. Traditional computer vision feature detection algorithms such as SIFT and SURF are used for ship image retrieval, but they have limited adaptability in dealing with scale, rotation, and illumination changes, especially in complex marine environments.

[0004] Regarding method 2, the convolutional neural network method based on deep learning is limited by the local receptive field when extracting image features. The reason is that CNN uses convolution operations, which makes the network only learn from local features and rely on the receptive field accumulated layer by layer to obtain global information. In addition, the convolutional neural network is sensitive to the spatial changes of features and performs poorly when facing spatial changes such as rotation, scaling or deformation of objects. The above defects make CNN perform poorly when extracting ship image features under complex marine backgrounds.

[0005] Regarding method three, the image retrieval technology based on traditional hash coding has deficiencies in many aspects. First, the traditional hash coding method relies on manual features or shallow feature transformations, which cannot effectively capture the complex features in the image, resulting in low image retrieval accuracy; second, the traditional hash coding is static, and the generated hash code cannot be adjusted according to the changes in the data, lacks adaptability, and is difficult to cope with changes in different scenarios and data; in addition, the traditional hash coding has a serious problem of information loss when facing high-dimensional data, resulting in low retrieval accuracy. These deficiencies reflect the limitations of traditional hash coding technology when facing the diversity and dynamic environment of ships in complex marine environments; finally, due to the catastrophic forgetting problem of the traditional deep hash network, all historical data needs to be retrained every time new data arrives, and when the amount of data is large, there will be a serious computational burden.

[0006] Vision Transformer (ViT) is a Transformer-based image classification model, which mainly consists of three parts: image feature embedding module, Transformer encoder module and MLP classification module. The image feature embedding module divides the input image into patches of fixed size and then flattens these patches into a series of vectors. The Transformer encoder module contains multiple encoder layers, each of which contains a multi-head self-attention sublayer and a feedforward neural network sublayer. In the ViT model, the image feature embedding module and the Transformer encoder module are mainly used for feature extraction, so they are collectively referred to as the ViT feature extraction module. Summary of the invention

[0007] The purpose of the present invention is to provide a method for solving the above-mentioned problems, which can efficiently and accurately complete retrieval tasks in complex and dynamic environments, overcome the problem of catastrophic forgetting in the incremental learning process, and use a small amount of historical data to enable the new network to maintain the hash code expression of the old network for the old data, while adding the ability to express new data, thereby reducing computing power and time costs. A ship image retrieval method based on deep hashing.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is as follows: a ship image retrieval method based on deep hashing comprises the following steps; S1, establish the initial ship image database D 0 ; Obtain multiple ship images of different categories, preprocess and label each ship image to obtain samples, and all samples constitute the ship image database D 0 ; S2, construct a deep hash network, including the ViT feature extraction modules M connected in sequence ViTand dynamic hash coding module M hash , where M ViT For D 0 Extract features from sample x and obtain the corresponding image features f(x), M hash Used to dynamically hash f(x) and output a real-valued hash code , and then converted into a binary hash code h(x); S3, with D 0 Train the deep hashing network to obtain the ship image retrieval model M 0 , use M 0 Right D 0 Generate binary hash codes for all samples in the search database; S4, updating the ship image retrieval model based on incremental learning, obtaining an updated model each time, and the updating method of the t-th incremental learning includes steps S41-S45; S41, obtaining the ship image database D of the tth incremental learning t , marked as a new database D new , t≥1, and when t=1, D t =D 0 ; from D 0 ~D t-1 Each category selects q samples to form the old database D old , mark the updated model after the t-1th time as M old ; S42, constructed to update M hash The total loss L total , including the cost function L stability And hash comparison loss L hash ; , , = , In the formula, x i D old The sample, x j D new Medium sample, , M hash Before and after the tth update, the sample x i The real-valued hash code of is the L2 norm, M hash After the tth update, the sample x j The real-valued hash code, S ij For sample x i and xj The similarity value of x i and x j If the categories are the same, then S ij =1, otherwise S ij =0,max is the maximum value operation, m is the boundary distance for distinguishing samples of different categories, and γ is the adjustment parameter of Lhash; S43, constructed to update M ViT The loss function L ViT ; , In the formula, , M ViT The corresponding samples x before and after the tth update i The image features of M ViT Adjustment parameters for the ability to remember old data, M ViT In D new The cross entropy loss calculated above; S44, construct the total loss of the model L, L = L total +L ViT ; S45, D old , D new After merging, enter M old , to minimize the total model loss L training M ViT and M hash , get the tth updated model M t ; S5, use M t For D new Generate binary hash codes for the samples and store them in the retrieval database; S6, based on M t Conduct ship image retrieval; Get the ship image x to be queried query , by M t Generate a binary hash code, search in the search database, and output the search results.

[0009] As a preferred embodiment: S1 specifically comprises: unifying the resolution of the collected ship images to 256 256, get samples by category classification and labeling, then rotate each sample by several angles, each rotation forms a new sample, and the sample and the new sample together form the initial ship image database D 0 .

[0010] As a preference: in S2, M hash The real-valued hash code is obtained according to the following formula and binary hash code h(x); , , Where W h , b h M hash The initial bias matrix and initial weight matrix, is the hyperbolic tangent function, is a symbolic function, if ≥0, then ,otherwise , K is the hash code length.

[0011] Preferably: the ViT feature extraction module is composed of the image feature embedding module and the Transformer encoder module of the ViT model; The image feature embedding module is used to divide the sample x into several image blocks, then flatten each image block into a one-dimensional vector, map it into a D-dimensional vector through a linear mapping layer and perform position encoding to obtain a position encoding vector; The Transformer encoder module includes multiple layers of serially connected Transformer encoders, which are used to input a position encoding vector and output image features f(x) after multiple feature extractions.

[0012] As a preferred embodiment: in S45, each training M hash When based on L total Update the weight matrix and bias matrix of the hash function during backpropagation; , In the formula, , are the weight matrices before and after the update, respectively, and when t=1, , , are the bias matrices before and after the update, respectively, and when t=1, , is the learning rate.

[0013] As a preferred embodiment: S6 is specifically: S61, preset a distance threshold τ and a preset number of search results M; S62, use M t Generate x query The binary hash code h(x query )Calculate h(x query ) and the Hamming distance of all binary hash codes in the retrieval database; S63, sorting the samples in the search database in ascending order according to the Hamming distance, and generating a sequence table; S64, selecting the first M samples in the sequence list whose Hamming distance is less than τ as the search results.

[0014] Preferably, the categories include seagoing vessels, container ships, three-five ships, civilian ships and law enforcement ships.

[0015] The present invention is suitable for identifying the increasing number of ship samples at sea checkpoints. The idea of ​​the present invention is: First, a deep hash network is constructed based on the ViT feature extraction module and the dynamic hash coding module. 0 Train the deep hashing network to obtain the ship image retrieval model M 0 , use M 0 Right D 0 Generate binary hash codes for all samples in the retrieval database.

[0016] Subsequently, ship images are continuously collected to form a ship image database regularly. For example, the ship image database collected for the first and second times is marked as D 1 , D 2 , the ship image database constructed multiple times forms a data stream D={D 1 ,D 2 ,…,D t}, and then proposed a dynamic hash code generation and update mechanism based on data stream combined with sample incremental learning. When a new database / data stream is input, the weight matrix and bias matrix of dynamic hash coding are updated to re-encode the new sample. Since the above two matrices are updated every time a database is input, the encoding method of the samples in each database is different, forming a dynamic encoding method. Specifically: (1) First define the hash function according to the formula Generate real-valued hash codes , this is to make the gradient transferable during the training process, so the continuous activation function tanh is used to approximate the sign function; later, in order to make M hash The output is binary coded, so Binary encoding is performed, and the present invention is based on the formula , the real-valued hash code Converted into a binary hash code h(x), is a sign function, ,but ,otherwise .

[0017] (2) When a new database arrives, the original hash function may no longer be applicable to the new samples, which will lead to increased hash conflicts, decreased retrieval accuracy and other issues. In order to solve this problem, the present invention introduces a dynamic hash code update mechanism. While updating the hash function, the hash code of the old data is kept unchanged as much as possible to reduce the impact on the existing index structure, while enabling the new hash function to effectively distinguish similar and dissimilar samples in the new data. The dynamic hash function update method is divided into the following steps: First, in order to keep the hash code of the old data stable, for the sample x in the old database i , we hope that the updated hash code is as consistent as possible with the original hash code, so we design the cost function , and in order to adapt to the sample x in the new database j Hash code learning, using hash code contrast loss to ensure that the similarity relationship is maintained in the hash space, and designing hash contrast loss L hash , which minimizes the hash code distance between similar samples and maximizes the distance between dissimilar samples. hash The γ in is used to balance the adaptability of new data and the stability of old data. In summary, for L stability , which means ensuring that the hash codes of the old data samples obtained by sampling are as consistent as possible between the two model outputs; for L hash , which means ensuring that the hash codes of similar images are brought closer and the hash codes of dissimilar images are pushed further away, thereby generating hash codes with higher discrimination.

[0018] (3) In order to further improve the adaptability and stability of the overall model when the data changes, the present invention also combines the dynamic hash code update mechanism with the incremental learning mechanism to jointly ensure the adaptability and stability of the system when the data changes. ViT , whose formula is used for M in each training ViT Fine-tune and update M ViT parameters to adapt to new data while maintaining performance on old data.

[0019] Compared with the prior art, the advantages of the present invention are: (1) Combining the powerful feature extraction capability of the ViT network: In the present invention, the deep features of the ship image are extracted through the ViT feature extraction module. Compared with the traditional convolutional neural network, ViT can better capture the global features of the image, thereby improving the accuracy and robustness of the ship image retrieval.

[0020] (2) Dynamic hash code generation and update mechanism: The present invention proposes a dynamic hash code generation and update mechanism, which can dynamically adjust the hash code when the data stream changes, so as to ensure the flexibility and adaptability of the system when facing large-scale and complex environment data, thereby reducing hash conflicts and improving retrieval efficiency.

[0021] (3) Incremental learning mechanism to support real-time updates: By introducing an incremental learning mechanism, the present invention can update the model when new data is input and preserve the expressive power of the old data as much as possible. This is done by randomly selecting a small amount of data from the old data set and mixing it with the new data set for training. This avoids the high computational and time costs of training the deep dynamic hash model using all the old data, ensuring that the training cost is reduced in the scenario of expanded data. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The deep hash network structure diagram of the present invention; Figure 2 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described below in conjunction with the accompanying drawings.

[0024] Example 1: See Figure 1 and Figure 2 ,A ship image retrieval method based on deep hashing includes the following steps; S1, establish the initial ship image database D 0 ; Obtain multiple ship images of different categories, preprocess and label each ship image to obtain samples, and all samples constitute the ship image database D 0 , the categories include seagoing vessels, container ships, three-five ships, civilian ships and law enforcement ships; S2, construct a deep hash network, including the ViT feature extraction modules M connected in sequence ViT and dynamic hash coding module M hash , where M ViT For D 0 Extract features from sample x and obtain the corresponding image features f(x), M hash Used to dynamically hash f(x) and output a real-valued hash code , and then converted into a binary hash code h(x); S3, with D 0 Train the deep hashing network to obtain the ship image retrieval model M 0 , use M 0 Right D 0 Generate binary hash codes for all samples in the search database; S4, updating the ship image retrieval model based on incremental learning, obtaining an updated model each time, and the updating method of the t-th incremental learning includes steps S41-S45; S41, obtaining the ship image database D of the tth incremental learning t , marked as a new database Dnew , t≥1, and when t=1, D t =D 0 ; from D 0 ~D t-1 Each category selects q samples to form the old database D old , mark the updated model after the t-1th time as M old ; The q value can be selected as needed, such as 30, 50, 80, etc.; S42, constructed to update M hash The total loss L total , including the cost function L stability And hash comparison loss L hash ; , , = , In the formula, x i D old The sample, x j D new Medium sample, , M hash Before and after the tth update, the sample x i The real-valued hash code of is the L2 norm, M hash After the tth update, the sample x j The real-valued hash code, S ij For sample x i and x j The similarity value of x i and x j If the categories are the same, then S ij =1, otherwise S ij =0,max is the maximum value operation, m is the boundary distance for distinguishing samples of different categories, and γ is the adjustment parameter of Lhash; S43, constructed to update M ViT The loss function L ViT ; , In the formula, , M ViT The corresponding samples x before and after the tth update i The image features of M ViT Adjustment parameters for the ability to remember old data, MViT In D new The cross entropy loss calculated above; S44, construct the total loss of the model L, L = L total +L ViT ; S45, D old , D new After merging, enter M old , to minimize the total model loss L training M ViT and M hash , get the tth updated model M t ; S5, use M t For D new Generate binary hash codes for the samples and store them in the retrieval database; S6, based on M t Conduct ship image retrieval; Get the ship image x to be queried query , by M t Generate a binary hash code, search in the search database, and output the search results.

[0025] Example 2: See Figure 1 and Figure 2 Based on Example 1, S1 specifically includes: unifying the resolution of the collected ship images to 256 256, get samples by category classification and labeling, then rotate each sample by several angles, each rotation forms a new sample, and the sample and the new sample together form the initial ship image database D 0 For example, there are 2000 ship images, which are divided into training set and test set by 1800 and 200 respectively. 1800 samples are obtained by classifying the 1800 ship images, and then the images are rotated by 90°, 180° and 270° respectively to obtain 5400 new samples. Therefore, the initial ship image database D 0 There are 7200 samples in total.

[0026] Example 3: See Figure 1 and Figure 2 , based on Example 1, in S2, M hash The real-valued hash code is obtained according to the following formula and binary hash code h(x); , , Where W h , b h M hash The initial bias matrix and initial weight matrix, is the hyperbolic tangent function, is a symbolic function, if ≥0, then ,otherwise , K is the hash code length.

[0027] Example 4: See Figure 1 and Figure 2 On the basis of Example 1, the ViT feature extraction module is composed of an image feature embedding module and a Transformer encoder module of the ViT model; the image feature embedding module is used to divide the sample x into several image blocks, and then flatten each image block into a one-dimensional vector, map it to a D-dimensional vector through a linear mapping layer and perform position encoding to obtain a position encoding vector; the Transformer encoder module includes multiple layers of serially connected Transformer encoders, which are used to input the position encoding vector and output image features f(x) after multiple feature extractions.

[0028] About the image feature embedding module: The image feature embedding module includes an image segmentation layer, a linear mapping layer, and a position embedding layer. The image segmentation layer is used to perform a segmentation operation, divide the sample x into several image blocks and flatten each image block into a one-dimensional vector. The linear mapping layer is used to map each one-dimensional vector into a D-dimensional vector. The position embedding layer is used to perform position encoding on each D-dimensional vector to obtain a position encoding vector.

[0029] About the Transformer encoder module: In the Transformer encoder module, each layer of the Transformer encoder includes a multi-head self-attention mechanism and a feedforward neural network, and the CLS label vector of the last layer of the Transformer encoder is used as the image feature f(x).

[0030] Example 5: See Figure 1 and Figure 2 Based on Example 1, in S45, each training M hash When based on L total Update the weight matrix and bias matrix of the hash function during backpropagation; , In the formula, , are the weight matrices before and after the update, respectively, and when t=1, , , are the bias matrices before and after the update, respectively, and when t=1, , is the learning rate.

[0031] Example 6: See Figure 1 and Figure 2 , based on Example 1, S6 specifically includes steps S61 to S64; S61, preset a distance threshold τ and a preset number of search results M; S62, use M t Generate x query The binary hash code h(x query )Calculate h(x query ) and the Hamming distance of all binary hash codes in the retrieval database; S63, sorting the samples in the search database in ascending order according to the Hamming distance, and generating a sequence table; S64, selecting the first M samples in the sequence list whose Hamming distance is less than τ as the search results.

[0032] Example 7: See Figure 1 and Figure 2 Based on Example 1, we conducted a comparative experiment example for the ship data retrieval scenario to verify the advantages of the present invention over the prior art in terms of accuracy and efficiency. In the experiment, a real ship image dataset was selected, which contained a large number of different types of ship images, such as seagoing ships, container ships, three-five ships, civilian ships, law enforcement ships, and other 22 categories, and the dataset was divided into 85% training set and 15% test set.

[0033] In the experiment, we selected the commonly used retrieval accuracy in the industry as the evaluation index, and compared the performance of the present invention with three other existing methods on these indicators. The experimental results of each model are shown in Table 1.

[0034] Table 1. Comparison of experimental results of image retrieval models Method_Hash code length (number of bits) 64-bit 128 bits 256 bits Traditional SIFT feature method 73.28% 74.2% 75.5% Traditional SURT feature method 75.3% 77.1% 78.4% Deep Hashing Network (CNN-based) 84.1% 86.5% 88.2% Method of the present invention 88.9% 91.2% 93.1% As can be seen from Table 1, as the length of the hash code increases, the retrieval accuracy of each method is improved, but the deep hash coding of the present invention always performs better than other methods at different hash code lengths. In particular, when the hash code length reaches 256 bits, the present invention can significantly improve the retrieval accuracy and achieve better results, while other methods still have a large gap in accuracy compared with the present invention.

[0035] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A ship image retrieval method based on deep hashing, characterized by: The steps include: S1, establish an initial ship image database D0; Obtain multiple ship images of different categories, pre-process and label each ship image to obtain samples, and all samples constitute the ship image database D0; S2, construct a deep hash network, including the ViT feature extraction modules M connected in sequence ViT and dynamic hash coding module M hash , where M ViT It is used to extract features of sample x in D0 and obtain the corresponding image features f(x), M hash Used to dynamically hash f(x) and output a real-valued hash code , and then converted into a binary hash code h(x); S3, use D0 to train the deep hash network to obtain the ship image retrieval model M0, use M0 to generate binary hash codes for all samples in D0, and store them in the retrieval database; S4, updating the ship image retrieval model based on incremental learning, obtaining an updated model each time, and the updating method of the t-th incremental learning includes steps S41-S45; S41, obtaining the ship image database D of the tth incremental learning t , marked as a new database D new , t≥1, and when t=1, D t =D0; from D0 to D t-1 Each category selects q samples to form the old database D old , mark the updated model after the t-1th time as M old ; S42, constructed to update M hash The total loss L total , including the cost function L stability And hash comparison loss L hash ; ， ， = ， In the formula, x i D old The sample, x j D new Medium sample, , M hash Before and after the tth update, the sample x i The real-valued hash code of is the L2 norm, M hash After the tth update, the sample x j The real-valued hash code, S ij For sample x i and x j The similarity value of x i and x j If the categories are the same, then S ij =1, otherwise S ij =0,max is the maximum value operation, m is the boundary distance for distinguishing samples of different categories, and γ is the adjustment parameter of Lhash; S43, constructed to update M ViT The loss function L ViT ; ， In the formula, , M ViT The corresponding samples x before and after the tth update i The image features of M ViT Adjustment parameters for the ability to remember old data, M ViT In D new The cross entropy loss calculated above; S44, construct the total loss of the model L, L = L total +L ViT ; S45, D old , D new After merging, enter M old , to minimize the total model loss L training M ViT and M hash , get the tth updated model M t ; S5, use M t For D new Generate binary hash codes for the samples and store them in the retrieval database; S6, based on M t Conduct ship image retrieval; Get the ship image x to be queried query , by M t Generate a binary hash code, search in the search database, and output the search results.

2. The ship image retrieval method based on deep hashing according to claim 1 is characterized in that: S1 specifically sets the resolution of the collected ship images to 256 256, samples are obtained by classification and labeling, and each sample is rotated by several angles, and each rotation forms a new sample, and the sample and the new sample together constitute the initial ship image database D0.

3. The ship image retrieval method based on deep hashing according to claim 1 is characterized in that: In S2, M hash The real-valued hash code is obtained according to the following formula and binary hash code h(x); ， ， Where W h , b h M hash The initial bias matrix and initial weight matrix, is the hyperbolic tangent function, is a symbolic function, if ≥0, then ,otherwise , K is the hash code length.

4. The ship image retrieval method based on deep hashing according to claim 1, characterized in that: The ViT feature extraction module consists of the image feature embedding module and the Transformer encoder module of the ViT model; The image feature embedding module is used to divide the sample x into several image blocks, then flatten each image block into a one-dimensional vector, map it into a D-dimensional vector through a linear mapping layer and perform position encoding to obtain a position encoding vector; The Transformer encoder module includes multiple layers of serially connected Transformer encoders, which are used to input a position encoding vector and output image features f(x) after multiple feature extractions.

5. The ship image retrieval method based on deep hashing according to claim 1 is characterized in that: In S45, each training M hash When based on L total Update the weight matrix and bias matrix of the hash function during backpropagation; ， In the formula, , are the weight matrices before and after the update, respectively, and when t=1, , , are the bias matrices before and after the update, respectively, and when t=1, , is the learning rate.

6. The ship image retrieval method based on deep hashing according to claim 1, characterized in that: S6 specifically includes: S61, preset a distance threshold τ and a preset number of search results M; S62, use M t Generate x query The binary hash code h(x query )Calculate h(x query ) and the Hamming distance of all binary hash codes in the retrieval database; S63, sorting the samples in the search database in ascending order according to the Hamming distance, and generating a sequence table; S64, selecting the first M samples in the sequence list whose Hamming distance is less than τ as the search results.

7. The ship image retrieval method based on deep hashing according to claim 1 is characterized in that: The categories mentioned include seagoing vessels, container ships, Type 35 ships, civilian ships and law enforcement vessels.

Images