An incremental image hash retrieval method based on feature enhancement and knowledge distillation

Abstract

The application relates to the technical field of image retrieval, and discloses an incremental image hash retrieval method based on feature enhancement and knowledge distillation, which comprises the following steps: acquiring a query image; using a first hash model to perform feature coding on the query image, to generate a query hash code corresponding to the query image, wherein the first hash model is obtained by performing incremental updating on a second hash model which has been pre-trained, using newly added training images and historical sampling images; determining a target image corresponding to the query image by using the Hamming distance between the query hash code and each hash code in an image database, wherein the target image is used for representing a retrieval result obtained by performing hash retrieval on the query image, and the image database is used for storing hash codes of all images in a first image category and a second image category. By implementing the method, the catastrophic forgetting of the model in the incremental updating process can be slowed down, and the precision and efficiency of image hash retrieval can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of image retrieval technology, specifically to an incremental image hashing retrieval method based on feature enhancement and knowledge distillation. Background Technology

[0002] Image hash retrieval is an efficient implementation method for large-scale image retrieval scenarios. It maps the high-dimensional real-valued feature vector corresponding to the query image into a compact binary hash code through a hash model, and then completes the image retrieval based on the Hamming distance between the query hash code and the hash codes already stored in the image database, thereby improving the retrieval speed while significantly reducing storage overhead.

[0003] However, most relevant hash retrieval algorithms are designed for static datasets, and can only train and retrieve hash models based on static datasets. In practical applications, image databases expand continuously over time, and the category distribution of newly added training images often differs significantly from that of existing images, i.e., concept drift occurs. If newly added training images under the newly added training image categories are used to train an existing hash model, it is highly likely that the hash model's encoding ability with existing image categories will rapidly decline, leading to a catastrophic forgetting problem and ultimately causing a significant drop in overall retrieval accuracy. Summary of the Invention

[0004] This invention provides an incremental image hash retrieval method based on feature enhancement and knowledge distillation to solve the catastrophic forgetting problem that occurs during incremental model updates in related technologies.

[0005] In a first aspect, the present invention provides an incremental image hash retrieval method based on feature enhancement and knowledge distillation. The method includes: acquiring a query image; using a first hash model to encode the query image to generate a query hash code corresponding to the query image, wherein the first hash model is obtained by incrementally updating a pre-trained second hash model using newly added training images and historical sampled images, wherein the first image category corresponding to the newly added training images is different from the second image category corresponding to the historical sampled images, and the second image category includes all image categories corresponding to all historical training stages; using the Hamming distance between the query hash code and each hash code in the image database to determine the target image corresponding to the query image, wherein the target image is used to represent the retrieval result obtained by hash retrieval of the query image, and the image database is used to store the hash codes of the first image category and all images under the second image category.

[0006] Secondly, the present invention provides an incremental image hash retrieval device based on feature enhancement and knowledge distillation. The device includes: a first acquisition module for acquiring a query image; a generation module for performing feature encoding on the query image using a first hash model to generate a query hash code corresponding to the query image, wherein the first hash model is obtained by incrementally updating a pre-trained second hash model using newly added training images and historical sampled images, wherein the first image category corresponding to the newly added training images is different from the second image category corresponding to the historical sampled images, and the second image category includes all image categories corresponding to all historical training stages; and a first determination module for determining a target image corresponding to the query image using the Hamming distance between the query hash code and each hash code in the image database, wherein the target image is used to represent the retrieval result obtained by hash retrieval of the query image, and the image database is used to store the hash codes of the first image category and all images under the second image category.

[0007] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the incremental image hash retrieval method based on feature enhancement and knowledge distillation described in the first aspect or any corresponding embodiment.

[0008] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the incremental image hash retrieval method based on feature enhancement and knowledge distillation described in the first aspect or any corresponding embodiment thereof.

[0009] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the incremental image hash retrieval method based on feature enhancement and knowledge distillation described in the first aspect or any corresponding embodiment thereof.

[0010] The incremental image hashing retrieval method based on feature enhancement and knowledge distillation provided in this embodiment includes image categories corresponding to all historical training stages before incrementally updating the second hashing model. Correspondingly, historical sampled images include image data corresponding to each second image category. By combining newly added training images and historical sampled images to incrementally update the second hashing model, the second hashing model can retain existing knowledge of the second image category while learning new knowledge of the first image category. This mitigates catastrophic forgetting during the incremental update process and effectively improves the accuracy and efficiency of image hashing retrieval. Attached Figure Description

[0011] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of an optional application scenario provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of the first process of the incremental image hash retrieval method based on feature enhancement and knowledge distillation provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the second process of the incremental image hash retrieval method based on feature enhancement and knowledge distillation provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the third process of the incremental image hash retrieval method based on feature enhancement and knowledge distillation provided in the embodiments of the present invention; Figure 5 This is a schematic diagram of the hash model structure provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the network structure of the feature extraction module provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of incremental update based on feature distillation according to an embodiment of the present invention; Figure 8 This is a schematic diagram of obtaining a first hash model by incrementally updating a second hash model based on newly added training images and historical sampled images, according to an embodiment of the present invention. Figure 9 This is a structural block diagram of an incremental image hash retrieval device based on feature enhancement and knowledge distillation provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0015] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0016] Image hash retrieval is an efficient implementation method for large-scale image retrieval scenarios. It uses a hash function (referred to as a hash model in this application) to map the high-dimensional real-valued feature vector corresponding to the query image into a compact binary hash code, and then completes the image retrieval based on the Hamming distance between the hash code corresponding to the query image (referred to as the query hash code in this application) and the hash codes already stored in the image database. This improves the retrieval speed while significantly reducing storage overhead.

[0017] Image hashing retrieval was initially data-agnostic, but with the development of machine learning and deep learning, research has increasingly focused on data-related image hashing algorithms. By mining image feature information and label information, the retrieval performance of image hashing retrieval has been continuously improved. However, the data in image databases tends to increase over time, and the category distribution of newly added training images often differs significantly from that of stored images, meaning that data often experiences concept drift. However, most data-related hashing retrieval algorithms are designed for static data, meaning that hash models can only be trained on fixed datasets. This approach can lead to catastrophic forgetting when dealing with continuous data streams in the real world. Although there is some research on online hashing for streaming data, these algorithms still assume that the data follows a fixed distribution and only process data online, thus failing to consider the existence of data concept drift.

[0018] For methods using deep neural networks as hashing models, training time is often affected by the training set, thus a subset of the data is typically used to learn the hash function. Even so, as the number of images in the image database continues to increase, simply expanding the training set leads to a linear increase in its size with the total number of images in the database. If the hashing model is trained using only image data from new categories, it will suffer catastrophic forgetting of older image data.

[0019] Incremental hashing is a method to solve this problem. The problem corresponding to incremental hashing can be defined as: assuming that over time, the images in the image database corresponding to the image hash retrieval system... Arriving in stages and batches. In the... The newly arrived image data in this phase is , This is the total number of images in this batch. It is a set of image categories in this batch of image data. yes A collection of image categories that appeared in the image database before the current phase. It is the set of all image categories that have appeared in the current stage, and the number of elements in this set is... It represents the total number of all image categories that have appeared at the current stage. It is image data The image category label vector, where each dimension indicates... Whether it belongs to this category is indicated by a value of 1 if it does and 0 if it does not. It is the label vector matrix of the new image data. A hash model exists for each stage. This is used to generate hash codes for the query image. The phase hash model is denoted as In the first During this phase, the incremental hash algorithm should receive new data. And combined with data from historical training phases and its corresponding tag information Hash code and the hash model during the historical training phase. The adaptive adjustment image hash retrieval system generates corresponding hash codes for new image data. and learn hash functions This allows for adaptation to an ever-increasing number of image categories, while the hash codes of existing image data in the image database remain unchanged. Specifically, the hash codes generated for new image data... It should still possess discriminative power, maintaining semantic similarity across all data in the image database. The learned hash function... We should adapt to changes in the distribution of image data, and while learning new knowledge, try to retain as much of the old knowledge we have already learned as possible. In the... In the online query phase, images in the image database are the first... The sum of the previous stages, the category of the query image received by the image hash retrieval system also comes from the previous stages. The image categories that appeared in each stage.

[0020] The goal of incremental hash retrieval is to design a hash retrieval algorithm and a set of hash retrieval system update strategies to update the hash model and generate hash codes for new image data when new image data is added. This enables the hash retrieval system to adapt well to changes in image categories and to retain old knowledge while learning new knowledge from new image data.

[0021] However, current research on incremental hashing methods is still insufficient.

[0022] Based on this, this application proposes an incremental image hash retrieval method based on feature enhancement and knowledge distillation. The hash center is generated by an incremental hash center generation algorithm, and then combined with a similarity loss function based on the hash center and a feature enhancement module to improve the hash model's ability to handle class imbalance. At the same time, sample replay and knowledge distillation are used to incrementally update the hash model to avoid the catastrophic forgetting problem that occurs during the incremental update process.

[0023] As an optional application scenario of this invention, such as Figure 1 As shown, this application scenario may include at least one electronic device and at least one server. Figure 1 The example shown includes an electronic device 110 and a server 120.

[0024] The electronic device 110 has a hash retrieval application 101 installed, and the user 130 can interact with the hash retrieval application 101 through the electronic device 110 and / or the access device of the electronic device 110.

[0025] For example, the hash retrieval application 101 can be any application that provides hash retrieval-related services. For instance, application 101 could be an application that provides hash retrieval functionality, which can be deployed within the hash retrieval application.

[0026] exist Figure 1 In the application scenario shown, if the hash retrieval application 101 is active, the electronic device 110 can display page 102 of the hash retrieval application 101. Page 102 may include various types of pages that the hash retrieval application 101 can provide, such as upload pages, loading pages, and search result display pages, etc.

[0027] In some implementations, electronic device 110 communicates with server 120 via network 140 to provide hash retrieval services for hash retrieval application 101.

[0028] In some implementations, electronic device 110 can specifically be a smartphone, tablet, laptop, PDA, or it can be a desktop computer, game console, smart TV, smart wearable device, in-vehicle terminal, VR (Virtual Reality) device, AR (Augmented Reality) device, etc. Moreover, electronic device 110 can also support any type of interface.

[0029] In some implementations, server 120 can be various types of computing systems or servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, standalone physical servers, server clusters, or distributed systems.

[0030] In some implementations, network 140 may be a wired or wireless network, examples of which include, but are not limited to, the Internet, intranets, local area networks, wide area networks, mobile communication networks, and combinations thereof.

[0031] According to an embodiment of the present invention, an incremental image hash retrieval method based on feature enhancement and knowledge distillation is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] This embodiment provides an incremental image hashing retrieval method based on feature enhancement and knowledge distillation, which can be used in electronic devices. Figure 2 This is a flowchart of an incremental image hashing retrieval method based on feature enhancement and knowledge distillation according to an embodiment of the present invention, as shown below. Figure 2 As shown, the process includes the following steps: Step S201: Obtain the query image.

[0033] The query image can be the input image for the image retrieval. Furthermore, the image category of the query image must be an image category that has already appeared in the image database. For example, if the query image category is "vegetables," then the image database may include images of vegetables, but may also include, but is not limited to, images of fruits, books, and other image categories.

[0034] In some cases, users can upload the query image to the hash retrieval application via its upload page. The electronic device can then respond to the user's upload and retrieve the query image.

[0035] Step S202: Use the first hash model to encode the features of the query image and generate the query hash code corresponding to the query image. The first hash model is obtained by incrementally updating the pre-trained second hash model using newly added training images and historical sampled images. The first image category corresponding to the newly added training image is different from the second image category corresponding to the historical sampled image, and the second image category includes the image categories corresponding to all historical training stages.

[0036] The hash code query here can be the hash code corresponding to the image. The hash code can be a string of binary numbers of 0 or 1 obtained by compressing an image using a hash model.

[0037] Newly added training images can be image data corresponding to the new training image categories added during the incremental update of the second hash model in this stage. Historical sampled images can be image data under the old image categories (i.e., the second image categories) corresponding to all historical training stages before the incremental update of the second hash model. In some cases, historical sampled images may not be all historical training images corresponding to all historical training stages; that is, they can be representative historical sampled images selected from all historical training images corresponding to all historical training stages, but the image categories of the historical sampled images are all the old image categories corresponding to all historical training stages.

[0038] A hash model is used to map input images into fixed-length hash codes, enabling compact representation and fast retrieval of image features. This hash model learns the content features and semantic information of images, converting high-dimensional image data into low-dimensional binary hash codes. This results in high similarity between hash codes corresponding to similar images and significant distinguishability between hash codes corresponding to different images.

[0039] For example, the second hash model can be a hash model that has been trained in three stages. Specifically, the initial hash model can be obtained through the first stage of training, the first initial hash model can be obtained through the second stage of training, and the second hash model can be obtained through the third stage of training.

[0040] In the first stage, the newly constructed hash model is trained using image data corresponding to the image category of fruit to obtain an initial hash model. At this point, the initial hash model can accurately identify the image data corresponding to the image category of fruit.

[0041] In the second stage, the newly added training images are image data corresponding to the image category of meat, and the historical sampled images are a small amount of image data corresponding to the image category of fruit. By combining the image data corresponding to the image category of meat and the small amount of image data corresponding to the image category of fruit, the initial hash model is incrementally updated to obtain the first initial hash model. At this time, the first initial hash model can accurately identify the image categories of meat and fruit.

[0042] For the third stage, the newly added training images are image data corresponding to the image category of "books", and the historical sampled images are a small amount of image data corresponding to the image category of "fruits" and a small amount of image data corresponding to the image category of "meat". Then, by combining the image data corresponding to the image category of "books", the small amount of image data corresponding to the image category of "fruits", and the small amount of image data corresponding to the image category of "meat", the first initial hash model is trained to obtain the second hash model. At this time, the second hash model can accurately identify the image categories of "books", "fruits", and "meat".

[0043] For this current stage, the newly added training images are image data corresponding to the image category of vegetables, and the historical sampled images are a small amount of image data corresponding to the image categories of fruits, meat, and images. Combining the image data corresponding to the image categories of vegetables, fruits, meat, and images, the second hash model is incrementally updated to obtain the first hash model. At this time, the first hash model can identify the image categories of vegetables, books, fruits, and meat.

[0044] Since the first hash model is obtained through continuous incremental updates, the newly constructed hash model, the initial hash model, the first initial hash model, the second hash model, and the first hash model all have the same model structure.

[0045] Step S203: Using the Hamming distance between the query hash code and each hash code in the image database, determine the target image corresponding to the query image. The target image is used to represent the retrieval result obtained by hash retrieval of the query image. The image database is used to store the hash codes of the first image category and all images under the second image category.

[0046] The image database can be used to store image data under all image categories and the corresponding hash codes. For example, in this stage, the first hash model can identify the image categories of vegetables, books, fruits, and meat. Therefore, the image database can store the image data corresponding to the image categories of vegetables, books, fruits, and meat.

[0047] In addition, the image data stored in the image database can also include newly added training images and historical sampled images. The hash code for newly added training images can be generated using the first hash model obtained through incremental updates, while the hash code for historical sampled images can be generated using the hash model of the corresponding stage.

[0048] The incremental image hashing retrieval method based on feature enhancement and knowledge distillation provided in this embodiment includes image categories corresponding to all historical training stages before incrementally updating the second hashing model. Correspondingly, historical sampled images include image data corresponding to each second image category. By combining newly added training images and historical sampled images to incrementally update the second hashing model, the second hashing model can retain existing knowledge of the second image category while learning new knowledge of the first image category. This mitigates catastrophic forgetting during the incremental update process and effectively improves the accuracy and efficiency of image hashing retrieval.

[0049] This embodiment provides an incremental image hashing retrieval method based on feature enhancement and knowledge distillation, which can be used in electronic devices. Figure 3 This is a flowchart of an incremental image hashing retrieval method based on feature enhancement and knowledge distillation according to an embodiment of the present invention, as shown below. Figure 3 As shown, the process includes the following steps: Step S301: Obtain the query image. For details, please refer to [link to relevant documentation]. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0050] Step S302: Use the first hash model to encode the features of the query image and generate the query hash code corresponding to the query image. The first hash model is obtained by incrementally updating the pre-trained second hash model using newly added training images and historical sampled images. The first image category corresponding to the newly added training image is different from the second image category corresponding to the historical sampled image, and the second image category includes the image categories corresponding to all historical training stages.

[0051] Specifically, the first hash model includes a feature processing module and a hash layer. The feature processing module includes a feature extraction module and a feature enhancement module. The input of the feature enhancement module is the output of the feature extraction module, the output of the feature enhancement module is the input of the hash layer, and the output of the hash layer is the output of the first hash model. The feature extraction module is used to extract the third feature corresponding to the query image, the feature enhancement module is used to enhance the third feature to obtain the fourth feature, and the hash layer is used to hash encode the fourth feature to obtain the query hash code.

[0052] For example, such as Figure 5 As shown, the feature extraction module, feature enhancement module, and hash layer are connected sequentially. The feature extraction module can be a pre-trained deep residual network ResNet34, and the network structure of ResNet34 can be as follows: Figure 6 As shown, ResNet34, excluding the last fully connected layer (fc1000), is used as a feature extraction module to extract 512-dimensional features from the input image, denoted as . And it can be expressed as .

[0053] This application can incrementally update the second hash model to obtain the first hash model by replaying historical sampled images. However, since the historical sampled images are a small number of sampled images corresponding to all historical image categories (i.e., the second image category), there is a class imbalance problem because the number of sampled images corresponding to the historical image category is small, while the number of training images corresponding to the newly added image category (i.e., the first image category) is large. In this case, the deep residual network ResNet34 can extract richer features from the first image category (which has many training images), but its ability to recognize historical image categories (also known as old image categories) is poor. To address this problem, this application adds a feature enhancement module after the feature extraction module. The feature enhancement module learns the importance of features and adjusts the features output by the feature extraction module according to the importance of the features. Enhancement is performed to obtain enhanced features. Finally, the enhanced features are used as input to the hash layer.

[0054] The hash layer is located after the feature enhancement module, and the hash layer can specifically be a fully connected layer. Its activation function can be Compared to And other activation functions, The output of the hash layer is closer to -1 or 1, which can reduce quantization error during the final binarization. dimensional real-valued vector , recorded as: ; in, Used to represent hash layers.

[0055] Due to the final hash code The values ​​of are discrete, therefore this application can use symbolic functions. For the output of the hash layer Quantization is performed to obtain the final hash code. , recorded as: .

[0056] For example, It can be done Input image Mapped to hash code ;in, Used to represent feature enhancement modules.

[0057] By setting a feature enhancement module after the feature extraction module, the feature enhancement module can learn the importance of the features output by the feature extraction module and enhance the features output by the feature extraction module according to the importance of the features. This can avoid the class imbalance problem caused by the small number of sampled images under the historical image category and the large number of newly added training images under the new image category.

[0058] In some optional implementations, the feature enhancement module includes a first fully connected layer, a second fully connected layer, and a weight calculation module; the output of the feature extraction module is the input of the first and second fully connected layers; the outputs of the first and second fully connected layers are the input of the weight calculation module; the first fully connected layer is used to map the third feature to a first component feature; the second fully connected layer is used to map the third feature to a second component feature; the weight calculation module is used to perform dot product attention calculation on the first and second component features to determine the target weight, and the target weight is used to achieve weighted fusion of the first and second component features to obtain the fourth feature.

[0059] For example, such as Figure 5 As shown, a parallel fully connected layer is introduced after the feature extraction module. and and use As and The activation function is used to obtain the fully connected layer. The first component feature of the output third feature in the feature space and fully connected layer The output third feature is the second component feature in the feature space. Then, the weight calculation module uses a dot product attention mechanism in this feature space to calculate the importance of the first and second component features. Then, based on the importance of the features Features of the first component The first weighted feature is obtained by weighting. And according to the importance of the features Features of the second component The second weighted feature is obtained by weighting. Finally, the first weighted feature and the second weighted feature are added together to obtain the enhanced feature. .

[0060] The above process can be specifically recorded as: .

[0061] By enhancing the third feature output by the feature extraction module through the feature enhancement module, the hash model can focus more on learning important features. For historical sampled images, although the amount of data in each second image category is small, the improved learning ability of the hash model to identify the important features of that second image category further enhances its ability to solve the class imbalance problem during incremental updates.

[0062] To better preserve old knowledge in the first hash model obtained through incremental updates, this application incrementally updates the second hash model using knowledge distillation. Exemplarily, a teacher-student model can be used to transfer capabilities to the second hash model. Here, the teacher model is the knowledge outputter, and the student model is the knowledge receiver. Depending on the transfer method, knowledge distillation can be divided into goal-based distillation (also known as soft-target distillation or Logits method distillation) and feature-based distillation. Goal-based distillation allows the student model to learn the teacher model's sofmax output in the classification layer, while feature-based distillation allows the student model to learn the teacher model's output in certain intermediate layers of the network. To improve the scalability of the hash model, and since the model structure of this application does not involve classification layers related to image categories, no expansion of the model structure is needed during category increment; only model parameters need to be updated. Therefore, goal-based distillation cannot be used. Accordingly, this application employs a feature-based knowledge distillation algorithm to improve the hash model's ability to preserve old knowledge.

[0063] Specifically, the process of incrementally updating the second hash model using newly added training images and historical sampled images to obtain the first hash model includes: Step S3021: Construct a third hash model using the second hash model.

[0064] Here, the second hash model can be used to initialize the third hash model. For example, the second hash model can be directly used as the third hash model, meaning the third hash model continues to use the model structure and parameters of the second hash model. In the early stages of incremental updates in this phase, the model parameters of the third hash model and the second hash model can be the same. However, as the incremental updates progress, the third hash model is gradually adjusted, and correspondingly, the model parameters of the third hash model and the second hash model will gradually become different.

[0065] Step S3022: Input the newly added training image and the historical sampled image into the third hash model to obtain the predicted real value vector output by the third hash model and the first feature output by the feature processing module in the third hash model.

[0066] The predicted real-valued vector here can be the result of processing by the fully connected layer in the hash layer of the third hash model, but without the sign function. The quantized real-valued vectors. Furthermore, the third hash model processes newly added training images to obtain predicted real-valued vectors corresponding to the newly added training images; the third hash model processes historical sampled images to obtain predicted real-valued vectors corresponding to the historical sampled images, that is, one predicted real-valued vector corresponds to one newly added training image or one historical sampled image.

[0067] Here, the first feature can be the feature processed by the feature enhancement module in the third hash model, that is, the feature before the hash layer processing in the third hash model. Moreover, the feature processing module (i.e., the feature extraction module and the feature enhancement module) in the third hash model processes the newly added training image to obtain the first feature corresponding to the newly added training image; the third hash model processes the historical sampled image to obtain the first feature corresponding to the historical sampled image, that is, one first feature corresponds to one newly added training image or one historical sampled image.

[0068] Step S3023: Input the newly added training image and the historical sampled image into the second hash model to obtain the second feature output by the feature processing module in the second hash model.

[0069] The second feature here can be the feature after processing by the feature enhancement module in the second hash model, that is, the feature before hash layer processing in the second hash model. Moreover, the feature processing module (i.e., the feature extraction module and the feature enhancement module) in the second hash model processes the newly added training image to obtain the second feature corresponding to the newly added training image; the third hash model processes the historical sampled image to obtain the second feature corresponding to the historical sampled image, that is, one second feature corresponds to one newly added training image or one historical sampled image.

[0070] Step S3024: Determine the center similarity loss value by using the distance between the predicted real value vector and the first label value corresponding to the predicted real value vector.

[0071] Here, the first label value corresponding to the predicted real-valued vector can be the label value of the image (new training image or historical sampled image) corresponding to the predicted real-valued vector. For example, the first label value can include a first predicted value and the target hash center corresponding to the predicted real-valued vector. As a specific example, the first predicted value can be 1. The target hash center corresponds to the image category of the sampled image corresponding to the predicted real-valued vector. That is, one image category corresponds to one hash center.

[0072] In some optional implementations, step S3024 above includes: Step a1: Determine the quantization loss value using the Manhattan distance between the predicted real-valued vector and the first preset value.

[0073] Step a2: Determine the similarity loss value using the Euclidean distance between the predicted real-valued vector and the target hash center. The target hash center is used to represent the hash center corresponding to the image category to which the first sampled image belongs. The first sampled image is used to represent the newly added training image or historical sampled image corresponding to the predicted real-valued vector.

[0074] Step a3: Determine the center similarity loss value based on the fusion result of the quantization loss value and the similarity loss value.

[0075] For hash models, the ultimate goal is to generate discrete hash codes. However, due to the sign function in the hash layer... Since hash codes are not differentiable everywhere and their differentiability is zero everywhere, the gradient vanishing problem occurs during the training of hash models, making backpropagation unsuitable for training them. Therefore, during training, the discrete constraints on the hash codes can be relaxed.

[0076] For example, using predicted real-valued vectors Instead of predicting hash codes Moreover, in order to reduce Quantified as Time error, limit The output of each dimension is closer to 1 or -1, therefore the quantization loss value can be expressed as: ,in, Used to represent the quantization loss value, Used to indicate Manhattan distance, This is used to represent the predicted real-valued vector corresponding to the i-th image (either a newly added training image or a historically sampled image), where N represents the total number of newly added training images and historically sampled images. Used to represent K-dimensional Hamming space.

[0077] For image category The i-th image The relaxed hash code (i.e., the predicted real-valued vector) generated by the third hash model is: This image category The corresponding hash center is (That is, the target hash center shown above). To make... and Minimize the distance between them and the hash center corresponding to other image categories. To maximize the distance between them, the similarity loss value can be expressed as: ,in, Used to represent the similarity loss value.

[0078] As a concrete example, the quantization loss value and the similarity loss value can be weighted and fused to obtain the first loss. For example, , Used to represent the weight of the quantization loss value This is used to represent the center similarity loss value, which is also the total loss value in the non-incremental update stage.

[0079] Furthermore, the non-incremental update phase of the hash model does not involve a knowledge distillation-based learning process. Therefore, for the hash model in the non-incremental update phase, its loss value can include only the center similarity loss value. In other words, for the hash model in the non-incremental update phase, its loss function can consist of only the quantization loss function and the similarity loss function.

[0080] By using quantization loss, each predicted real-value vector in the output can be constrained to approach a first preset value or its opposite. By using similarity loss, the similarity between the relaxed hash code of each sampled image and its image category can be preserved as much as possible. Thus, the weighted fusion of quantization loss and similarity loss can more accurately quantify the loss that occurs during model processing, thereby enabling the model to converge quickly.

[0081] Step S3025: Determine the knowledge distillation loss value using the Euclidean distance between the first feature and the second feature.

[0082] like Figure 7 As shown, in After obtaining the second hash model in the first stage, the second hash model extracts... Knowledge of this stage, let's denote the second hash model at this point as In the first In the current stage, a batch of images belonging to new image categories are added to the image database. Therefore, the second hash model needs to be incrementally updated so that the first hash model can recognize the new image categories. This is to reduce the model's dependence on the new image categories during learning. Forgetting of knowledge at a certain stage can The second hash model of the stage is used as the teacher model, The initial third hash model is used as the student model, and feature-based knowledge distillation is used to constrain the images input to the third hash model. Enhanced features output by the t-stage feature processing module The image input to the second hash model And the enhanced features output by the t-1 stage feature processing module Proximity. Based on this, the knowledge distillation loss value can be expressed as: ,in, Used to represent the enhanced features output by the t-stage feature processing module. , Enhanced features used to represent the output of the feature processing module at stage t-1 , This is used to represent the set of historical sampled images shown above, which is the memory pool mentioned later. Used to represent the set corresponding to newly added training images. Used to represent the knowledge distillation loss value. Used to represent Euclidean distance.

[0083] Step S3026: Using the fusion result of the center similarity loss value and the knowledge distillation loss value, update the model parameters of the third hash model to obtain the first hash model.

[0084] For the incremental update phase of the hash model, the loss value includes not only the center similarity loss value shown above, but also the knowledge distillation loss value. Accordingly, the total loss value for the incremental update phase can be expressed as: ,in, Used to represent the weight of the knowledge distillation loss value. Used to represent the total loss value during the incremental update phase.

[0085] As a specific example, such as Figure 8 As shown, the incremental update process based on feature enhancement and knowledge distillation can include: For the non-incremental update stage, the images stored in the image database during this stage can be used as input to the newly constructed hash model. The feature processing module of the newly constructed hash model extracts and enhances features from the input images to obtain enhanced features. These enhanced features are then input into the hash layer to obtain the relaxed hash code (i.e., the predicted real-value vector shown above) output by the hash layer. Subsequently, the quantization loss and similarity loss are calculated by combining the relaxed hash code with the corresponding label values. Finally, the newly constructed hash model is trained using the center similarity loss obtained by weighted fusion of the quantization loss and similarity loss values ​​to obtain the initial hash model. After obtaining the initial hash model, the images in the image database of this stage are filtered to obtain historical sampling images. These filtered historical sampling images are used to form... Figure 8 The memory pool shown. And as... Figure 8 As shown, the initial hashing module can identify image categories such as birds, cars, and books.

[0086] In the incremental update phase, a new batch of images arrives, and the categories of these new images are different from those in the historical training phase. Based on this, using the Hadamard matrix and incremental hash center generation algorithm described earlier, new hash centers are generated for the new image categories. Then, the memory pool formed in the historical training phase and the new images in this phase are used together as input to the initial hash model, and the initial hash model is trained using the center similarity loss and knowledge distillation loss described earlier, resulting in the incrementally updated hash model, as shown in the first initial hash model. And as... Figure 8 As shown, the first initial hash module can identify not only image categories such as birds, cars, and books, but also dog image categories.

[0087] Step S3027: Use the first hash model to perform feature encoding on the query image to generate the query hash code corresponding to the query image. Please refer to the previous text for details.

[0088] The hash center and the hash code are the same. A vector in Hamming space, where each image category has a corresponding hash center in Hamming space. hash centers There is sufficient distance between each pair of elements, that is:

[0089] in, Used to represent Hamming distance It is the preset distance between two hash centers, also known as the minimum distance. According to the Gilbert-Varshamov lower bound theory in coding theory, and The following relationship exists between them: .

[0090] in, It is a combination number. When If the information is known, then iteration is possible. Calculate the maximum value that satisfies the above formula. This is the minimum distance between the two hash centers.

[0091] Due to the limitations of discrete constraints, solutions satisfying the above formulas are not easily obtained. Considering the need for compressed storage of hash codes in computers, the dimension of hash codes is often chosen to be... The power of. When the dimension of Hamming space. When taking powers of 2, the hash center can be quickly generated using the Hadamard matrix.

[0092] The Hadak matrix is ​​a special type of square matrix, for Hadamard matrix of order: (1) The elements in the matrix can only be 0 or 1; (2) It can only be a multiple of 1, 2, or 4; (3) Any row or column in the Hadamah is orthogonal to each other.

[0093] Based on the relationship between Hamming distance and inner product, the above formula... Desirable .when When taking powers of 2, the Hadamard matrix can be generated iteratively using the following formula: .

[0094] After obtaining the Hadak code matrix Each column in the algorithm can be used as a hash center to assign to an image category. However, the above method only works in... Generated in Wiehamming space There are 1 hash center that cannot handle image categories with more than 100 hash centers. The training set is used. In scenarios with incremental image categories, the number of image categories is constantly increasing, which may exceed the number of hash centers constructed from the Hadamard matrix. Based on this, this application proposes an incremental hash center generation method.

[0095] In some optional implementations, the process of determining the hash center corresponding to the first image category includes: Step b1: Determine whether the number of current hash centers is greater than the second preset value. The current hash centers include at least the hash centers corresponding to the second image category, and one image category corresponds to one hash center.

[0096] Step b2: If not, select an unused column from the pre-constructed Hadamard matrix as the hash center corresponding to the first image category.

[0097] Step b3, if yes, then determine the hash center corresponding to the first image category using the first relation, which is: , in, Used to indicate the current number of hash centers. Used to represent the i-th current hash center, Used to represent the hash center corresponding to the (m+1)th first image category Used to represent Hamming distance Used to indicate a preset distance. Used to represent the dimension of Hamming space.

[0098] For updates to a hash model based on a hash center, when a new image category appears, a new hash center must first be generated for the new image category before subsequent incremental updates to the hash model can be performed. Wienham space can be generated using the Hadak code matrix. One hash center, Each hash center can be assigned to There are [number] image categories. If the current number of image categories is greater than [number]... In this case, hash centers can be generated incrementally.

[0099] For example, suppose there is already a set of hash centers. In order to obtain a new hash center The distance between the vectors in the new hash center and the existing hash centers should be large enough, i.e.:

[0100] Based on the relationship between Hamming distance and inner product, the above formula can be equivalently expressed as:

[0101] As a concrete example, the above formula can be solved using the lp-box integer programming algorithm. According to the lp-box integer programming algorithm, binary constraints... It can be equivalent to These are two continuous constraints. Therefore, It can be equivalent to

[0102] Its corresponding augmented Lagrangian function is

[0103] Optimization using the alternating direction multiplier method Gradient descent is used when updating each variable; where, Used to represent an m-dimensional vector of all 1s; p is used to represent a k-dimensional vector of all 1s; p is used to represent The order of the norm is taken as p=2 in the experiment.

[0104] renew ,fixed .

[0105]

[0106] in, Used to represent The identity matrix, and Used to represent the dimension of Hamming space.

[0107] renew ,fixed .

[0108]

[0109] renew ,fixed .

[0110]

[0111] The above describes an incremental method for generating hash centers when adding a new image category. Used to represent the penalty parameter in the augmented Lagrange function.

[0112] When the number of hash centers does not exceed the second preset value, unused columns are directly selected from the Hadamard matrix as hash centers for new categories. This ensures good orthogonality and uniform distribution among hash centers of different categories, making each category's hash centers distinct and improving the discriminative power of hash encoding. When the number of hash centers exceeds the second preset value, new hash centers are dynamically calculated based on the first relation, combined with Hamming distance and preset distance. This ensures sufficient distinguishability between new and existing hash centers, avoiding confusion caused by excessive hash center density due to an increase in the number of categories.

[0113] Step S303: Using the Hamming distance between the query hash code and each hash code in the image database, the target image corresponding to the query image is determined. The target image represents the retrieval result obtained by hash retrieval of the query image. The image database stores the hash codes of all images under the first image category and the second image category. For details, please refer to... Figure 2 Step S203 of the illustrated embodiment will not be described again here.

[0114] The incremental image hashing retrieval method based on feature enhancement and knowledge distillation provided in this embodiment, during the incremental update phase, comprehensively and accurately quantifies the overall loss of the model during the incremental learning process by fusing the center similarity loss value and the knowledge distillation loss value. Based on the total loss obtained from this fusion, the model parameters of the third hashing model are updated and optimized, enabling the model to quickly and stably inherit old knowledge while learning knowledge of new image categories. This accelerates model convergence and improves model training efficiency and stability.

[0115] This embodiment provides an incremental image hashing retrieval method based on feature enhancement and knowledge distillation, which can be used in electronic devices. Figure 4 This is a flowchart of an incremental image hashing retrieval method based on feature enhancement and knowledge distillation according to an embodiment of the present invention, as shown below. Figure 4 As shown, the process includes the following steps: Step S401: Obtain the query image. For details, please refer to [link to relevant documentation]. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0116] Step S402: The query image is feature-encoded using a first hash model to generate a query hash code corresponding to the query image. The first hash model is obtained by incrementally updating a pre-trained second hash model using newly added training images and historical sampled images. The first image category corresponding to the newly added training image is different from the second image category corresponding to the historical sampled images, and the second image category includes image categories corresponding to all historical training stages. For details, please refer to [link to details]. Figure 2 Step S202 of the illustrated embodiment will not be described again here.

[0117] Step S403: Using the Hamming distance between the query hash code and each hash code in the image database, the target image corresponding to the query image is determined. The target image represents the retrieval result obtained by hash retrieval of the query image. The image database stores the hash codes of all images under the first image category and the second image category. For details, please refer to [link to details]. Figure 2 Step S203 of the illustrated embodiment will not be described again here.

[0118] Replaying historical sampled images allows for the preservation of a small number of representative sampled images for each image category after the model has learned from the images at each stage, which can then be used for incremental updates of the model in subsequent stages. This method significantly reduces the hash model's tendency to forget old knowledge by retaining a small number of sampled images. For example, a limited storage space (called a memory pool) can be used to store the representative sampled images. After the incremental update of the model at each stage, the sampled images in the memory pool are also updated, allowing the updated sampled images in the memory pool to be used again for incremental updates of the first hash model along with the new sampled images for the next stage.

[0119] Step S404: Obtain the first number corresponding to the historical sampled images, and the total number of categories corresponding to the first image category and the second image category.

[0120] The first number corresponding to the historical sampled images here can be the total capacity of the memory pool as shown above. As a specific example, the total capacity M of the memory pool can be fixed. Here, the total number of categories corresponding to the first and second image categories can be the total number of all image categories that have appeared so far. As a specific example, the total number of categories can be expressed as... .

[0121] Step S405: Using the ratio of the first number to the total number of categories, determine the third number of downsampled images for each image category.

[0122] For example, a representative sampled image corresponding to each image category can be represented as follows: Since the historical sampled images do not include the first image category corresponding to the newly added training images, the number of sampled images corresponding to each second image category in the historical sampled images must be greater than [a certain number]. Therefore, historical sampled images can be deleted to ensure that each second image category corresponds to... One sampled image.

[0123] Step S406: Select a third number of second sampled images from the newly added training images for each first image category.

[0124] Since the historical sampling images do not include newly added training images under the first image category, the newly added training images can be filtered to ensure that each image under the first image category corresponds to a certain category in the updated historical sampling images. One sampled image.

[0125] In some alternative implementations, step S406 includes: Step c1: For each first image category, select all category sampling images corresponding to the first image category from the newly added training images.

[0126] Step c2: Input the category sampling image into the first hash model to obtain the fifth feature output by the feature processing module in the first hash model.

[0127] Step c3: Average the fifth feature corresponding to all sampled images of all categories to obtain the sixth feature.

[0128] Step c4: Using the Euclidean distance nearest neighbor principle between the fifth and sixth features, the second sampled image is selected from the category sampled images.

[0129] When there are multiple first image categories, we can first select all sampled images corresponding to each first image category from the newly added training images. For example, the first image categories specifically include the image category of broccoli, the image category of spinach, and the image category of eggplant. Accordingly, we can first select all sampled images of broccoli corresponding to the image category of broccoli from the newly added training images, select all sampled images of spinach corresponding to the image category of spinach from the newly added training images, and select all sampled images of eggplant corresponding to the image category of eggplant from the newly added training images.

[0130] In some cases, if The specific value is 3. If there are three sampled images of broccoli corresponding to the image category, then all sampled images of broccoli can be determined as the second sampled images. If there are fewer than three sampled images of spinach corresponding to the image category, then all sampled images of spinach can also be determined as the second sampled images. If there are more than three sampled images of eggplant corresponding to the image category, then the relationship between each eggplant-related sampled image and the average features of all eggplant-related sampled images is used to select representative eggplant-related sampled images from all eggplant-related sampled images as the second sampled images.

[0131] By using the Euclidean distance between the fifth and sixth features corresponding to each category of sampled images, representative second sampled images can be selected from each category of sampled images. Such high-quality second sampled images can avoid catastrophic forgetting when incrementally updating the first hash model in the next stage.

[0132] Step S407: Update the historical sampling image using all the second sampling images to obtain the updated historical sampling image. The updated historical sampling image is used to incrementally update the first hash model in the next stage.

[0133] In some alternative implementations, step S407 includes: Step d1: Delete a fourth number of historical sampled images from each second image category in the historical sampled images to obtain the deleted historical sampled images. The fourth number is determined based on the difference between the third and fifth numbers, and the fifth number is the number of historical sampled images in each second image category.

[0134] Step d2: Using the combination of all second-sampled images and the deleted historical sampled images, the updated historical sampled images are obtained.

[0135] By first deleting historical sampled images and then combining the second sampled image with the deleted historical sampled images, it can be ensured that the image categories of the updated historical sampled images include both the first and second image categories. In addition, the number of updated historical sampled images is kept at a certain level to avoid disrupting the model's learning of newly added training images due to an excessive number of historical sampled images.

[0136] As a specific example, the incremental update algorithm for the sampled images in the memory pool after stage t can be: from the newly added training images Select all category sampling images corresponding to the i-th first image category. The sampled images X of each category are then input into the first hash model to obtain the fifth feature of each category of sampled images output by the feature processing module. ,in, Used to represent the first hash model The output before the hash layer; then through... Determine the sixth feature corresponding to the i-th first image category. ; and then through The second sampled image is selected from all sampled images corresponding to the i-th first image category. ,in, Used to indicate that they have been selected. The features of the second sampled image. Then, the second sampled image corresponding to each first image category can be added to the memory pool (that is, the set of historical sampled images) to obtain the updated memory pool; finally, a fourth number of historical sampled images are deleted from each second image category in the updated memory pool, wherein the deleted fourth number of historical sampled images can be the historical sampled images that are ranked relatively late in the corresponding second image category.

[0137] The incremental image hashing retrieval method based on feature enhancement and knowledge distillation provided in this embodiment determines a third quantity for each image category based on the ratio of a first quantity to the total number of categories. Then, based on this third quantity, second sampled images are selected from newly added training images for each first image category. All second sampled images are then used to update the historical sampled images. This balances the number of sampled images for each image category, preventing the model from becoming biased towards a particular image category due to excessive differences in the number of sampled images between categories, and ensuring the hash model's balanced learning of new and old categories. Simultaneously, dynamically selecting representative sampled images according to a ratio controls the size of the historical training set while ensuring training effectiveness, reducing the computational and storage overhead of subsequent incremental updates.

[0138] This embodiment also provides an incremental image hashing retrieval device based on feature enhancement and knowledge distillation. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0139] This embodiment provides an incremental image hashing retrieval device based on feature enhancement and knowledge distillation, such as... Figure 9 As shown, it includes: The first acquisition module 910 is used to acquire the query image.

[0140] The generation module 920 is used to encode the features of the query image using the first hash model to generate the query hash code corresponding to the query image. The first hash model is obtained by incrementally updating the pre-trained second hash model using newly added training images and historical sampled images. The first image category corresponding to the newly added training image is different from the second image category corresponding to the historical sampled image, and the second image category includes the image categories corresponding to all historical training stages.

[0141] The first determining module 930 is used to determine the target image corresponding to the query image by using the Hamming distance between the query hash code and each hash code in the image database. The target image is used to represent the retrieval result obtained by hash retrieval of the query image. The image database is used to store the hash codes of the first image category and all images under the second image category.

[0142] In some optional implementations, the second hash model includes a feature processing module; the apparatus for incrementally updating the second hash model using newly added training images and historical sampled images to obtain the first hash model includes: A building block for constructing a third hash model using the second hash model.

[0143] The first input module is used to input newly added training images and historical sampled images into the third hash model to obtain the predicted real-value vector output by the third hash model and the first feature output by the feature processing module in the third hash model.

[0144] The second input module is used to input newly added training images and historical sampled images into the second hash model to obtain the second feature output by the feature processing module in the second hash model.

[0145] The second determining module is used to determine the center similarity loss value by using the distance between the predicted real value vector and the first label value corresponding to the predicted real value vector.

[0146] The third determination module is used to determine the knowledge distillation loss value by utilizing the Euclidean distance between the first feature and the second feature.

[0147] The first update module is used to update the model parameters of the third hash model by using the fusion result of the center similarity loss value and the knowledge distillation loss value, so as to obtain the first hash model.

[0148] In some optional implementations, the first tag value includes a first preset value and a target hash center; the second determining module includes: The first determining unit is used to determine the quantization loss value by using the Manhattan distance between the predicted real value vector and the first preset value.

[0149] The second determining unit is used to determine the similarity loss value by using the Euclidean distance between the predicted real value vector and the target hash center. The target hash center is used to represent the hash center corresponding to the image category to which the first sampled image belongs. The first sampled image is used to represent the newly added training image or historical sampled image corresponding to the predicted real value vector.

[0150] The third determining unit is used to determine the center similarity loss value based on the fusion result of the quantization loss value and the similarity loss value of the second sub-loss value.

[0151] In some optional implementations, the device for determining the hash center corresponding to the first image category includes: The fourth determining module is used to determine whether the number of current hash centers is greater than the second preset value. The current hash centers include at least the hash centers corresponding to the second image category, and one image category corresponds to one hash center.

[0152] The selection module is used to select unused columns from the pre-constructed Hadamard matrix as hash centers for the first image category if no other options are found.

[0153] The fifth determining module is used to determine the hash center corresponding to the first image category if the condition is met, using the first relational expression: , in, Used to indicate the current number of hash centers. Used to represent the i-th current hash center, Used to represent the hash center corresponding to the (m+1)th first image category Used to represent Hamming distance Used to indicate a preset distance. Used to represent the dimension of Hamming space.

[0154] In some optional implementations, the first hash model includes a feature processing module and a hash layer. The feature processing module includes a feature extraction module and a feature enhancement module. The input of the feature enhancement module is the output of the feature extraction module, the output of the feature enhancement module is the input of the hash layer, and the output of the hash layer is the output of the first hash model. The feature extraction module is used to extract the third feature corresponding to the query image, the feature enhancement module is used to enhance the third feature to obtain the fourth feature, and the hash layer is used to hash encode the fourth feature to obtain the query hash code.

[0155] In some optional implementations, the feature enhancement module includes a first fully connected layer, a second fully connected layer, and a weight calculation module; the output of the feature extraction module is the input of the first and second fully connected layers; the outputs of the first and second fully connected layers are the input of the weight calculation module; the first fully connected layer is used to map the third feature to a first component feature; the second fully connected layer is used to map the third feature to a second component feature; the weight calculation module is used to perform dot product attention calculation on the first and second component features to determine the target weight, and the target weight is used to achieve weighted fusion of the first and second component features to obtain the fourth feature.

[0156] In some alternative embodiments, the device further includes: The second acquisition module is used to acquire the first number corresponding to the historical sampled images, and the total number of categories corresponding to the first image category and the second image category.

[0157] The sixth determining module is used to determine the third number of downsampled images for each image category by using the ratio of the first number to the total number of categories.

[0158] The filtering module is used to filter a third number of second sampled images from the newly added training images for each first image category.

[0159] The second update module is used to update the historical sampled images using all the second sampled images to obtain updated historical sampled images. The updated historical sampled images are used to incrementally update the first hash model in the next stage.

[0160] In some alternative implementations, the filtering module includes: The first filtering unit is used to filter out all category sample images corresponding to the first image category from the newly added training images for each first image category.

[0161] The input unit is used to input the category sampled image into the first hash model to obtain the fifth feature output by the feature processing module in the first hash model.

[0162] The processing unit is used to average the fifth feature corresponding to all categories of sampled images to obtain the sixth feature.

[0163] The second filtering unit is used to filter out the second sampled image from the category sampled images by using the Euclidean distance nearest neighbor principle between the fifth feature and the sixth feature.

[0164] In some alternative implementations, the second update module includes: The deletion unit is used to delete a fourth number of historical sampled images from each second image category in the historical sampled images to obtain deleted historical sampled images. The fourth number is determined based on the difference between the third number and the fifth number, and the fifth number is the number of historical sampled images in each second image category.

[0165] The combination unit is used to obtain an updated historical sampled image by combining all the second sampled images with the deleted historical sampled images.

[0166] The incremental image hash retrieval device based on feature enhancement and knowledge distillation provided in this embodiment of the invention can execute the incremental image hash retrieval method based on feature enhancement and knowledge distillation provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method. Further functional descriptions of the above modules and units are the same as in the corresponding embodiments described above, and will not be repeated here.

[0167] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0168] The following is a detailed reference. Figure 10The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 1001, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 1002 or a program loaded from memory 1008 into random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the electronic device. The processor 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0169] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; memory devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic devices to exchange data via wireless or wired communication with other devices. Although Figure 10 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0170] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 1009, or installed from a memory 1008, or installed from a ROM 1002. When the computer program is executed by the processor 1001, it performs the functions defined in the incremental image hash retrieval method based on feature enhancement and knowledge distillation according to embodiments of the present invention.

[0171] Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0172] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the incremental image hashing retrieval method based on feature enhancement and knowledge distillation shown in the above embodiments is implemented.

[0173] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0174] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. An incremental image hashing retrieval method based on feature enhancement and knowledge distillation, characterized in that, The method includes: Retrieve the query image; The query image is feature-encoded using a first hash model to generate a query hash code corresponding to the query image. The first hash model is obtained by incrementally updating a pre-trained second hash model using newly added training images and historical sampled images. The first image category corresponding to the newly added training image is different from the second image category corresponding to the historical sampled image, and the second image category includes the image categories corresponding to all historical training stages. The target image corresponding to the query image is determined by using the Hamming distance between the query hash code and each hash code in the image database. The target image is used to represent the retrieval result obtained by hash retrieval of the query image. The image database is used to store the hash codes of the first image category and all images under the second image category.

2. The method according to claim 1, characterized in that, The second hash model includes a feature processing module; The process of incrementally updating the second hash model using the newly added training images and the historical sampled images to obtain the first hash model includes: Using the second hash model, construct the third hash model; The newly added training image and the historical sampled image are input into the third hash model to obtain the predicted real value vector output by the third hash model and the first feature output by the feature processing module in the third hash model; The newly added training image and the historical sampled image are input into the second hash model to obtain the second feature output by the feature processing module in the second hash model; The center similarity loss value is determined by using the distance between the predicted real value vector and the first label value corresponding to the predicted real value vector; The knowledge distillation loss value is determined using the Euclidean distance between the first feature and the second feature; The model parameters of the third hash model are updated using the fusion result of the center similarity loss value and the knowledge distillation loss value to obtain the first hash model.

3. The method according to claim 2, characterized in that, The first label value includes a first preset value and a target hash center; the step of determining the center similarity loss value by utilizing the distance between the predicted real value vector and the first label value corresponding to the predicted real value vector includes: The quantization loss value is determined using the Manhattan distance between the predicted real-valued vector and the first preset value; The similarity loss value is determined by using the Euclidean distance between the predicted real value vector and the target hash center. The target hash center is used to represent the hash center corresponding to the image category to which the first sampled image belongs. The first sampled image is used to represent a newly added training image or a historical sampled image corresponding to the predicted real value vector. The center similarity loss value is determined based on the fusion result of the quantization loss value and the similarity loss value.

4. The method according to claim 3, characterized in that, The process of determining the hash center corresponding to the first image category includes: Determine whether the number of current hash centers is greater than a second preset value. The current hash centers include at least the hash centers corresponding to the second image category, and one hash center corresponds to one image category. If not, then select an unused column from the pre-constructed Hadamard matrix as the hash center corresponding to the first image category; If so, the hash center corresponding to the first image category is determined by the first relation, which is: ， in, Used to indicate the number of current hash centers Used to represent the i-th current hash center, Used to represent the hash center corresponding to the (m+1)th image of the first image category. Used to represent Hamming distance Used to indicate a preset distance. Used to represent the dimension of Hamming space.

5. The method according to claim 1, characterized in that, The first hash model includes a feature processing module and a hash layer, wherein the feature processing module includes a feature extraction module and a feature enhancement module; The input of the feature enhancement module is the output of the feature extraction module, the output of the feature enhancement module is the input of the hash layer, and the output of the hash layer is the output of the first hash model. The feature extraction module is used to extract the third feature corresponding to the query image, the feature enhancement module is used to enhance the third feature to obtain the fourth feature, and the hash layer is used to hash the fourth feature to obtain the query hash code.

6. The method according to claim 5, characterized in that, The feature enhancement module includes a first fully connected layer, a second fully connected layer, and a weight calculation module; The output of the feature extraction module is the input of the first fully connected layer and the second fully connected layer; The outputs of the first fully connected layer and the second fully connected layer serve as the inputs to the weight calculation module; The first fully connected layer is used to map the third feature to a first component feature; The second fully connected layer is used to map the third feature to the second component feature; The weight calculation module is used to perform dot product attention calculation on the first component feature and the second component feature to determine the target weight. The target weight is used to achieve weighted fusion of the first component feature and the second component feature to obtain the fourth feature.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Obtain the first number corresponding to the historical sampled images, and the total number of categories corresponding to the first image category and the second image category; Using the ratio of the first quantity to the total number of categories, a third quantity of downsampled images for each image category is determined; From the newly added training images, select the third number of second sampled images for each of the first image categories; The historical sampled image is updated using all the second sampled images to obtain the updated historical sampled image, which is used to incrementally update the first hash model in the next stage.

8. The method according to claim 7, characterized in that, The step of selecting the third number of second sampled images from the newly added training images for each of the first image categories includes: For each of the first image categories, all category sample images corresponding to the first image category are selected from the newly added training images; The sampled image of the category is input into the first hash model to obtain the fifth feature output by the feature processing module in the first hash model; The fifth feature is averaged across all sampled images of the aforementioned categories to obtain the sixth feature. The second sampled image is selected from the category sampled images using the Euclidean nearest neighbor principle between the fifth feature and the sixth feature.

9. The method according to claim 7, characterized in that, The step of updating the historical sampled image using all the second sampled images to obtain the updated historical sampled image includes: The fourth number of historical sampled images is deleted from each of the second image categories in the historical sampled images to obtain the deleted historical sampled images. The fourth number is determined based on the difference between the third number and the fifth number. The fifth number is the number of historical sampled images in each of the second image categories. The updated historical sampled image is obtained by combining all the second sampled images with the deleted historical sampled images.

10. An electronic device, characterized in that, include: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes the computer instructions to perform the incremental image hashing retrieval method based on feature enhancement and knowledge distillation as described in any one of claims 1 to 9.

Images