Image retrieval method, device and computer-readable storage medium

Abstract

The present invention relates to the field of computer processing technology, and specifically provides an image retrieval method, device and computer-readable storage medium, aiming to solve the technical problem of efficiently retrieving matching images. To this end, the method of the present invention includes: obtaining a first model and a second model for image retrieval, the first model being a model obtained by compressing the second model through knowledge distillation technology; retrieving a second group of images whose similarity with the target image is higher than a preset level from the first group of images through the first model; and retrieving images matching the target image from the second group of images through the second model. According to the present invention, through the knowledge distillation technology, a lightweight small model is prepared in advance for preliminary image retrieval, and a complex large model is used for the second retrieval, the two retrievals are efficient, and the final retrieval result has a high accuracy rate.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and specifically provides a picture retrieval method, device and computer-readable storage medium. Background Art

[0002] With the continuous development of the hardware industry, video analysis has become an indispensable safety guarantee in modern life. Among them, intelligent analysis of pedestrians is an important component. Pedestrian re-identification technology uses image processing related technologies in the field of computer vision to retrieve pedestrian images across fields and cameras based on given pedestrian information, breaking through the limitations of the field of view brought by a single camera. Pedestrian re-identification technology has a wide range of applications. For example, in commercial service scenarios, by collecting customers' shopping behaviors, information about customers' gender, age, shopping preferences, purchasing power, etc. can be analyzed, which is of great reference significance for shopping malls and merchants to plan activities and provide targeted services, thereby improving sales performance.

[0003] The key technology for pedestrian re-identification tasks is image retrieval, that is, given a pedestrian image or pedestrian video as a query sample, the similarity relationship of the images is used to re-sort the images in the large-scale database shot across lenses to find the most matching pedestrian sample image, and then the ID (image identifier, indicating the identity of the pedestrian in the image) information of the sample is retrieved through analysis to obtain the accuracy of the re-identification task. However, when the database is too large, it is very inefficient to search for images from tens or millions of data and return the sorted results. This cannot meet the needs of real-time retrieval in scenarios that require a large amount of commercial use. Summary of the invention

[0004] In order to overcome the above-mentioned defects, the present invention is proposed to provide a picture retrieval method, device and computer-readable storage medium that solve or at least partially solve the technical problem of efficiently retrieving matching pictures.

[0005] In a first aspect, the present invention provides a method for image retrieval, the method comprising:

[0006] Obtaining a first model and a second model for image retrieval, wherein the first model is a model obtained by compressing the second model through a knowledge distillation technology;

[0007] Retrieving, by using the first model, from the first group of pictures a second group of pictures whose similarity to the target pictures is higher than a preset level;

[0008] Using the second model, a picture matching the target picture is retrieved from the second group of pictures.

[0009] In a technical solution of the above-mentioned image retrieval method, the step of "retrieving, by means of the first model, from the first group of images a second group of images having a similarity with the target image higher than a preset level" includes:

[0010] extracting a-dimensional features of the target image and the first group of images through the first model, calculating a first similarity between the target image and the first group of images according to the a-dimensional features of the target image and the first group of images, and selecting the second group of images from the first group of images according to the first similarity;

[0011] and / or,

[0012] The step of “calculating a first similarity between the target image and the first group of images according to the a-dimensional features of the target image and the first group of images” includes:

[0013] Calculating the Euclidean distance between the target image and the first group of images according to the a-dimensional features of the target image and the first group of images, and determining and calculating a first similarity between the target image and the first group of images according to the distance;

[0014] and / or,

[0015] The step of “selecting the second group of pictures from the first group of pictures according to the first similarity” includes:

[0016] Retrieving k pictures with the highest first similarity from the first group of pictures as the second group of pictures;

[0017] and / or,

[0018] The step of “retrieve a picture matching the target picture from the second group of pictures by using the second model” includes:

[0019] Extracting b-dimensional features of the target image and the second group of images through the second model, calculating a second similarity between the target image and the second group of images according to the b-dimensional features of the target image and the second group of images, and selecting images matching the target image from the second group of images according to the second similarity, where b is greater than a;

[0020] and / or,

[0021] Before the step of "selecting pictures matching the target picture from the second group of pictures according to the second similarity", the method further includes:

[0022] adjusting a second similarity between the target image and the second group of images according to the first similarity between the target image and the second group of images;

[0023] and / or,

[0024] The difference between the distribution of the first similarity between the target image and the second group of images and the distribution of the second similarity between the target image and the second group of images is within a preset interval.

[0025] In a second aspect, a picture retrieval device is provided, the device comprising:

[0026] A model acquisition module, which acquires a first model and a second model for image retrieval, wherein the first model is a model obtained by compressing the second model through knowledge distillation technology;

[0027] A first retrieval module, which retrieves a second group of pictures whose similarity to the target picture is higher than a preset level from the first group of pictures through the first model;

[0028] A second retrieval module, which retrieves pictures matching the target picture from the second group of pictures by using the second model;

[0029] The first retrieval module extracts a-dimensional features of the target image and the first group of images through the first model, calculates a first similarity between the target image and the first group of images according to the a-dimensional features of the target image and the first group of images, and selects the second group of images from the first group of images according to the first similarity;

[0030] The second retrieval module extracts b-dimensional features of the target image and the second group of images through the second model, calculates a second similarity between the target image and the second group of images based on the b-dimensional features of the target image and the second group of images, selects images matching the target image from the second group of images based on the second similarity, where b is greater than a; and adjusts the second similarity between the target image and the second group of images based on the first similarity between the target image and the second group of images.

[0031] In a third aspect, a control device is provided, which includes a processor and a storage device, wherein the storage device is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by the processor to execute the image retrieval method described in any one of the technical solutions of the above-mentioned image retrieval method.

[0032] In a fourth aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, wherein the program codes are suitable for being loaded and run by a processor to execute the above-mentioned image retrieval method described in any one of the technical solutions of the above-mentioned image retrieval method.

[0033] The above one or more technical solutions of the present invention have at least one or more of the following beneficial effects:

[0034] In a technical solution for implementing the present invention, the image retrieval method may include the following steps: obtaining a first model and a second model for image retrieval, wherein the first model is a model obtained by compressing the second model through the knowledge distillation technology. In comparison, the prior art is equivalent to using only the second model; through the first model, a second group of images whose similarity with the target image is higher than a preset level is retrieved from the first group of images. Since the first model is a small model after knowledge distillation, it has a small scale, fast calculation, and relatively low accuracy, so the screening of the second group of images can be completed quickly. At the same time, since the second group of images retains multiple images that are most similar to the target image, even if the accuracy of the first model is insufficient, the matching images can be retained in the second group of images; through the second model, images matching the target image are retrieved from the second group of images. The second model has a large scale, complex calculation, and high accuracy, but since the number of the second group of images is limited, the matching images of the target image can be retrieved quickly and accurately. In the technical solution of the present invention, a lightweight small model is prepared in advance for preliminary image retrieval through the knowledge distillation technology, and a complex large model is used for the second retrieval. The two retrievals are efficient and the final retrieval result has a high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. Among them:

[0036] Figure 1 is a schematic flow chart of the main steps of a picture retrieval method according to an embodiment of the present invention;

[0037] Figure 2 is a schematic flow chart of the main steps of a picture retrieval method according to an embodiment of the present invention;

[0038] Figure 3 is a schematic diagram of a first model training method used in an image retrieval method according to an embodiment of the present invention;

[0039] Figure 4 is a schematic diagram of a second model training method used in an image retrieval method according to an embodiment of the present invention;

[0040] Figure 5 is a schematic diagram of Euclidean distance calculation according to an image retrieval method according to an embodiment of the present invention;

[0041] Figure 6 It is a schematic diagram of the workflow of the image retrieval method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] Some embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0043] In the description of the present invention, "module" and "processor" may include hardware, software or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, and memories, and may also include software parts, such as program codes, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware, or a combination of the two. Non-temporary computer-readable storage media include any suitable medium that can store program codes, such as a magnetic disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, and the like. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The term "at least one A or B" or "at least one of A and B" has a similar meaning to "A and / or B", and may include only A, only B, or A and B. The singular terms "one" and "the" may also include plural forms.

[0044] See attached Figure 1 , Figure 1 It is a flowchart diagram of the main steps of a picture retrieval method according to an embodiment of the present invention.

[0045] The technical solution of this embodiment can be used for various types of image retrieval, and is not limited to pedestrian retrieval when performing ReID (using computer technology to determine whether a specific pedestrian exists in an image or video sequence) tasks (that is, given a pedestrian image or a pedestrian video image as a query image, the similarity relationship of the images is used to reorder the images in a large-scale database, and the most similar pedestrian image with the same ID is found as the pedestrian retrieval return result).

[0046] like Figure 1 As shown, the image retrieval method in the embodiment of the present invention mainly includes the following steps:

[0047] Step S110, obtaining a first model and a second model for image retrieval, wherein the first model is a model obtained by compressing the second model through knowledge distillation technology.

[0048] The knowledge distillation technology introduces the teacher model (complex but with excellent prediction accuracy) and the student model (simplified, low complexity, more suitable for reasoning deployment). Through training, the output distribution of the student model and the teacher model is made as close as possible, the classification accuracy of the student model is improved, and knowledge transfer is realized. In this embodiment, the first model is the student model and the second model is the teacher model.

[0049] Step S120: Retrieve, from the first group of pictures, a second group of pictures whose similarity to the target picture is higher than a preset level through the first model.

[0050] The first model obtained through knowledge distillation technology has fast inference speed and low precision. When performing pedestrian image retrieval, the base database (i.e., the first group of images) can be quickly searched first, the base database range can be narrowed (the second group of images), and the time required for pedestrian retrieval can be shortened.

[0051] Step S130: Retrieve a picture matching the target picture from the second group of pictures using the second model.

[0052] During the secondary search, the high calculation accuracy of the second model is fully utilized. Although the calculation of the second model is complex, the scope of the base database has been narrowed, so the secondary search efficiency is extremely high.

[0053] According to the technical solution of this embodiment, a lightweight small model is prepared in advance for preliminary image retrieval through knowledge distillation technology, and a complex large model is used for the second retrieval. The two retrievals are efficient and the final retrieval result has a high accuracy rate.

[0054] like Figure 2 As shown, the image retrieval method in the embodiment of the present invention mainly includes the following steps:

[0055] Step S210, obtaining a first model and a second model for image retrieval, wherein the first model is a model obtained by compressing the second model through knowledge distillation technology.

[0056] Assume that the first model is the Resnet18 model and the second model is the Resnet152 model. The Resnet18 model is obtained by performing knowledge distillation on the Resnet152 model, as follows:

[0057] The training process of the Resnet152 model can be as follows Figure 3As shown, the model cuts the input image into a fixed size of 384×192, extracts features (converts the image into a feature vector, represents the image with the feature vector, and in this embodiment, uses the ResNet deep neural network to convert the pedestrian image into a deep feature vector of fixed dimension) to obtain a 2048×24×12-dimensional feature vector, which is then compressed to 2048×1×1 dimensions through a pooling layer, and then reduced to 512 dimensions through a 1×1 convolution layer to obtain a triplet vector (triplet embedding) and calculate the triplet loss. The triplet loss value can be used to optimize the distance between the positive and negative sample images, and realize the similarity calculation of the sample images. The other branch is activated through the BatchNorm layer (an algorithm often used in deep networks to accelerate neural network training, convergence speed and stability), and then the Arcface function is used to map the feature vector into logits (the output of the model's fully connected layer) indicating the similarity between the target image and the base image, and calculate the cross entropy loss function for classification learning.

[0058] The output logits of the trained teacher model Resnet152 are used as the target of the lighter student model Resnet18. Only the student model is back-propagated to train the logits distribution of the student model to be close to the logits distribution of the teacher model. The knowledge distillation process is as follows: Figure 4 As shown in the figure, the input image passes through the teacher model and the student model at the same time, and the KL divergence (relative entropy) loss is calculated based on the output logits to shorten the distance between positive and negative examples, and the parameters of the student model are updated through the gradient backpropagation technology. During the distillation process, not only the KL divergence loss is used to optimize the student model, but also the cross entropy loss and triplet loss are used to optimize the student model similar to the teacher model.

[0059] Step S220, extracting the a-dimensional features of the target image and the first group of images through the first model, calculating the first similarity between the target image and the first group of images according to the a-dimensional features of the target image and the first group of images, and selecting the second group of images from the first group of images according to the first similarity.

[0060] Specifically, the Euclidean distance between the target image and the first group of images may be calculated according to the a-dimensional features of the target image and the first group of images, and the first similarity between the target image and the first group of images may be calculated according to the distance.

[0061] In this embodiment, a small model is used to extract the features of the base image and initially retrieve the base image. Figure 5As shown, first, the target image and the base library are respectively used to extract 256-dimensional feature vectors through the student model. For the target image, it is compared one by one with D base library images, and the Euclidean distance (Euclidean distance) between each pair is calculated. The Euclidean distance represents the real distance between two points in a multi-dimensional space, and the calculation formula is as follows:

[0062]

[0063] The above formula is the normalization formula of the feature vector, and the normalized feature second norm is 1. The following formula is the calculation formula of the Euclidean distance, where X and Y respectively represent different feature vectors, and m represents the dimension of the feature space.

[0064] Specifically, in this embodiment, the k images with the highest first similarity can be retrieved from the first group of images as the second group of images.

[0065] After calculating the Euclidean distance, the base library sample images are sorted from near to far according to the distance. The closer the distance, the higher the similarity. The first K sample images with the highest similarity are selected from the base library with a data volume of D (K << D). Because the base library data is very messy, with different pedestrian images under different lenses, so if you want to query the pedestrian with the same ID as the target image in such a huge base library data, then very likely the similarity of this sample image should be among the first K. This embodiment uses a small model for fast inference to preliminarily screen the base library, saving time for the retrieval operation of a larger model later.

[0066] In this embodiment, a similarity threshold can also be set. When using the first model for preliminary retrieval, feature similarity comparison is performed. The base library sample images with a similarity greater than the threshold are retained, and the base library sample images with a similarity less than the threshold are deleted. The remaining sample images are then used for secondary retrieval. However, for pedestrian retrieval, usually there is no unified standard for the base library sample images themselves. For sample images with poor quality, the features obtained by the small model are not very expressive, and the similarity may not be very high. Setting a higher threshold will result in deleting too many base library sample images during preliminary screening. For some IDs of images, there are a large number of corresponding sample images in the base library, and at this time the corresponding effective screening threshold will be relatively high. Therefore, when setting the similarity threshold, how to choose the size of the threshold is a very complex problem. In this embodiment, the first K sample images with the highest similarity are fixedly returned for each ID, which is a feasible solution for different ID sample images and is the simplest and most efficient method.

[0067] Step S230, extracting b-dimensional features of the target image and the second group of images through the second model, calculating the second similarity between the target image and the second group of images based on the b-dimensional features of the target image and the second group of images, adjusting the second similarity between the target image and the second group of images based on the first similarity between the target image and the second group of images, and selecting images matching the target image from the second group of images based on the second similarity, where b is greater than a.

[0068] In this embodiment, the adjustment method is not limited, and generally, the first similarity and the second similarity can be directly added to achieve cascading.

[0069] Here, a cascade fusion method is used to fuse the 512-dimensional logits vector of Resnet152 and the 256-dimensional logits vector of Resnet18 into a 768-dimensional vector for the final retrieval. The process of the secondary target retrieval is almost the same as the initial one. Here, only the K filtered base sample images are used to calculate the Euclidean distance with the query image, and the distance ranking of the K images is obtained. The corresponding performance indicators mAP, Top1, and Top10 are calculated to assist in selecting the final matching sample images. That is, multiple models are jointly trained, combined with global and local feature learning, feature fusion is performed, and retrieval performance is improved. The image base filtered out by the initial sorting result is optimized again, and a secondary sort is performed to obtain the final result.

[0070] Here, the second similarity may not be adjusted, that is, the target image matching results are screened out based only on the logits output by the second model. This method is more efficient, but less accurate than the cascade logits method.

[0071] In this embodiment, the difference between the distribution of the first similarity between the target image and the second group of images and the distribution of the second similarity between the target image and the second group of images is within a preset interval.

[0072] In this embodiment, there is no restriction on the preset interval, which only reflects that the logits distribution output by the first model is close to that of the second model, that is, the first model and the second model have similar performance after knowledge distillation.

[0073] In order to solve the problem of slow retrieval speed, this embodiment designs a multi-model fast retrieval base database solution for ReID tasks. The process is as follows: Figure 6 As shown in the figure: by jointly training a large model and a small model, the small model is first used to sort the base database for the first time, and the top K similar sample images are quickly screened out from the base database of large batches of data. Then, through multi-model feature fusion, a second sorting is performed on the K base database to obtain the sorting result and the first hit pedestrian image.

[0074] It is understood by those skilled in the art that the present invention implements all or part of the processes in the method of the above embodiment, and can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0075] Furthermore, the present invention also provides a control device. In an embodiment of a control device according to the present invention, the control device includes a processor and a storage device, the storage device can be configured to store a program for executing the image retrieval method of the above method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to the program for executing the image retrieval method of the above method embodiment. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The control device can be a control device device formed by various electronic devices.

[0076] Furthermore, the present invention also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present invention, the computer-readable storage medium can be configured to store a program for executing the image retrieval method of the above method embodiment, and the program can be loaded and run by the processor to implement the above image retrieval method. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present invention is a non-temporary computer-readable storage medium.

[0077] Further, it should be understood that since the setting of each module is only for illustrating the functional units of the device of the present invention, the physical devices corresponding to these modules may be the processor itself, or a part of the software in the processor, a part of the hardware, or a part of the combination of software and hardware. Therefore, the number of each module in the figure is only schematic.

[0078] Those skilled in the art will appreciate that the modules in the device can be adaptively split or merged. Such splitting or merging of specific modules will not cause the technical solution to deviate from the principle of the present invention, and therefore, the technical solutions after splitting or merging will fall within the protection scope of the present invention.

[0079] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A picture retrieval method, characterized in that: The method comprises: Obtaining a first model and a second model for image retrieval, wherein the first model is a model obtained by compressing the second model through a knowledge distillation technology; Retrieving, by using the first model, from the first group of pictures a second group of pictures whose similarity to the target pictures is higher than a preset level; Retrieving, by using the second model, a picture matching the target picture from the second group of pictures; The step of “retrieving, from the first group of pictures, a second group of pictures whose similarity to the target pictures is higher than a preset level through the first model” includes: extracting a-dimensional features of the target image and the first group of images through the first model, calculating a first similarity between the target image and the first group of images according to the a-dimensional features of the target image and the first group of images, and selecting the second group of images from the first group of images according to the first similarity; The step of “retrieve a picture matching the target picture from the second group of pictures by using the second model” includes: Extracting b-dimensional features of the target image and the second group of images through the second model, calculating a second similarity between the target image and the second group of images according to the b-dimensional features of the target image and the second group of images, and selecting images matching the target image from the second group of images according to the second similarity, where b is greater than a; Before the step of "selecting pictures matching the target picture from the second group of pictures according to the second similarity", the method further includes: According to the first similarity between the target picture and the second group of pictures, a second similarity between the target picture and the second group of pictures is adjusted.

2. The image retrieval method according to claim 1, characterized in that: The step of “calculating a first similarity between the target image and the first group of images according to the a-dimensional features of the target image and the first group of images” includes: The Euclidean distance between the target image and the first group of images is calculated according to the a-dimensional features of the target image and the first group of images, and the first similarity between the target image and the first group of images is calculated according to the size of the distance.

3. The image retrieval method according to claim 1, characterized in that: The step of "selecting the second group of pictures from the first group of pictures according to the first similarity" includes: K pictures with the highest first similarity are retrieved from the first group of pictures as the second group of pictures.

4. The image retrieval method according to claim 1, characterized in that: The difference between the distribution of the first similarity between the target image and the second group of images and the distribution of the second similarity between the target image and the second group of images is within a preset interval.

5. A picture retrieval device, characterized in that: The device comprises: A model acquisition module, which acquires a first model and a second model for image retrieval, wherein the first model is a model obtained by compressing the second model through knowledge distillation technology; A first retrieval module, which retrieves a second group of pictures whose similarity to the target picture is higher than a preset level from the first group of pictures through the first model; A second retrieval module, which retrieves pictures matching the target picture from the second group of pictures by using the second model; The first retrieval module extracts a-dimensional features of the target image and the first group of images through the first model, calculates a first similarity between the target image and the first group of images according to the a-dimensional features of the target image and the first group of images, and selects the second group of images from the first group of images according to the first similarity; The second retrieval module extracts b-dimensional features of the target image and the second group of images through the second model, calculates a second similarity between the target image and the second group of images based on the b-dimensional features of the target image and the second group of images, selects images matching the target image from the second group of images based on the second similarity, where b is greater than a; and adjusts the second similarity between the target image and the second group of images based on the first similarity between the target image and the second group of images.

6. A control device, comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by the processor to execute the image retrieval method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the image retrieval method according to any one of claims 1 to 4.

Images