Homologous image retrieval method and system

Abstract

Disclosed is a homologous image retrieval method and system, including dividing a teacher network and a student network into multiple modules, respectively, and performing loss constraints and knowledge distillation on a shallow feature extraction module and a deep feature extraction module; using the pseudo labels output by the teacher network training on shallow features as the labels output by the student network after shallow feature learning, and calculating the soft classification cross entropy loss; performing similarity calculation on the encoding outputs of the teacher network and the student network of deep features, and adding the hard classification cross entropy loss of the teacher network and the student network to the total loss function to complete the heuristic network lightweight training. This application constrains the feature learning capabilities of models at different levels of depth through heuristic learning of the teacher network by the student network, and simultaneously considers the model representation capabilities at the label level and the feature level, so that the student network can achieve image representation capabilities that are not lower than those of the teacher network, thereby achieving model lightweighting.

Description

Technical Field

[0001] The present invention relates to the technical field of homologous image retrieval, and in particular to a homologous image retrieval method and system. Background Art

[0002] Homologous image retrieval aims to identify the original image of a target image from a massive image collection. Its basic implementation involves using a feature extractor (such as SIFT or convolutional neural networks) to obtain a high-dimensional representation of image information. Similarity comparison retrieval is then performed within the target source library. The use of convolutional neural networks as feature extractors has become increasingly mainstream. To address the vast number of image categories, contrastive learning techniques have been applied to homologous image retrieval. However, this approach often requires a large batch size, which is difficult to meet in industry. Therefore, in practical applications, more complex networks are often used to compensate for the smaller batch size, while improvements are made to the network structure or loss function. Generally speaking, the challenges of homologous image retrieval technology lie in the large number of model parameters, slow inference speed, and difficulty in deploying to end devices. This is primarily due to the large amount of category information required during training, which requires a large model parameter count to store this complex information.

[0003] The current convolutional neural network-based homology image retrieval aims to mine representative and compact image features. However, in the face of a large number of image categories, the feature extractor requires a larger and deeper network structure to carry a higher information discrimination capability. When deploying the model on a terminal device, such a model with a large number of parameters will cause a bottleneck in the reasoning speed, so the lightweighting of the model becomes the main optimization goal of the present invention. Knowledge distillation is a common heuristic model lightweighting method. Common knowledge extraction methods only apply the representation information at the label level or only apply the representation information at the feature level. This makes it difficult for the distillation process to integrate the shallow features and deep features mined by the teacher model, making the lightweight model less generalizable. Summary of the Invention

[0004] In order to solve the technical problems in the existing technology that large-parameter models will cause bottlenecks in inference speed, the distillation process is difficult to integrate shallow features and deep features mined by the teacher model, and the lightweight model has insufficient generalization, the present invention proposes a homologous image retrieval method and system to solve the above technical problems.

[0005] According to a first aspect of the present invention, a homologous image retrieval method is proposed, comprising:

[0006] S1: The teacher network and the student network are divided into multiple modules, and the shallow feature extraction module and the deep feature extraction module are respectively subjected to loss constraints and knowledge distillation;

[0007] S2: On the shallow features, the pseudo labels output by the teacher network training are used as the labels output by the student network after shallow feature learning, and the soft classification cross entropy loss is calculated;

[0008] S3: Calculate the similarity between the encoding outputs of the teacher network and the student network of deep features, add the hard classification cross entropy loss of the teacher network and the student network to the total loss function, and complete the heuristic network lightweight training.

[0009] In some specific embodiments, the teacher network and the student network simultaneously adopt a network structure with four modules connected in series, the teacher network adopts ResNet101 as the backbone network, and the student network adopts ResNet18 as the backbone network.

[0010] In some specific embodiments, a classification head consisting of a fully connected layer and a Softmax is added after the first, second, and last modules of the teacher network and the student network, respectively. i (·) is used to calculate the feature classification ability under different depth features and perform distillation learning of classification ability.

[0011] In some specific embodiments, a feature representation head g consisting of a fully connected layer is added after the third module of the teacher network and the student network. i (·), which is used to enable the student network to learn high-level semantic feature representations of images.

[0012] In some specific embodiments, the calculation formula of the total loss function is: L total =αL softCE1 +βL softCE2 +γL dist3 +δL hardCE1 +εL hardCE2 ,in, α+β+γ+δ+ε=1, is the Softmax output of the first module of the teacher network at temperature t, is the Softmax output of the first module of the student network at temperature t, Indicates that the temperature of Softmax is 1, ‖·‖2 indicates the L2 norm calculation, Represents the output of the third module of the teacher network after passing through the feature representation head.

[0013] In some specific embodiments, the teacher network and the student network are trained on the input dataset simultaneously and in parallel until the loss function converges.

[0014] In some specific embodiments, a student network is used as the reasoning and model deployment, and the similarity of the Euclidean distance of the output feature vector rows of the test sample after encoding by the four network modules is calculated, and the picture with the highest similarity in the base library is selected as the prediction source picture.

[0015] According to a second aspect of the present invention, a computer-readable storage medium is provided, on which one or more computer programs are stored. When the one or more computer programs are executed by a computer processor, the above-mentioned method is implemented.

[0016] According to the third aspect of the present invention, a homologous image retrieval system is proposed, including a teacher model and a student model, the teacher model and the student model are respectively divided into multiple modules, the shallow feature extraction module and the deep feature extraction module are respectively subjected to loss constraints and knowledge distillation, the pseudo labels output by the teacher model training on the shallow features are used as the labels output by the student model after shallow feature learning, the soft classification cross entropy loss is calculated, the similarity is calculated for the encoded outputs of the teacher model and the student model of the deep features, the hard classification cross entropy loss of the teacher model and the student model is added to the total loss function, and the heuristic model lightweight training is completed.

[0017] In some specific embodiments, the teacher model and the student model simultaneously adopt a network structure with four modules in series, the teacher model adopts ResNet101 as the backbone network, and the student model adopts ResNet18 as the backbone network.

[0018] In some specific embodiments, a classification head f consisting of a fully connected layer and a Softmax is added after the first, second, and last modules of the teacher model and the student model, respectively. i (·) is used to calculate the feature classification ability under different depth features and perform distillation learning of classification ability. A feature representation head g consisting of a fully connected layer is added after the third module of the teacher model and the student model. i (·), which is used to enable the student model to learn high-level semantic feature representations of images.

[0019] In some specific embodiments, the calculation formula of the total loss function is: L total =αL softCE1 +βL softCE2 +γL dist3 +δL hardCE1 +εL hardCE2 ,in, α+β+γ+δ+ε=1, is the Softmax output of the first module of the teacher model at temperature t, is the Softmax output of the first module of the student model at temperature t, Indicates that the temperature of Softmax is 1, ‖·‖2 indicates the L2 norm calculation, Represents the output of the third module of the teacher model after passing through the feature representation head.

[0020] In some specific embodiments, the teacher model and the student model train the input dataset simultaneously and in parallel until the loss function converges.

[0021] In some specific embodiments, a student model is used for inference and model deployment, and the similarity of the Euclidean distance of the output feature vector rows of the test sample after encoding by the four network modules is calculated, and the picture with the highest similarity in the base library is selected as the prediction source picture.

[0022] The present invention proposes a homologous image retrieval method and system. Through the heuristic learning of the teacher network by the student network, and considering the model representation capabilities at the label level and the feature level at the same time, the model feature learning capabilities at different levels of depth are constrained, so that the student network can achieve image representation capabilities that are no less than that of the teacher network, and has a model size that is much smaller than that of the teacher network, thereby achieving model lightweighting. This is of great significance and value to the application and industrial implementation of homologous image retrieval algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many of the intended advantages of the embodiments will be readily apparent as they become better understood by reference to the following detailed description. Other features, objects, and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments made with reference to the following drawings:

[0024] Figure 1 is a flowchart of a homologous image retrieval method according to an embodiment of the present application;

[0025] Figure 2 This is an algorithm framework diagram of a homologous image retrieval method according to a specific embodiment of the present application;

[0026] Figure 3 This is a network structure diagram of homologous image retrieval according to an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0028] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0029] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] Figure 1 FIG. 1 shows a flow chart of a homologous image retrieval method according to an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0031] S101: Divide the teacher network and the student network into multiple modules, and perform loss constraints and knowledge distillation on the shallow feature extraction module and the deep feature extraction module respectively.

[0032] In a specific embodiment, the teacher network and the student network simultaneously adopt a network structure of four modules in series, the teacher network uses ResNet101 as the backbone network, and the student network uses ResNet18 as the backbone network. A classification head consisting of a fully connected layer and a Softmax layer is added after the first, second, and last modules of the teacher network and the student network, respectively. i (·) is used to calculate the feature classification ability under different depth features and to perform distillation learning of classification ability. A feature representation head g consisting of a fully connected layer is added after the third module of the teacher network and the student network. i (·), which is used to enable the student network to learn high-level semantic feature representations of images.

[0033] S102: On the shallow features, the pseudo labels output by the teacher network training are used as the labels output by the student network after shallow feature learning, and the soft classification cross entropy loss is calculated.

[0034] S103: Calculate the similarity between the encoding outputs of the teacher network and the student network of the deep features, add the hard classification cross entropy loss of the teacher network and the student network to the total loss function, and complete the heuristic network lightweight training.

[0035] In a specific embodiment, the calculation formula of the total loss function is: L total =αL softCE1 +βL softCE2 +γL dist3 +δL hardCE1 +εL hardCE2 ,in, α+β+γ+δ+ε=1, is the Softmax output of the first module of the teacher network at temperature t, is the Softmax output of the first module of the student network at temperature t, Indicates that the temperature of Softmax is 1, ‖·‖2 indicates the L2 norm calculation, Represents the output of the third module of the teacher network after passing through the feature representation head.

[0036] A network structure for homologous image retrieval in this application is as follows Figure 2 As shown in Figure 1, the overall network is divided into two branches: the first and second branches use a network structure with four modules in series. The first branch uses ResNet101 as the backbone network, aiming to build a "teacher network" with a more complex structure and higher retrieval accuracy. The second branch uses ResNet18 as the backbone network for knowledge distillation learning and lightweight model deployment. The classification head f is added after the first, second and last modules of the upper and lower networks respectively. i (·), consisting of a fully connected layer and Softmax, is used to calculate the feature classification ability under different depth features and perform distillation learning of classification ability. Feature representation head g is added after the third module. i (·), which consists of a fully connected layer (performing moderate feature dimensionality reduction), and is used to enable the second-way “student network” to learn high-level semantic feature representations of images.

[0037] In a specific embodiment, the loss function of the model consists of five parts, as shown in the following formula: total =αL softCE1 +βL softCE2 +γL dist3 +δL hardCE1 +εL hardCE2 ,in: α+β+γ+δ+ε=1, is the Softmax output of the first module of the first branch (teacher network) at temperature t, expressed as is the Softmax output of the first module of the second branch (student network) at temperature t, Indicates that the temperature of Softmax is 1, ‖·‖2 indicates the L2 norm calculation, y represents the output of the third module of the first branch (teacher network) after passing through the feature representation head. The other symbols are similar. y represents the true value of each category. In the experiment, t = 7, α = 0.25, β = 0.25, γ = 0.2, δ = 0.15, and ε = 0.15 are set. These values ​​can be adjusted according to actual conditions.

[0038] In a specific embodiment, during model training, the two branch networks train the input data set simultaneously and in parallel until the loss function converges.

[0039] In a specific embodiment, only the second path network is used for inference and model deployment during the inference process. The output feature vector v∈R after the test sample is encoded by the four network modules is used. 1*n The similarity calculation of Euclidean distance is performed, where n is the dimension parameter of the representation vector, which is set to 128 in the experiment. Generally, the image with the highest similarity in the base database is selected as the prediction source image.

[0040] The inventors of this application have discovered through experiments that multi-module, multi-stage distillation achieves better results than single-module knowledge distillation. During the distillation process, knowledge is not only transferred at the label level but also constrained at the feature level. This effectively improves the student network's feature representation capabilities, which are crucial for homology image retrieval. Therefore, after heuristic iterative training, the second-pass network can maintain accuracy with fewer parameters.

[0041] The present invention uses the second network to heuristically learn the first network, and simultaneously considers the model representation capabilities at the label level and the feature level, thereby constraining the model feature learning capabilities at different levels of depth, so that the second network can achieve image representation capabilities that are no less than those of the first network, and has a model size that is much smaller than that of the first network, thereby achieving model lightweighting. This is of great significance and value to the application and industrial implementation of homologous image retrieval algorithms.

[0042] Figure 3 FIG. 1 shows an architecture diagram of a homologous image retrieval system according to an embodiment of the present application. Figure 3 As shown, the system includes a teacher model 301 and a student model 302, wherein the teacher model 301 and the student model 302 are respectively divided into multiple modules, and the shallow feature extraction module and the deep feature extraction module are respectively subjected to loss constraints and knowledge distillation, and the pseudo labels output by the teacher model 301 training on the shallow features are used as the labels output by the student model 302 after shallow feature learning, and the soft classification cross entropy loss is calculated. The similarity is calculated for the encoding outputs of the teacher model 301 and the student model 302 of the deep features, and the hard classification cross entropy loss of the teacher model 301 and the student model 302 is added to the total loss function to complete the heuristic model lightweight training.

[0043] Reference below Figure 4 , which shows a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. Figure 4 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0044] like Figure 4 As shown, the computer system includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage unit 408 into a random access memory (RAM) 403. Various programs and data required for the operation of the system 400 are also stored in the RAM 403. The CPU 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0045] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, and the like; an output section 407 including a liquid crystal display (LCD) and speakers; a storage section 408 including a hard disk; and a communication section 409 including a network interface card such as a LAN card or a modem. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. Removable media 411, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 410 as needed, so that computer programs read therefrom can be installed into the storage section 408 as needed.

[0046] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0047] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0048] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0049] The modules described in the embodiments of the present application may be implemented by software or hardware.

[0050] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or it may exist independently and not be assembled into the electronic device. The above computer-readable storage medium carries one or more programs. When the above one or more programs are executed by the electronic device, the electronic device: divides the teacher network and the student network into multiple modules respectively, and performs loss constraints and knowledge distillation on the shallow feature extraction module and the deep feature extraction module respectively; uses the pseudo labels output by the teacher network training as the labels output by the student network after shallow feature learning on the shallow features, and calculates the soft classification cross entropy loss; calculates the similarity of the encoding outputs of the teacher network and the student network of the deep features, adds the hard classification cross entropy loss of the teacher network and the student network to the total loss function, and completes the heuristic network lightweight training.

[0051] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A homologous image retrieval method, characterized in that: include: S1: The teacher network and the student network are divided into multiple modules, and the shallow feature extraction module and the deep feature extraction module are respectively subjected to loss constraints and knowledge distillation; S2: Using the pseudo labels output by the teacher network training on the shallow features as the labels output by the student network after shallow feature learning, and calculating the soft classification cross entropy loss; S3: Calculate the similarity between the encoding outputs of the teacher network and the student network of the deep features, add the hard classification cross entropy loss of the teacher network and the student network to the total loss function, and complete the heuristic network lightweight training; The teacher network and the student network both use a network structure with four modules connected in series. The teacher network uses ResNet101 as the backbone network, and the student network uses ResNet18 as the backbone network. A classification head consisting of a fully connected layer and Softmax is added after the first, second and last modules of the teacher network and the student network respectively. , which is used to calculate the feature classification ability under different depth features and perform distillation learning of classification ability. A feature representation head consisting of a fully connected layer is added after the third module of the teacher network and the student network. , used to enable the student network to learn high-level semantic feature representations of images; The student network is used as the reasoning and model deployment, and the output feature vectors of the test samples after encoding by the four network modules are used to calculate the similarity of the Euclidean distance, and the picture with the highest similarity in the base library is selected as the prediction source picture.

2. The homologous image retrieval method according to claim 1, characterized in that: The calculation formula of the total loss function is: ,in, , , The first module of the teacher network is at a temperature of Softmax output under The first module of the student network is at a temperature of Softmax output under Indicates that the temperature of Softmax is 1, express Norm calculation, Represents the output of the third module of the teacher network after passing through the feature representation head.

3. The homologous image retrieval method according to claim 1, characterized in that: The teacher network and the student network train the input data set simultaneously and in parallel until the loss function converges.

4. A computer-readable storage medium having one or more computer programs stored thereon, characterized in that: When the one or more computer programs are executed by a computer processor, the method according to any one of claims 1 to 3 is implemented.

5. A homologous image retrieval system, characterized in that: The method comprises a teacher model and a student model, wherein the teacher model and the student model are respectively divided into multiple modules, and the shallow feature extraction module and the deep feature extraction module are respectively subjected to loss constraints and knowledge distillation, and the pseudo labels output by the teacher model training are used as the labels output by the student model after shallow feature learning on the shallow features, and the soft classification cross entropy loss is calculated. The similarity is calculated for the encoding outputs of the teacher model and the student model of the deep features, and the hard classification cross entropy loss of the teacher model and the student model is added to the total loss function to complete the heuristic model lightweight training; the teacher model and the student model simultaneously adopt a network structure of four modules in series, the teacher model adopts ResNet101 as the backbone network, and the student model adopts ResNet18 as the backbone network, and a classification head consisting of a fully connected layer and Softmax is added after the first, second and last modules of the teacher model and the student model respectively. , which is used to calculate the feature classification ability under different depth features and perform distillation learning of classification ability. A feature representation head consisting of a fully connected layer is added after the third module of the teacher model and the student model. , used to enable the student model to learn high-level semantic feature representations of images; The student model is used for inference and model deployment. The output feature vectors of the test samples after encoding by the four network modules are used for Euclidean distance similarity calculation, and the picture with the highest similarity in the base library is selected as the prediction source picture.

6. The homologous image retrieval system according to claim 5, characterized in that: The calculation formula of the total loss function is: ,in, , , The first module of the teacher model is at a temperature of Softmax output under The first module of the student model is at a temperature of Softmax output under Indicates that the temperature of Softmax is 1, express Norm calculation, Represents the output of the third module of the teacher model after passing through the feature representation head.

7. The homologous image retrieval system according to claim 5, characterized in that: The teacher model and the student model train the input data set simultaneously and in parallel until the loss function converges.

Images