A model training method and related equipment for asymmetric image retrieval

Abstract

The embodiments of the present application belong to the fields of image processing technology and digital medicine, and relate to a model training method and related equipment for asymmetric image retrieval. The present application uses a feature space mapping model instead of distillation learning to achieve asymmetric retrieval. Because the features of the large model and the small model are remapped to a new feature space, it can break through the upper limit of the accuracy of distillation learning and obtain higher asymmetric retrieval accuracy. At the same time, when the large model needs to be updated after the amount of data accumulates for a period of time, distillation learning needs to retrain all models. However, since the training of the small model in this method is relatively independent of the large model, only the large model and its mapping model can be trained and updated for parameters, saving the resources required to update the model.

Description

Technical Field

[0001] The present application relates to the fields of image processing technology and digital medicine, and in particular to a model training method and related equipment for asymmetric image retrieval. Background Art

[0002] With the development of the internet and multimedia technologies, and the widespread use of mobile devices, image retrieval has become an indispensable feature of multimedia technology. For example, medical technology supports platforms for disease-assisted diagnosis, health management, and remote consultations. Symmetric retrieval is a traditional and commonly used image retrieval technique. This involves using the same model to extract features from both the image in the retrieval database and the image to be retrieved, and then calculating the similarity between the two features. This method is the most classic and offers high accuracy, but it is also limited by the computational complexity required in practical applications. For example, it is difficult to support a model with a large number of parameters for fast image feature extraction on mobile devices. In this application context, asymmetric image retrieval technology has begun to develop.

[0003] Asymmetric image retrieval is defined as follows: Model A (typically a high-parameter model) is used to extract features from the retrieval database offline on the server, while Model B (typically a low-parameter model) is used to extract features from the image to be retrieved on the mobile device. Finally, the features extracted by the two different models (non-homologous features) are used to perform distance calculations to determine the similarity between the two images. The most critical issue facing asymmetric retrieval is that the features generated by the two models are not homologous. Therefore, how to effectively measure the non-homologous features generated from the same image sample is a key research focus.

[0004] Previous asymmetric retrieval methods mostly used distillation learning to train small models, so that the features generated by the small models are as close as possible to the features of the large model. However, the applicant found that although the larger the number of small model parameters in the traditional implementation method, the higher the accuracy, the highest accuracy will not exceed that of the stacked retrieval. This shows that the traditional asymmetric retrieval method has the problem of limited accuracy. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to propose a model training method and related equipment for asymmetric image retrieval to solve the problem of limited accuracy of traditional asymmetric retrieval methods.

[0006] In order to solve the above technical problems, the present application provides a model training method for asymmetric image retrieval, which adopts the following technical solutions:

[0007] Acquire model training data, wherein the model training data includes an original image I and a similar image K corresponding to the original image I;

[0008] Calling an initial first feature extraction model and an initial second feature extraction model;

[0009] The original image I and the similar image K are respectively input into the initial first feature extraction model to perform first feature extraction processing to obtain the first original feature f G-I and the first similarity feature f G-K ;

[0010] The original image I and the similar image K are respectively input into the initial second feature extraction model for second feature extraction processing to obtain the second original feature f Q-I and the second similarity feature f Q-K ;

[0011] The first original feature f G-I And the first similarity feature f G-K are respectively input into the first spatial mapping model for first feature mapping processing to obtain the first original mapping feature f TG-I And the first similarity mapping feature f TG-K ;

[0012] The second original feature f Q-I And the second similarity feature f Q-K are respectively input into the second spatial mapping model for second feature mapping processing to obtain the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K ;

[0013] According to the joint loss function, the first original feature f G-I , the first similarity feature f G-K The second original feature f Q-I The second similarity feature f Q-K , the first original mapping feature f TG-I , the first similarity mapping feature f TG-K , the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K Performing model training processing on the initial first feature extraction model and the initial second feature extraction model;

[0014] After the model training process traverses all the original images I and all the similar images K, a trained target first feature extraction model and a trained target second feature extraction model are obtained.

[0015] Furthermore, the joint loss function represents L total for:

[0016] L total =Lsim +L KL +L CLS

[0017] Among them, L sim Represents the mapping similarity loss function, L KL represents the intra-class similarity loss function, L CLS represents the classification loss function.

[0018] Furthermore, the mapping similarity loss function L sim Expressed as:

[0019] L sim =|f TG-I -f TQ-I | 2

[0020] Among them, f TG-I represents the first original mapping feature, f TQ-I represents the second original mapping feature.

[0021] Furthermore, the intra-class similarity loss function L KL Expressed as:

[0022] L KL =KL(Softmax(f TG-I ,y,W,b),Softmax(f TQ-K ,y,W,b))

[0023] +KL(Softmax(f TQ-I ,y,W,b),Softmax(f TG-K ,y,W,b))

[0024] Among them, KL represents the calculation of KL divergence, Softmax represents the classification function of multiple classifications during training, and f TG-I represents the first original mapping feature, f TQ-K represents the second similarity mapping feature, f TQ-I represents the second original mapping feature, f TG-K represents the first similarity mapping feature, y represents the category to which the same group of original images I and similar images K belong, and W,b represents the weight parameter of the network classification head.

[0025] Furthermore, the classification loss function L CLS Expressed as:

[0026] L CLS =-(log(Softmax(f G-I ,y,W,b))+log(Softmax(f Q-I,y,W,b)))

[0027] Among them, Softmax represents the classification function of multiple classifications during training, f G-I represents the first original feature, f Q-I represents the second original feature.

[0028] Furthermore, after the step of obtaining the trained target first feature extraction model and the target second feature extraction model after the model training process traverses all the original images I and all the similar images K, the following steps are also included:

[0029] Get the search gallery and the gallery to be searched;

[0030] Performing a third feature extraction process on all images in the search gallery according to the target first feature extraction model to obtain search gallery features;

[0031] Performing a third feature mapping process on the retrieval gallery feature according to the first spatial mapping model to obtain a retrieval gallery mapping feature;

[0032] Performing a fourth feature extraction process on all images in the to-be-searched gallery according to the target second feature extraction model to obtain features of the to-be-searched gallery;

[0033] Performing fourth feature mapping processing on the features of the to-be-retrieved image library according to the second spatial mapping model to obtain mapping features of the to-be-retrieved image library;

[0034] Constructing the retrieval library features and the index table of the retrieval library according to the inverted index method;

[0035] Calculate the feature distance between the feature of the image library to be retrieved and the feature mapping of the image library to be retrieved according to the L2 distance algorithm;

[0036] Determining, based on the feature distances, target retrieval gallery mapping features having the closest feature distances to a preset number of features corresponding to each of the to-be-retrieved gallery mapping features;

[0037] A search image corresponding to the mapping feature of the target search library is obtained in the to-be-searched library according to the index table to obtain a target search result.

[0038] In order to solve the above technical problems, the embodiment of the present application further provides a model training device for asymmetric image retrieval, which adopts the following technical solution:

[0039] A training data acquisition module, configured to acquire model training data, wherein the model training data includes an original image I and a similar image K corresponding to the original image I;

[0040] A model calling module, used to call an initial first feature extraction model and an initial second feature extraction model;

[0041] The first feature extraction module is used to input the original image I and the similar image K into the initial first feature extraction model to perform first feature extraction processing to obtain the first original feature f G-I and the first similarity feature f G-K ;

[0042] The second feature extraction module is used to input the original image I and the similar image K into the initial second feature extraction model to perform second feature extraction processing to obtain the second original feature f Q-I and the second similarity feature f Q-K ;

[0043] The first feature mapping module is used to transform the first original feature f G-I And the first similarity feature f G-K are respectively input into the first spatial mapping model for first feature mapping processing to obtain the first original mapping feature f TG-I And the first similarity mapping feature f TG-K ;

[0044] The second feature mapping module is used to transform the second original feature f Q-I And the second similarity feature f Q-K are respectively input into the second spatial mapping model for second feature mapping processing to obtain the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K ;

[0045] Model training module, for the joint loss function, the first original feature f G-I , the first similarity feature f G-K The second original feature f Q-I The second similarity feature f Q-K , the first original mapping feature f TG-I , the first similarity mapping feature f TG-K , the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K Performing model training processing on the initial first feature extraction model and the initial second feature extraction model;

[0046] The training end module is used to obtain the trained target first feature extraction model and the target second feature extraction model after the model training process traverses all the original images I and all the similar images K.

[0047] Furthermore, the device further comprises:

[0048] The image library acquisition module is used to obtain the search image library and the image library to be searched;

[0049] A third feature extraction module is used to perform a third feature extraction process on all images in the search gallery according to the target first feature extraction model to obtain search gallery features;

[0050] A third feature mapping module, configured to perform a third feature mapping process on the retrieval gallery features according to the first spatial mapping model to obtain retrieval gallery mapping features;

[0051] a fourth feature extraction module, configured to perform fourth feature extraction processing on all images in the to-be-searched gallery according to the target second feature extraction model to obtain features of the to-be-searched gallery;

[0052] a fourth feature mapping module, configured to perform fourth feature mapping processing on the features of the to-be-retrieved image library according to the second spatial mapping model to obtain mapping features of the to-be-retrieved image library;

[0053] An index table construction module, configured to construct the search library features and the index table of the search library in an inverted index manner;

[0054] A feature distance calculation module, configured to calculate the feature distance between the feature of the image library to be retrieved and the feature mapping of the image library to be retrieved according to an L2 distance algorithm;

[0055] a target retrieval gallery mapping feature confirmation module, configured to confirm, based on the feature distances, a preset number of target retrieval gallery mapping features that are closest in feature distance to each of the gallery mapping features to be retrieved;

[0056] The retrieval result acquisition module is used to obtain the retrieval image corresponding to the mapping feature of the target retrieval library in the to-be-retrieved library according to the index table, and obtain the target retrieval result.

[0057] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0058] It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the model training method applied to asymmetric image retrieval as described above are implemented.

[0059] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0060] The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the model training method applied to asymmetric image retrieval as described above.

[0061] The present application provides a model training method for asymmetric image retrieval, comprising: obtaining model training data, wherein the model training data includes an original image I and a similar image K corresponding to the original image I; calling an initial first feature extraction model and an initial second feature extraction model; inputting the original image I and the similar image K into the initial first feature extraction model respectively to perform first feature extraction processing, and obtaining a first original feature f G-I and the first similarity feature f G-K The original image I and the similar image K are respectively input into the initial second feature extraction model for second feature extraction processing to obtain the second original feature f Q-I and the second similarity feature f Q-K ; The first original feature f G-I And the first similarity feature f G-K are respectively input into the first spatial mapping model for first feature mapping processing to obtain the first original mapping feature f TG-I And the first similarity mapping feature f TG-K ; The second original feature f Q-I And the second similarity feature f Q-K are respectively input into the second spatial mapping model for second feature mapping processing to obtain the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K ; According to the joint loss function, the first original feature f G-I , the first similarity feature f G-K The second original feature f Q-I The second similarity feature f Q-K , the first original mapping feature f TG-I , the first similarity mapping feature f TG-K , the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-KThe initial first feature extraction model and the initial second feature extraction model are subjected to model training processing; after the model training processing traverses all the original images I and all the similar images K, a trained target first feature extraction model and a trained target second feature extraction model are obtained. Compared with the existing technology, the present application uses a feature space mapping model instead of distillation learning to achieve asymmetric retrieval. Because the features of the large model and the small model are remapped to a new feature space, the accuracy upper limit of distillation learning can be broken through, and higher asymmetric retrieval accuracy can be obtained. At the same time, when the large model needs to be updated after the amount of data accumulates for a period of time, distillation learning needs to retrain all models. However, since the training of the small model in this method is relatively independent of the large model, only the parameter training and update of the large model and its mapping model can be performed, saving the resources required for updating the model. In addition, the original image I and its similar image K are used as training pairs during training. The use of LsIm loss supervised learning in feature space mapping can enable the model to better explore the similarity within the sample class, and adding LKL loss in multi-classification can protect the distance relationship between similar samples during sample mapping, which is equivalent to mapping not only the features of a sample, but the entire sample and its neighborhood relationship. The model obtained in this way can greatly improve the retrieval robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0063] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0064] Figure 2 This is a flowchart of the implementation of the model training method for asymmetric image retrieval provided in Example 1 of the present application;

[0065] Figure 3 1 is a schematic diagram of a model training method for asymmetric image retrieval provided in Example 1 of the present application;

[0066] Figure 4 This is a schematic diagram of the structure of a model training device for asymmetric image retrieval provided in Example 2 of the present application;

[0067] Figure 5 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0069] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0070] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0071] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0072] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0073] Terminal devices 101, 102, and 103 can be various electronic devices with display screens and support web browsing, including but not limited to smartphones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV), laptop computers, desktop computers, etc.

[0074] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0075] It should be noted that the model training method for asymmetric image retrieval provided in the embodiments of the present application is generally executed by a server / terminal device. Accordingly, the model training device for asymmetric image retrieval is generally set in the server / terminal device.

[0076] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0077] Example 1

[0078] Continue to refer Figure 2 , which shows a flow chart of an embodiment of a model training method for asymmetric image retrieval according to the present application. The model training method for asymmetric image retrieval includes: step S201, step S202, step S203, step S204, step S205, step S206, step S207, and step S208.

[0079] In step S201 , model training data is obtained, where the model training data includes an original image I and a similar image K corresponding to the original image I.

[0080] In the medical application scenario of the present application, the original image may be a medical image, and the object contained in the original image belongs to the type of lesion, that is, the part of the body where the disease occurs. Medical images refer to images of internal tissues obtained non-invasively for medical treatment or medical research, such as images of the stomach, abdomen, heart, knee, and brain, such as CT (Computed Tomography), MRI (Magnetic Resonance Imaging), US (ultrasonic), X-ray images, electroencephalograms, and optical photographic images generated by medical instruments.

[0081] In step S202, an initial first feature extraction model and an initial second feature extraction model are called.

[0082] In step S203, the original image I and the similar image K are respectively input into the initial first feature extraction model to perform first feature extraction processing to obtain the first original feature f G-I and the first similarity feature f I-K .

[0083] In step S204, the original image I and the similar image K are respectively input into the initial second feature extraction model for second feature extraction processing to obtain the second original feature f Q-I and the second similarity feature f Q-K .

[0084] In step S205, the first original feature f G-I and the first similarity feature f G-K are respectively input into the first spatial mapping model for first feature mapping processing to obtain the first original mapping feature f TG-I And the first similarity mapping feature f TG-K .

[0085] In step S206, the second original feature f Q-I and the second similarity feature f Q-K are respectively input into the second spatial mapping model for second feature mapping processing to obtain the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K .

[0086] In step S207, according to the joint loss function, the first original feature f G-I , the first similarity feature f G-K , the second original feature f Q-I , the second similarity feature f Q-K , the first original mapping feature f TG-I , the first similarity mapping feature f TG-K , the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K The initial first feature extraction model and the initial second feature extraction model are subjected to model training processing.

[0087] In step S208 , after the model training process traverses all original images I and all similar images K, a trained target first feature extraction model and a trained target second feature extraction model are obtained.

[0088] In the embodiments of this application, Figure 3 The model training diagram of the model training method for asymmetric image retrieval provided in the first embodiment of the present application is shown. Assume that the original image I is any image that needs to be registered in the retrieval library, and image K is k similar images (the number of images is greater than or equal to 1, generally the top 10) with similar content to the original image I. Images I and K are respectively subjected to forward reasoning of the large model G and the small model Q to obtain the large model feature f G-I 、f G-K and small model features f Q-I 、f Q-KThe two sets of features are then passed through the forward reasoning of the feature space mapping models TG and TQ to obtain the mapped features f TG-I 、f TG-K and f TQ-I 、f TQ-K (The dimension of the mapped features remains consistent with the original features). According to the settings of this method, the two sets of mapped features obtained should fall into the same category in the same feature space (as shown in the figure, the large circle represents the entire feature space, and the dotted ellipse represents the range of the same category in the same feature space).

[0089] In an embodiment of the present application, a model training method for asymmetric image retrieval is provided, comprising: obtaining model training data, wherein the model training data includes an original image I and a similar image K corresponding to the original image I; calling an initial first feature extraction model and an initial second feature extraction model; inputting the original image I and the similar image K into the initial first feature extraction model respectively to perform first feature extraction processing, and obtaining a first original feature f G-I and the first similarity feature f G-K ; Input the original image I and the similar image K into the initial second feature extraction model for second feature extraction processing to obtain the second original feature f Q-I and the second similarity feature f Q-K ; The first original feature f G-I and the first similarity feature f G-K are respectively input into the first spatial mapping model for first feature mapping processing to obtain the first original mapping feature f TG-I And the first similarity mapping feature f TG-K ; The second original feature f Q-I and the second similarity feature f Q-K are respectively input into the second spatial mapping model for second feature mapping processing to obtain the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K ; According to the joint loss function, the first original feature f G-I , the first similarity feature f G-K , the second original feature f Q-I , the second similarity feature f Q-K , the first original mapping feature f TG-I , the first similarity mapping feature f TG-K , the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-KThe initial first feature extraction model and the initial second feature extraction model are subjected to model training processing; after the model training processing traverses all original images I and all similar images K, the trained target first feature extraction model and the target second feature extraction model are obtained. Compared with the existing technology, the present application uses a feature space mapping model instead of distillation learning to achieve asymmetric retrieval. Because the features of the large model and the small model are remapped to a new feature space, the accuracy upper limit of distillation learning can be broken through, and higher asymmetric retrieval accuracy can be obtained. At the same time, when the large model needs to be updated after the amount of data accumulates for a period of time, distillation learning needs to retrain all models. However, since the training of the small model in this method is relatively independent of the large model, only the parameter training and update of the large model and its mapping model can be performed, saving the resources required for updating the model. In addition, the original image I and its similar image K are used as training pairs during training. The use of LsIm loss supervised learning in feature space mapping can enable the model to better explore the similarity within the sample class, and adding LKL loss in multi-classification can protect the distance relationship between similar samples during sample mapping, which is equivalent to mapping not only the features of a sample, but the entire sample and its neighborhood relationship. The model obtained in this way can greatly improve the retrieval robustness.

[0090] In some optional implementations of the embodiments of the present application, the joint loss function represents L total for:

[0091] L total =L sim +L KL +L CLS

[0092] Among them, L sim Represents the mapping similarity loss function, L KL represents the intra-class similarity loss function, L CLS represents the classification loss function.

[0093] In some optional implementations of the embodiments of the present application, the mapping similarity loss function L sim Expressed as:

[0094] L sim =|f TG-I -f TQ-I | 2

[0095] Among them, f TG-I represents the first original mapping feature, f TQ-I Represents the second original mapping feature.

[0096] In some optional implementations of the present application, the intra-class similarity loss function L Kl Expressed as:

[0097] L KL =KL(Softmax(f TG-I ,y,W,b),Softmax(f TQ-K ,y,W,b))

[0098] +KL(Softmax(f TQ-I ,y,W,b),Softmax(f TG-K ,y,W,b))

[0099] Among them, KL represents the calculation of KL divergence, Softmax represents the classification function of multiple classifications during training, and f TG-I represents the first original mapping feature, f TQ-K represents the second similarity mapping feature, f TQ-I represents the second original mapping feature, f TG-K Represents the first similarity mapping feature, y represents the category to which the same group of original images I and similar images K belong, and W,b represents the weight parameter of the network classification head.

[0100] In some optional implementations of the present application, the classification loss function L CLS Expressed as:

[0101] L CLS =-(log(Softmax(f G-I ,y,W,b))+log(Softmax(f Q-I ,y,W,b)))

[0102] Among them, Softmax represents the classification function of multiple classifications during training, f G-I represents the first original feature, f Q-I Represents the second original feature.

[0103] In some optional implementations of the embodiments of the present application, after step S208, the following steps are further included:

[0104] Get the search gallery and the gallery to be searched;

[0105] Performing a third feature extraction process on all images in the search gallery according to the target first feature extraction model to obtain search gallery features;

[0106] Performing a third feature mapping process on the retrieval gallery feature according to the first spatial mapping model to obtain a retrieval gallery mapping feature;

[0107] Performing fourth feature extraction processing on all images in the to-be-retrieved gallery according to the target second feature extraction model to obtain features of the to-be-retrieved gallery;

[0108] Performing fourth feature mapping processing on the features of the image library to be retrieved according to the second spatial mapping model to obtain mapping features of the image library to be retrieved;

[0109] Constructing the retrieval library features and the index table of the retrieval library according to the inverted index method;

[0110] Calculate the feature distance between the features of the image library to be retrieved and the mapping features of the image library to be retrieved according to the L2 distance algorithm;

[0111] Determine, based on the feature distance, a preset number of target retrieval gallery mapping features that are closest to each to-be-retrieved gallery mapping feature;

[0112] In the to-be-retrieved image library, a retrieval image corresponding to the mapping features of the target retrieval image library is obtained according to the index table to obtain the target retrieval result.

[0113] In this embodiment of the application, after the training phase is completed, when using it, we first have a search library Θ and a to-be-searched library Φ. The goal is to find all images with similar content in the search library Θ for each image in the to-be-searched library Φ. The specific usage process is as follows:

[0114] ① First, use the large model G to extract the features of all images in the gallery Θ, and then generate the mapped features fTG-Θ after mapping by the mapping model TG and record them in the retrieval library;

[0115] ② Then use the small model Q to extract the features of each image in the to-be-retrieved library Φ, and generate the small model shape mapping feature fTQ-Φ after mapping by the mapping model TQ;

[0116] ③ Use the inverted index method to build an index table, and use the L2 distance as the metric to calculate the distance between each feature pair in fTQ-Φ and fTG-Θ, and finally calculate the preset number of features that are closest to each feature in fTQ-Φ;

[0117] ④ Obtain search results based on the index table query.

[0118] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0119] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0120] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware via computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0121] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0122] Example 2

[0123] Further references Figure 4 , as a response to the above Figure 2 The present application provides an embodiment of a model training device for asymmetric image retrieval, which is similar to the embodiment of the present invention. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0124] like Figure 4 As shown, the model training device 200 applied to asymmetric image retrieval in an embodiment of the present application includes:

[0125] A training data acquisition module 210 is configured to acquire model training data, wherein the model training data includes an original image I and a similar image K corresponding to the original image I;

[0126] A model calling module 220 is used to call an initial first feature extraction model and an initial second feature extraction model;

[0127] The first feature extraction module 230 is used to input the original image I and the similar image K into the initial first feature extraction model to perform first feature extraction processing to obtain the first original feature f G-I and the first similarity feature f G-K ;

[0128] The second feature extraction module 240 is used to input the original image I and the similar image K into the initial second feature extraction model to perform second feature extraction processing to obtain the second original feature f Q-I and the second similarity feature f Q-K ;

[0129] The first feature mapping module 250 is used to transform the first original feature f G-I and the first similarity feature f G-K are respectively input into the first spatial mapping model for first feature mapping processing to obtain the first original mapping feature f TG-I And the first similarity mapping feature f TG-K ;

[0130] The second feature mapping module 260 is used to transform the second original feature f Q-I and the second similarity feature f Q-K are respectively input into the second spatial mapping model for second feature mapping processing to obtain the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K ;

[0131] The model training module 270 is used to train the model according to the joint loss function and the first original feature f G-I , the first similarity feature f G-K , the second original feature f Q-I , the second similarity feature f Q-K , the first original mapping feature f TG-I , the first similarity mapping feature f TG-K , the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K Performing model training processing on the initial first feature extraction model and the initial second feature extraction model;

[0132] The training end module 280 is used to obtain the trained target first feature extraction model and the target second feature extraction model after the model training process traverses all original images I and all similar images K.

[0133] In an embodiment of the present application, a model training device 200 for asymmetric image retrieval is provided, comprising: a training data acquisition module 210 for acquiring model training data, wherein the model training data includes an original image I and a similar image K corresponding to the original image I; a model calling module 220 for calling an initial first feature extraction model and an initial second feature extraction model; a first feature extraction module 230 for inputting the original image I and the similar image K into the initial first feature extraction model for first feature extraction processing, and obtaining a first original feature f G-I and the first similarity feature f G-K The second feature extraction module 240 is used to input the original image I and the similar image K into the initial second feature extraction model for second feature extraction processing to obtain the second original feature f Q-I and the second similarity feature f Q-K ; The first feature mapping module 250 is used to transform the first original feature f G-I and the first similarity feature f G-K are respectively input into the first spatial mapping model for first feature mapping processing to obtain the first original mapping feature f TG-I And the first similarity mapping feature f TG-K ; The second feature mapping module 260 is used to convert the second original feature f Q-I and the second similarity feature f Q-K are respectively input into the second spatial mapping model for second feature mapping processing to obtain the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K ; Model training module 270, for according to the joint loss function, the first original feature f G-I , the first similarity feature f G-K , the second original feature f Q-I , the second similarity feature f Q-K , the first original mapping feature f TG-I , the first similarity mapping feature f TG-K , the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-KThe initial first feature extraction model and the initial second feature extraction model are subjected to model training processing; the training end module 280 is used to obtain the trained target first feature extraction model and the target second feature extraction model after the model training processing traverses all original images I and all similar images K. Compared with the existing technology, the present application uses a feature space mapping model instead of distillation learning to achieve asymmetric retrieval. Because the features of the large model and the small model are remapped to a new feature space, the accuracy upper limit of distillation learning can be broken through, and higher asymmetric retrieval accuracy can be obtained. At the same time, when the large model needs to be updated after the amount of data accumulates for a period of time, distillation learning needs to retrain all models. However, since the training of the small model in this method is relatively independent of the large model, only the parameter training and update of the large model and its mapping model can be performed, saving the resources required for updating the model. In addition, the original image I and its similar image K are used as training pairs during training. The use of LsIm loss supervised learning in feature space mapping can enable the model to better explore the similarity within the sample class, and adding LKL loss in multi-classification can protect the distance relationship between similar samples during sample mapping, which is equivalent to mapping not only the features of a sample, but the entire sample and its neighborhood relationship. The model obtained in this way can greatly improve the retrieval robustness.

[0134] In some optional implementations of the embodiments of the present application, the apparatus further includes:

[0135] The image library acquisition module is used to obtain the search image library and the image library to be searched;

[0136] A third feature extraction module is used to perform a third feature extraction process on all images in the search gallery according to the target first feature extraction model to obtain the search gallery features;

[0137] A third feature mapping module is used to perform a third feature mapping process on the retrieval gallery features according to the first spatial mapping model to obtain retrieval gallery mapping features;

[0138] A fourth feature extraction module is used to perform fourth feature extraction processing on all images in the to-be-searched gallery according to the target second feature extraction model to obtain features of the to-be-searched gallery;

[0139] A fourth feature mapping module, configured to perform fourth feature mapping processing on the features of the image library to be retrieved according to the second spatial mapping model to obtain mapping features of the image library to be retrieved;

[0140] An index table construction module is used to construct the retrieval library features and the index table of the retrieval library according to the inverted index method;

[0141] A feature distance calculation module is used to calculate the feature distance between the features of the image library to be retrieved and the mapping features of the image library to be retrieved according to the L2 distance algorithm;

[0142] A target retrieval gallery mapping feature confirmation module is used to confirm, based on feature distances, a preset number of target retrieval gallery mapping features that are closest in feature distance to each to-be-retrieved gallery mapping feature;

[0143] The retrieval result acquisition module is used to obtain the retrieval image corresponding to the mapping feature of the target retrieval library in the to-be-retrieved library according to the index table, and obtain the target retrieval result.

[0144] To solve the above technical problems, the present application also provides a computer device. Figure 5 , Figure 5 This is a basic structural block diagram of the computer device according to an embodiment of the present application.

[0145] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected through a system bus. It should be noted that the figure only shows the computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0146] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0147] The memory 310 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory 310 may be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 may also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the memory 310 may also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store an operating system and various application software installed on the computer device 300, such as computer-readable instructions for a model training method for asymmetric image retrieval. In addition, the memory 310 can also be used to temporarily store various types of data that have been output or are about to be output.

[0148] In some embodiments, the processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to execute computer-readable instructions or process data stored in the memory 310, such as executing computer-readable instructions for the model training method for asymmetric image retrieval.

[0149] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0150] The computer device provided in the present application uses a feature space mapping model instead of distillation learning to achieve asymmetric retrieval. Because the features of the large model and the small model are remapped to a new feature space, the accuracy limit of distillation learning can be broken through, and higher asymmetric retrieval accuracy can be obtained. At the same time, when the large model needs to be updated after the amount of data accumulates for a period of time, distillation learning needs to retrain all models. However, since the training of the small model is relatively independent of the large model in this method, only the parameters of the large model and its mapping model can be trained and updated, saving the resources required for updating the model. In addition, the original image I and its similar image K are used as training pairs during training, and the LsIm loss supervised learning is used in the feature space mapping, which can enable the model to better explore the similarity within the sample class, and adding LKL loss in multi-classification can protect the distance relationship between similar samples during sample mapping, which is equivalent to mapping not only the features of a sample, but the entire sample and its neighborhood relationship. The model obtained in this way can greatly improve the retrieval robustness.

[0151] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the model training method applied to asymmetric image retrieval as described above.

[0152] The computer-readable storage medium provided in the present application uses a feature space mapping model instead of distillation learning to achieve asymmetric retrieval. Because the features of the large model and the small model are remapped to a new feature space, the accuracy limit of distillation learning can be broken through, and higher asymmetric retrieval accuracy can be obtained. At the same time, when the large model needs to be updated after the amount of data has accumulated for a period of time, distillation learning needs to retrain all models. However, since the training of the small model in this method is relatively independent of the large model, parameter training and updating can only be performed on the large model and its mapping model, saving the resources required for updating the model. In addition, the original image I and its similar image K are used as training pairs during training, and the LsIm loss supervised learning is used in the feature space mapping, which can enable the model to better explore the similarity within the sample class, and adding LKL loss in multi-classification can protect the distance relationship between similar samples during sample mapping, which is equivalent to mapping not only the features of a sample, but the entire sample and its neighborhood relationship. The model obtained in this way can greatly improve the retrieval robustness.

[0153] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0154] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A model training method for asymmetric image retrieval, characterized in that: The steps include: Acquire model training data, wherein the model training data includes an original image I and a similar image K corresponding to the original image I; Call the initial first feature extraction model G and the initial second feature extraction model Q, where G is the large model and Q is the small model; The original image I and the similar image K are respectively input into the initial first feature extraction model to perform first feature extraction processing to obtain the first original feature and the first similarity feature ; The original image I and the similar image K are respectively input into the initial second feature extraction model for second feature extraction processing to obtain the second original feature and the second similar feature ; The first original feature And the first similar feature Input them into the first spatial mapping model TG for the first feature mapping process to obtain the first original mapping feature and the first similarity mapping feature ; The second original feature And the second similar feature Input them into the second spatial mapping model TQ for the second feature mapping process to obtain the second original mapping feature and the second similarity mapping feature ; According to the joint loss function, the first original feature The first similar feature The second original feature The second similar feature , the first original mapping feature , the first similarity mapping feature , the second original mapping feature And the second similarity mapping feature Performing model training processing on the initial first feature extraction model and the initial second feature extraction model; After the model training process traverses all the original images I and all the similar images K, a trained target first feature extraction model and a trained target second feature extraction model are obtained; After the model training process traverses all the original images I and all the similar images K to obtain the trained target first feature extraction model and the trained target second feature extraction model, the following steps are also included: Get the search gallery and the gallery to be searched; Performing a third feature extraction process on all images in the search gallery according to the target first feature extraction model to obtain search gallery features; Performing a third feature mapping process on the retrieval gallery feature according to the first spatial mapping model to obtain a retrieval gallery mapping feature; Performing a fourth feature extraction process on all images in the to-be-searched gallery according to the target second feature extraction model to obtain features of the to-be-searched gallery; Performing fourth feature mapping processing on the features of the to-be-retrieved image library according to the second spatial mapping model to obtain mapping features of the to-be-retrieved image library; Constructing the retrieval library features and the index table of the retrieval library according to the inverted index method; Calculate the feature distance between the feature of the image library to be retrieved and the feature mapping of the image library to be retrieved according to the L2 distance algorithm; Determining, based on the feature distances, target retrieval gallery mapping features having the closest feature distances to a preset number of features corresponding to each of the to-be-retrieved gallery mapping features; A search image corresponding to the mapping feature of the target search library is obtained in the to-be-searched library according to the index table to obtain a target search result.

2. The model training method for asymmetric image retrieval according to claim 1, characterized in that: The joint loss function is expressed as for: in, represents the mapping similarity loss function, represents the intra-class similarity loss function, represents the classification loss function.

3. The model training method for asymmetric image retrieval according to claim 2, characterized in that: The mapping similarity loss function Expressed as: in, represents the first original mapping feature, represents the second original mapping feature.

4. The model training method for asymmetric image retrieval according to claim 2, characterized in that: The intra-class similarity loss function Expressed as: in, Represents calculation Divergence, Represents the classification function of multiple classifications during training, represents the first original mapping feature, represents the second similarity mapping feature, represents the second original mapping feature, represents the first similarity mapping feature, Indicates the categories to which the same set of original images I and similar images K belong, Represents the weight parameters of the network classification head.

5. The model training method for asymmetric image retrieval according to claim 2, characterized in that: The classification loss function Expressed as: in, Represents the classification function of multiple classifications during training, represents the first original feature, represents the second original feature, Indicates the categories to which the same set of original images I and similar images K belong, Represents the weight parameters of the network classification head.

6. A model training device for asymmetric image retrieval, characterized in that: include: A training data acquisition module, configured to acquire model training data, wherein the model training data includes an original image I and a similar image K corresponding to the original image I; The model calling module is used to call the initial first feature extraction model G and the initial second feature extraction model Q, where G is a large model and Q is a small model; The first feature extraction module is used to input the original image I and the similar image K into the initial first feature extraction model to perform first feature extraction processing to obtain the first original feature and the first similarity feature ; The second feature extraction module is used to input the original image I and the similar image K into the initial second feature extraction model to perform second feature extraction processing to obtain the second original feature. and the second similar feature ; The first feature mapping module is used to transform the first original feature And the first similar feature Input them into the first spatial mapping model TG for the first feature mapping process to obtain the first original mapping feature and the first similarity mapping feature ; The second feature mapping module is used to transform the second original feature And the second similar feature Input them into the second spatial mapping model TQ for the second feature mapping process to obtain the second original mapping feature and the second similarity mapping feature ; Model training module, for the joint loss function, the first original feature The first similar feature The second original feature The second similar feature , the first original mapping feature , the first similarity mapping feature , the second original mapping feature And the second similarity mapping feature Performing model training processing on the initial first feature extraction model and the initial second feature extraction model; A training end module, configured to obtain a trained target first feature extraction model and a trained target second feature extraction model after the model training process traverses all the original images I and all the similar images K; The device further comprises: The image library acquisition module is used to obtain the search image library and the image library to be searched; A third feature extraction module is used to perform a third feature extraction process on all images in the search gallery according to the target first feature extraction model to obtain search gallery features; A third feature mapping module, configured to perform a third feature mapping process on the retrieval gallery features according to the first spatial mapping model to obtain retrieval gallery mapping features; a fourth feature extraction module, configured to perform fourth feature extraction processing on all images in the to-be-searched gallery according to the target second feature extraction model to obtain features of the to-be-searched gallery; a fourth feature mapping module, configured to perform fourth feature mapping processing on the features of the to-be-retrieved image library according to the second spatial mapping model to obtain mapping features of the to-be-retrieved image library; An index table construction module, configured to construct the search library features and the index table of the search library in an inverted index manner; A feature distance calculation module, configured to calculate the feature distance between the feature of the image library to be retrieved and the feature mapping of the image library to be retrieved according to an L2 distance algorithm; a target retrieval gallery mapping feature confirmation module, configured to confirm, based on the feature distances, a preset number of target retrieval gallery mapping features that are closest in feature distance to each of the gallery mapping features to be retrieved; The retrieval result acquisition module is used to obtain the retrieval image corresponding to the mapping feature of the target retrieval library in the to-be-retrieved library according to the index table, and obtain the target retrieval result.

7. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the model training method for asymmetric image retrieval according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the model training method for asymmetric image retrieval according to any one of claims 1 to 5.

Images