Model training method applied to asymmetric image retrieval and related equipment

Abstract

The embodiment of the invention belongs to the technical field of image processing and the field of digital medical treatment, and relates to a model training method applied to asymmetric image retrieval and related equipment. According to the method, the asymmetric retrieval is realized by using the feature space mapping model instead of distillation learning, and the features of the large model and the small model are remapped to a new feature space, so that the precision upper limit of distillation learning can be broken through, and higher asymmetric retrieval precision can be obtained; meanwhile, when the large model needs to be updated after the data volume is accumulated for a period of time, distillation learning needs to retrain all models, and due to the fact that training of the small model and the large model are relatively independent in the method, parameter training and updating can be conducted only for the large model and the mapping model of the large model, and resources needed for updating the models are saved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology and digital medicine, and in particular to a model training method and related equipment for asymmetric image retrieval. Background Art

[0002] With the development of the Internet and multimedia technology and the extensive use of mobile devices, image retrieval has become an indispensable function in multimedia technology. For example, medical technology can support disease-assisted diagnosis, health management, remote consultation and other functions. The traditional commonly used image retrieval technology is symmetric retrieval, that is, the same model is used to extract features from the images in the retrieval library and the images to be retrieved and calculate the similarity between the two features. This method is the most classic and has high accuracy, but it is also limited by the amount of calculation in actual use. For example, it is difficult to support a model with a large number of parameters to perform very fast image feature extraction in mobile devices. In this application context, asymmetric image retrieval technology has begun to develop gradually.

[0003] Asymmetric image retrieval is defined as: using model A (usually a large-parameter model) to extract retrieval library features offline on the server side, and using model B (usually a small-parameter model) to extract features of the image to be retrieved on the mobile side, and finally using the features extracted by the two different models (non-homologous features) to perform distance calculations to determine the similarity between the two images. The most critical problem facing asymmetric retrieval is that the features generated by the two models are not homologous. How to measure the non-homologous features generated by the same image sample is the research focus of asymmetric retrieval.

[0004] Previous asymmetric retrieval methods mostly use distillation learning to train small models so that the features generated by the small model are as close as possible to the features of the large model. However, the applicant found that although the larger the number of small model parameters in the traditional implementation method, the higher the accuracy, the highest accuracy will not exceed that of the stacked retrieval. This shows that the traditional asymmetric retrieval method has the problem of limited accuracy. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a model training method and related equipment for asymmetric image retrieval to solve the problem of limited accuracy of traditional asymmetric retrieval methods.

[0006] In order to solve the above technical problems, the embodiment of the present application provides a model training method for asymmetric image retrieval, which adopts the following technical solution:

[0007] Acquire model training data, wherein the model training data includes an original image I and a similar image K corresponding to the original image I;

[0008] Calling an initial first feature extraction model and an initial second feature extraction model;

[0009] The original image I and the similar image K are respectively input into the initial first feature extraction model to perform first feature extraction processing to obtain the first original feature f G-I and the first similarity feature f G-K ;

[0010] The original image I and the similar image K are respectively input into the initial second feature extraction model for second feature extraction processing to obtain the second original feature f Q-I and the second similarity feature f Q-K ;

[0011] The first original feature f G-I And the first similar feature f G-K are respectively input into the first spatial mapping model for first feature mapping processing to obtain the first original mapping feature f TG-I And the first similarity mapping feature f TG-K ;

[0012] The second original feature f Q-I And the second similar feature f Q-K are respectively input into the second spatial mapping model for second feature mapping processing to obtain the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K ;

[0013] According to the joint loss function, the first original feature f G-I , the first similar feature f G-K The second original feature f Q-I The second similar feature f Q-K , the first original mapping feature f TG-I , the first similarity mapping feature f TG-K , the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K Performing model training processing on the initial first feature extraction model and the initial second feature extraction model;

[0014] After the model training process traverses all the original images I and all the similar images K, a trained target first feature extraction model and a trained target second feature extraction model are obtained.

[0015] Furthermore, the joint loss function represents L total for:

[0016] L total =Lsim +L KL +L CLS

[0017] Among them, L sim represents the mapping similarity loss function, L KL represents the intra-class similarity loss function, L CLS represents the classification loss function.

[0018] Furthermore, the mapping similarity loss function L sim It is expressed as:

[0019] L sim =|f TG-I -f TQ-I | 2

[0020] Among them, f TG-I represents the first original mapping feature, f TQ-I represents the second original mapping feature.

[0021] Furthermore, the intra-class similarity loss function L KL It is expressed as:

[0022] L KL =KL(Softmax(f TG-I ,y,W,b),Softmax(f TQ-K ,y,W,b))

[0023] +KL(Softmax(f TQ-I ,y,W,b),Softmax(f TG-K ,y,W,b))

[0024] Among them, KL represents the calculation of KL divergence, Softmax represents the classification function of multiple classifications during training, and f TG-I represents the first original mapping feature, f TQ-K represents the second similarity mapping feature, f TQ-I represents the second original mapping feature, f TG-K represents the first similarity mapping feature, y represents the category to which the same group of original images I and similar images K belong, and W,b represents the weight parameter of the network classification head.

[0025] Furthermore, the classification loss function L CLS It is expressed as:

[0026] L CLS =-(log(Softmax(f G-I ,y,W,b))+log(Softmax(f Q-I,y,W,b)))

[0027] Among them, Softmax represents the classification function of multiple classifications during training, f G-I represents the first original feature, f Q-I represents the second original feature.

[0028] Furthermore, after the step of obtaining the trained target first feature extraction model and the target second feature extraction model after the model training process traverses all the original images I and all the similar images K, the following steps are also included:

[0029] Get the search gallery and the gallery to be searched;

[0030] Performing a third feature extraction process on all images in the search gallery according to the target first feature extraction model to obtain search gallery features;

[0031] Performing a third feature mapping process on the retrieval gallery feature according to the first spatial mapping model to obtain a retrieval gallery mapping feature;

[0032] Performing fourth feature extraction processing on all images in the to-be-searched gallery according to the target second feature extraction model to obtain features of the to-be-searched gallery;

[0033] Performing fourth feature mapping processing on the features of the to-be-searched gallery according to the second spatial mapping model to obtain mapping features of the to-be-searched gallery;

[0034] Constructing the retrieval gallery features and the index table of the retrieval gallery in an inverted index manner;

[0035] Calculate the feature distance between the feature of the to-be-searched gallery and the feature of the to-be-searched gallery mapping according to the L2 distance algorithm;

[0036] Determine, according to the feature distance, a preset number of target retrieval gallery mapping features with the closest feature distances corresponding to each of the to-be-retrieved gallery mapping features;

[0037] A search image corresponding to the mapping feature of the target search library is obtained in the to-be-searched library according to the index table to obtain a target search result.

[0038] In order to solve the above technical problems, the embodiment of the present application further provides a model training device for asymmetric image retrieval, which adopts the following technical solution:

[0039] A training data acquisition module, used to acquire model training data, wherein the model training data includes an original image I and a similar image K corresponding to the original image I;

[0040] A model calling module, used for calling an initial first feature extraction model and an initial second feature extraction model;

[0041] The first feature extraction module is used to input the original image I and the similar image K into the initial first feature extraction model to perform first feature extraction processing to obtain the first original feature f G-I and the first similarity feature f G-K ;

[0042] The second feature extraction module is used to input the original image I and the similar image K into the initial second feature extraction model to perform second feature extraction processing to obtain the second original feature f Q-I and the second similarity feature f Q-K ;

[0043] The first feature mapping module is used to transform the first original feature f G-I And the first similar feature f G-K are respectively input into the first spatial mapping model for first feature mapping processing to obtain the first original mapping feature f TG-I And the first similarity mapping feature f TG-K ;

[0044] The second feature mapping module is used to transform the second original feature f Q-I And the second similar feature f Q-K are respectively input into the second spatial mapping model for second feature mapping processing to obtain the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K ;

[0045] A model training module is used to train the model according to the joint loss function and the first original feature f G-I , the first similar feature f G-K The second original feature f Q-I The second similar feature f Q-K , the first original mapping feature f TG-I , the first similarity mapping feature f TG-K , the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K Performing model training processing on the initial first feature extraction model and the initial second feature extraction model;

[0046] The training end module is used to obtain the trained target first feature extraction model and the target second feature extraction model after the model training process traverses all the original images I and all the similar images K.

[0047] Furthermore, the device also includes:

[0048] The gallery acquisition module is used to acquire the search gallery and the gallery to be searched;

[0049] A third feature extraction module, used for performing a third feature extraction process on all images in the search gallery according to the target first feature extraction model to obtain search gallery features;

[0050] A third feature mapping module, configured to perform a third feature mapping process on the retrieval gallery feature according to the first spatial mapping model to obtain a retrieval gallery mapping feature;

[0051] A fourth feature extraction module, used for performing fourth feature extraction processing on all images in the to-be-searched gallery according to the target second feature extraction model to obtain features of the to-be-searched gallery;

[0052] A fourth feature mapping module, used for performing a fourth feature mapping process on the features of the to-be-searched gallery according to the second spatial mapping model to obtain mapping features of the to-be-searched gallery;

[0053] An index table construction module, used to construct the search library features and the index table of the search library according to the inverted index method;

[0054] A feature distance calculation module, used for calculating the feature distance between the feature of the to-be-searched gallery and the feature mapping of the to-be-searched gallery according to an L2 distance algorithm;

[0055] A target retrieval gallery mapping feature confirmation module is used to confirm, according to the feature distance, a preset number of target retrieval gallery mapping features that are closest in feature distance to each of the gallery mapping features to be retrieved;

[0056] The retrieval result acquisition module is used to obtain the retrieval image corresponding to the mapping feature of the target retrieval library in the to-be-retrieved library according to the index table, and obtain the target retrieval result.

[0057] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0058] It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the model training method applied to asymmetric image retrieval as described above are implemented.

[0059] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0060] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the model training method applied to asymmetric image retrieval as described above are implemented.

[0061] The present application provides a model training method for asymmetric image retrieval, comprising: obtaining model training data, wherein the model training data includes an original image I and a similar image K corresponding to the original image I; calling an initial first feature extraction model and an initial second feature extraction model; inputting the original image I and the similar image K into the initial first feature extraction model respectively to perform first feature extraction processing, and obtaining a first original feature f G-I and the first similarity feature f G-K ; Input the original image I and the similar image K into the initial second feature extraction model for second feature extraction processing to obtain the second original feature f Q-I and the second similarity feature f Q-K ; The first original feature f G-I And the first similar feature f G-K are respectively input into the first spatial mapping model for first feature mapping processing to obtain the first original mapping feature f TG-I And the first similarity mapping feature f TG-K ; The second original feature f Q-I And the second similar feature f Q-K are respectively input into the second spatial mapping model for second feature mapping processing to obtain the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K ; According to the joint loss function, the first original feature f G-I , the first similar feature f G-K The second original feature f Q-I The second similar feature f Q-K , the first original mapping feature f TG-I , the first similarity mapping feature f TG-K , the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-KThe initial first feature extraction model and the initial second feature extraction model are subjected to model training processing; after the model training processing traverses all the original images I and all the similar images K, a trained target first feature extraction model and a trained target second feature extraction model are obtained. Compared with the prior art, the present application uses a feature space mapping model instead of distillation learning to achieve asymmetric retrieval. Because the features of the large model and the small model are remapped to a new feature space, the accuracy limit of distillation learning can be broken through, and higher asymmetric retrieval accuracy can be obtained. At the same time, when the large model needs to be updated after the amount of data has accumulated for a period of time, distillation learning needs to retrain all models. Since the training of the small model is relatively independent of the large model in the present method, only the parameters of the large model and its mapping model can be trained and updated, saving the resources required for updating the model. In addition, the original image I and its similar image K are used as training pairs during training. The use of LsIm loss supervised learning in feature space mapping can enable the model to better mine the similarities within the sample class, and adding LKL loss in multi-classification can protect the distance relationship between similar samples when mapping samples, which is equivalent to mapping not only the features of a sample, but the mapping of the entire sample and its neighborhood relationship. The model obtained in this way can greatly improve the robustness of retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0063] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0064] Figure 2 It is a flow chart for implementing the model training method for asymmetric image retrieval provided in the first embodiment of the present application;

[0065] Figure 3 It is a model training schematic diagram of a model training method applied to asymmetric image retrieval provided in Example 1 of the present application;

[0066] Figure 4 It is a structural schematic diagram of a model training device for asymmetric image retrieval provided in Example 2 of the present application;

[0067] Figure 5 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0069] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0070] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0071] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0072] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0073] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0074] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0075] It should be noted that the model training method for asymmetric image retrieval provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the model training device for asymmetric image retrieval is generally set in the server / terminal device.

[0076] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0077] Embodiment 1

[0078] Continue to refer Figure 2 , shows a flow chart of an embodiment of a model training method for asymmetric image retrieval according to the present application. The model training method for asymmetric image retrieval includes: step S201, step S202, step S203, step S204, step S205, step S206, step S207 and step S208.

[0079] In step S201 , model training data is obtained, where the model training data includes an original image I and a similar image K corresponding to the original image I.

[0080] In the medical application scenario of the present application, the original image may be a medical image, and the type of object contained in the original image is a lesion, that is, a part of the body where a disease occurs. Medical images refer to images of internal tissues obtained in a non-invasive manner for medical treatment or medical research, such as images of the stomach, abdomen, heart, knees, and brain, such as CT (Computed Tomography), MRI (Magnetic Resonance Imaging), US (ultrasonic), X-ray images, electroencephalograms, and optical photographic lights generated by medical instruments.

[0081] In step S202, an initial first feature extraction model and an initial second feature extraction model are called.

[0082] In step S203, the original image I and the similar image K are respectively input into the initial first feature extraction model for first feature extraction processing to obtain the first original feature f G-I and the first similarity feature f I-K .

[0083] In step S204, the original image I and the similar image K are respectively input into the initial second feature extraction model for second feature extraction processing to obtain the second original feature f Q-I and the second similarity feature f Q-K .

[0084] In step S205, the first original feature f G-I and the first similarity feature f G-K are respectively input into the first spatial mapping model for first feature mapping processing to obtain the first original mapping feature f TG-I And the first similarity mapping feature f TG-K .

[0085] In step S206, the second original feature f Q-I and the second similarity feature f Q-K are respectively input into the second spatial mapping model for second feature mapping processing to obtain the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K .

[0086] In step S207, according to the joint loss function, the first original feature f G-I , the first similarity feature f G-K , the second original feature f Q-I , the second similarity feature f Q-K , the first original mapping feature f TG-I , the first similarity mapping feature f TG-K , the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K Model training processing is performed on the initial first feature extraction model and the initial second feature extraction model.

[0087] In step S208, after the model training process traverses all original images I and all similar images K, a trained target first feature extraction model and a trained target second feature extraction model are obtained.

[0088] In the embodiments of the present application, Figure 3 The model training schematic diagram of the model training method for asymmetric image retrieval provided in the first embodiment of the present application is shown. Assume that the original image I is any image that needs to be registered in the retrieval library, and image K is k similar images (the number of images is greater than or equal to 1, generally the top 10) with similar content to the original image I. Images I and K are respectively forward-reasoned by the large model G and the small model Q to obtain the large model feature f G-I 、f G-K and the small model feature f Q-I 、f Q-KThe two sets of features are then mapped through the forward reasoning of the feature space mapping models TG and TQ to obtain the mapped features f TG-I 、f TG-K and f TQ-I 、f TQ-K (The dimension of the mapped features remains consistent with the original features). According to the setting of this method, the two sets of mapped features obtained should fall into the same category in the same feature space (as shown in the figure, the large circle represents the entire feature space, and the dotted ellipse represents the range of the same category in the same feature space).

[0089] In an embodiment of the present application, a model training method for asymmetric image retrieval is provided, comprising: obtaining model training data, wherein the model training data includes an original image I and a similar image K corresponding to the original image I; calling an initial first feature extraction model and an initial second feature extraction model; inputting the original image I and the similar image K into the initial first feature extraction model respectively to perform a first feature extraction process, and obtaining a first original feature f G-I and the first similarity feature f G-K ; Input the original image I and the similar image K into the initial second feature extraction model for second feature extraction processing to obtain the second original feature f Q-I and the second similarity feature f Q-K ; The first original feature f G-I and the first similarity feature f G-K are respectively input into the first spatial mapping model for first feature mapping processing to obtain the first original mapping feature f TG-I And the first similarity mapping feature f TG-K ; The second original feature f Q-I and the second similarity feature f Q-K are respectively input into the second spatial mapping model for second feature mapping processing to obtain the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K ; According to the joint loss function, the first original feature f G-I , the first similarity feature f G-K , the second original feature f Q-I , the second similarity feature f Q-K , the first original mapping feature f TG-I , the first similarity mapping feature f TG-K , the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-KThe initial first feature extraction model and the initial second feature extraction model are subjected to model training processing; after the model training processing traverses all original images I and all similar images K, the trained target first feature extraction model and the target second feature extraction model are obtained. Compared with the prior art, the present application uses a feature space mapping model instead of distillation learning to achieve asymmetric retrieval. Because the features of the large model and the small model are remapped to a new feature space, the accuracy limit of distillation learning can be broken through, and higher asymmetric retrieval accuracy can be obtained. At the same time, when the large model needs to be updated after the amount of data has accumulated for a period of time, distillation learning needs to retrain all models. Since the training of the small model is relatively independent of the large model in the present method, only the parameters of the large model and its mapping model can be trained and updated, saving the resources required for updating the model. In addition, the original image I and its similar image K are used as training pairs during training. The use of LsIm loss supervised learning in feature space mapping can enable the model to better mine the similarities within the sample class, and adding LKL loss in multi-classification can protect the distance relationship between similar samples when mapping samples, which is equivalent to mapping not only the features of a sample, but the mapping of the entire sample and its neighborhood relationship. The model obtained in this way can greatly improve the robustness of retrieval.

[0090] In some optional implementations of the embodiments of the present application, the joint loss function is expressed as L total for:

[0091] L total =L sim +L KL +L CLS

[0092] Among them, L sim represents the mapping similarity loss function, L KL represents the intra-class similarity loss function, L CLS represents the classification loss function.

[0093] In some optional implementations of the embodiments of the present application, the mapping similarity loss function L sim It is expressed as:

[0094] L sim =|f TG-I -f TQ-I | 2

[0095] Among them, f TG-I represents the first original mapping feature, f TQ-I Represents the second original mapping feature.

[0096] In some optional implementations of the embodiments of the present application, the intra-class similarity loss function L Kl It is expressed as:

[0097] L KL =KL(Softmax(f TG-I ,y,W,b),Softmax(f TQ-K ,y,W,b))

[0098] +KL(Softmax(f TQ-I ,y,W,b),Softmax(f TG-K ,y,W,b))

[0099] Among them, KL represents the calculation of KL divergence, Softmax represents the classification function of multiple classifications during training, and f TG-I represents the first original mapping feature, f TQ-K represents the second similarity mapping feature, f TQ-I represents the second original mapping feature, f TG-K represents the first similarity mapping feature, y represents the category to which the same group of original images I and similar images K belong, and W,b represents the weight parameter of the network classification head.

[0100] In some optional implementations of the embodiments of the present application, the classification loss function L CLS It is expressed as:

[0101] L CLS =-(log(Softmax(f G-I ,y,W,b))+log(Softmax(f Q-I ,y,W,b)))

[0102] Among them, Softmax represents the classification function of multiple classifications during training, f G-I represents the first original feature, f Q-I Represents the second original feature.

[0103] In some optional implementations of the embodiments of the present application, after step S208, the following steps are further included:

[0104] Get the search gallery and the gallery to be searched;

[0105] Performing a third feature extraction process on all images in the retrieval gallery according to the target first feature extraction model to obtain retrieval gallery features;

[0106] Performing a third feature mapping process on the retrieval gallery feature according to the first spatial mapping model to obtain a retrieval gallery mapping feature;

[0107] Performing fourth feature extraction processing on all images in the to-be-searched gallery according to the target second feature extraction model to obtain features of the to-be-searched gallery;

[0108] Performing fourth feature mapping processing on the features of the gallery to be searched according to the second spatial mapping model to obtain mapping features of the gallery to be searched;

[0109] Constructing the retrieval gallery features and the index table of the retrieval gallery by means of inverted index;

[0110] Calculate the feature distance between the feature of the image library to be retrieved and the feature mapping of the image library to be retrieved according to the L2 distance algorithm;

[0111] According to the feature distance, determine the target retrieval gallery mapping features with the closest feature distances to a preset number of features corresponding to each to-be-retrieved gallery mapping feature;

[0112] In the to-be-searched gallery, a search image corresponding to the mapping feature of the target search gallery is obtained according to the index table to obtain the target search result.

[0113] In the embodiment of the present application, after the training phase is completed, when using, we first have a search library Θ and a to-be-searched library Φ, the purpose is to find all images with similar content in the search library Θ for each image in the to-be-searched library Φ. The specific use process is as follows:

[0114] ① First, use the large model G to extract the features of all images in the gallery Θ, and then generate the mapped features fTG-Θ after mapping by the mapping model TG and record them in the retrieval library;

[0115] ② Then use the small model Q to extract the features of each image in the to-be-retrieved library Φ, and generate the small model shape mapping feature fTQ-Φ after mapping by the mapping model TQ;

[0116] ③ Use the inverted index method to build an index table, and use the L2 distance as the metric to calculate the distance between each feature pair in fTQ-Φ and fTG-Θ, and finally calculate the preset number of features that are closest to each feature in fTQ-Φ;

[0117] ④ Obtain search results based on index table query.

[0118] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0119] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0120] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0121] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0122] Embodiment 2

[0123] Further references Figure 4 , as a response to the above Figure 2 The present application provides an embodiment of a model training device for asymmetric image retrieval, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0124] like Figure 4 As shown, the model training device 200 applied to asymmetric image retrieval in the embodiment of the present application includes:

[0125] A training data acquisition module 210 is used to acquire model training data, wherein the model training data includes an original image I and a similar image K corresponding to the original image I;

[0126] A model calling module 220, used to call an initial first feature extraction model and an initial second feature extraction model;

[0127] The first feature extraction module 230 is used to input the original image I and the similar image K into the initial first feature extraction model to perform first feature extraction processing to obtain the first original feature f G-I and the first similarity feature f G-K ;

[0128] The second feature extraction module 240 is used to input the original image I and the similar image K into the initial second feature extraction model to perform second feature extraction processing to obtain the second original feature f Q-I and the second similarity feature f Q-K ;

[0129] The first feature mapping module 250 is used to transform the first original feature f G-I and the first similarity feature f G-K are respectively input into the first spatial mapping model for first feature mapping processing to obtain the first original mapping feature f TG-I And the first similarity mapping feature f TG-K ;

[0130] The second feature mapping module 260 is used to transform the second original feature f Q-I and the second similarity feature f Q-K are respectively input into the second spatial mapping model for second feature mapping processing to obtain the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K ;

[0131] The model training module 270 is used to train the model according to the joint loss function and the first original feature f G-I , the first similarity feature f G-K , the second original feature f Q-I , the second similarity feature f Q-K , the first original mapping feature f TG-I , the first similarity mapping feature f TG-K , the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K Performing model training processing on the initial first feature extraction model and the initial second feature extraction model;

[0132] The training end module 280 is used to obtain a trained target first feature extraction model and a target second feature extraction model after the model training process traverses all original images I and all similar images K.

[0133] In an embodiment of the present application, a model training device 200 for asymmetric image retrieval is provided, comprising: a training data acquisition module 210, used to acquire model training data, wherein the model training data includes an original image I and a similar image K corresponding to the original image I; a model calling module 220, used to call an initial first feature extraction model and an initial second feature extraction model; a first feature extraction module 230, used to input the original image I and the similar image K into the initial first feature extraction model for first feature extraction processing, and obtain a first original feature f G-I and the first similarity feature f G-K The second feature extraction module 240 is used to input the original image I and the similar image K into the initial second feature extraction model for second feature extraction processing to obtain the second original feature f Q-I and the second similarity feature f Q-K ; The first feature mapping module 250 is used to transform the first original feature f G-I and the first similarity feature f G-K are respectively input into the first spatial mapping model for first feature mapping processing to obtain the first original mapping feature f TG-I And the first similarity mapping feature f TG-K ; The second feature mapping module 260 is used to convert the second original feature f Q-I and the second similarity feature f Q-K are respectively input into the second spatial mapping model for second feature mapping processing to obtain the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K ; Model training module 270, for according to the joint loss function, the first original feature f G-I , the first similarity feature f G-K , the second original feature f Q-I , the second similarity feature f Q-K , the first original mapping feature f TG-I , the first similarity mapping feature f TG-K , the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-KThe initial first feature extraction model and the initial second feature extraction model are subjected to model training processing; the training end module 280 is used to obtain the trained target first feature extraction model and the target second feature extraction model after the model training processing traverses all original images I and all similar images K. Compared with the prior art, the present application uses a feature space mapping model instead of distillation learning to achieve asymmetric retrieval. Because the features of the large model and the small model are remapped to a new feature space, the accuracy limit of distillation learning can be broken through, and higher asymmetric retrieval accuracy can be obtained. At the same time, when the large model needs to be updated after the amount of data has accumulated for a period of time, distillation learning needs to retrain all models. Since the training of the small model is relatively independent of the large model in the present method, only the parameters of the large model and its mapping model can be trained and updated, saving the resources required for updating the model. In addition, the original image I and its similar image K are used as training pairs during training. The use of LsIm loss supervised learning in feature space mapping can enable the model to better mine the similarities within the sample class, and adding LKL loss in multi-classification can protect the distance relationship between similar samples when mapping samples, which is equivalent to mapping not only the features of a sample, but the mapping of the entire sample and its neighborhood relationship. The model obtained in this way can greatly improve the robustness of retrieval.

[0134] In some optional implementations of the embodiments of the present application, the device further includes:

[0135] The gallery acquisition module is used to acquire the search gallery and the gallery to be searched;

[0136] A third feature extraction module is used to perform a third feature extraction process on all images in the search gallery according to the target first feature extraction model to obtain the search gallery features;

[0137] A third feature mapping module, used for performing a third feature mapping process on the retrieval gallery feature according to the first spatial mapping model to obtain a retrieval gallery mapping feature;

[0138] A fourth feature extraction module, used for performing fourth feature extraction processing on all images in the to-be-searched gallery according to the target second feature extraction model to obtain features of the to-be-searched gallery;

[0139] A fourth feature mapping module, used for performing fourth feature mapping processing on the features of the gallery to be searched according to the second spatial mapping model to obtain mapping features of the gallery to be searched;

[0140] An index table construction module is used to construct the retrieval library features and the index table of the retrieval library according to the inverted index method;

[0141] A feature distance calculation module is used to calculate the feature distance of the feature of the to-be-searched gallery and the feature of the to-be-searched gallery mapping according to the L2 distance algorithm;

[0142] A target retrieval gallery mapping feature confirmation module is used to confirm, based on feature distances, a preset number of target retrieval gallery mapping features that are closest in feature distance to each to-be-retrieved gallery mapping feature;

[0143] The retrieval result acquisition module is used to obtain the retrieval image corresponding to the mapping feature of the target retrieval library in the to-be-retrieved library according to the index table, and obtain the target retrieval result.

[0144] To solve the above technical problems, the present application also provides a computer device. Figure 5 , Figure 5 This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0145] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0146] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0147] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions for the model training method for asymmetric image retrieval, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.

[0148] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions or process data stored in the memory 310, such as running the computer-readable instructions of the model training method applied to asymmetric image retrieval.

[0149] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0150] The computer device provided in the present application uses a feature space mapping model instead of distillation learning to achieve asymmetric retrieval. Because the features of the large model and the small model are remapped to a new feature space, the accuracy limit of distillation learning can be broken through, and higher asymmetric retrieval accuracy can be obtained. At the same time, when the large model needs to be updated after the amount of data has accumulated for a period of time, distillation learning needs to retrain all models. However, since the training of the small model in the present method is relatively independent of the large model, only the parameter training and updating of the large model and its mapping model can be performed, saving the resources required for updating the model. In addition, the original image I and its similar image K are used as training pairs during training, and the LsIm loss supervised learning is used in the feature space mapping, which can enable the model to better mine the similarity within the sample class, and adding LKL loss in multi-classification can protect the distance relationship between similar samples when mapping samples, which is equivalent to mapping not only the features of a sample, but the mapping of the entire sample and its neighborhood relationship. The model obtained in this way can greatly improve the retrieval robustness.

[0151] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the model training method applied to asymmetric image retrieval as described above.

[0152] The computer-readable storage medium provided in the present application uses a feature space mapping model instead of distillation learning to achieve asymmetric retrieval. Because the features of the large model and the small model are remapped to a new feature space, the accuracy upper limit of distillation learning can be broken through, and higher asymmetric retrieval accuracy can be obtained. At the same time, when the large model needs to be updated after the amount of data has accumulated for a period of time, distillation learning needs to retrain all models. However, since the training of the small model in the present method is relatively independent of the large model, only the parameter training and updating of the large model and its mapping model can be performed, saving the resources required for updating the model. In addition, the original image I and its similar image K are used as training pairs during training, and the LsIm loss supervised learning is used in the feature space mapping, which can enable the model to better mine the similarity within the sample class, and adding LKL loss in multi-classification can protect the distance relationship between similar samples when mapping samples, which is equivalent to mapping not only the features of a sample, but the mapping of the entire sample and its neighborhood relationship. The model obtained in this way can greatly improve the robustness of retrieval.

[0153] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0154] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A model training method for asymmetric image retrieval, characterized in that: The steps include: Acquire model training data, wherein the model training data includes an original image I and a similar image K corresponding to the original image I; Calling an initial first feature extraction model and an initial second feature extraction model; The original image I and the similar image K are respectively input into the initial first feature extraction model to perform first feature extraction processing to obtain the first original feature f G-I and the first similarity feature f G-K ; The original image I and the similar image K are respectively input into the initial second feature extraction model for second feature extraction processing to obtain the second original feature f Q-I and the second similarity feature f Q-K ; The first original feature f G-I And the first similar feature f G-K are respectively input into the first spatial mapping model for first feature mapping processing to obtain the first original mapping feature f TG-I And the first similarity mapping feature f TG-K ; The second original feature f Q-I And the second similar feature f Q-K are respectively input into the second spatial mapping model for second feature mapping processing to obtain the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K ; According to the joint loss function, the first original feature f G-I , the first similar feature f G-K The second original feature f Q-I The second similar feature f Q-K , the first original mapping feature f TG-I , the first similarity mapping feature f TG-K , the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K Performing model training processing on the initial first feature extraction model and the initial second feature extraction model; After the model training process traverses all the original images I and all the similar images K, a trained target first feature extraction model and a trained target second feature extraction model are obtained.

2. The model training method for asymmetric image retrieval according to claim 1, characterized in that: The joint loss function is expressed as L total for: L total =L sim +L KL +L CLS Among them, L sim represents the mapping similarity loss function, L KL represents the intra-class similarity loss function, L CLS represents the classification loss function.

3. The model training method for asymmetric image retrieval according to claim 2, characterized in that: The mapping similarity loss function L sim It is expressed as: L sim =|f TG-I -f TQ-I | 2 Among them, f TG-I represents the first original mapping feature, f TQ-I represents the second original mapping feature.

4. The model training method for asymmetric image retrieval according to claim 2, characterized in that: The intra-class similarity loss function L KL It is expressed as: L KL =KL(Softmax(f TG-I ,y,W,b),Softmax(f TQ-K ,y,W,b))+KL(Softmax(f TQ-I ,y,W,b),Softmax(f TG-K ,y,w,b)) Among them, KL represents the calculation of KL divergence, Softmax represents the classification function of multiple classifications during training, and f TG-I represents the first original mapping feature, f TQ-K represents the second similarity mapping feature, f TQ-I represents the second original mapping feature, f TG-K represents the first similarity mapping feature, y represents the category to which the same group of original images I and similar images K belong, and W,b represents the weight parameter of the network classification head.

5. The model training method for asymmetric image retrieval according to claim 2, characterized in that: The classification loss function L CLS It is expressed as: L CLG =-(log(Softmax(f G-I ,y,W,b))+log(Softmax(f Q-I ,y,W,b))) Among them, Softmax represents the classification function of multiple classifications during training, f G-I represents the first original feature, f Q-I represents the second original feature.

6. The model training method for asymmetric image retrieval according to claim 1, characterized in that: After the model training process traverses all the original images I and all the similar images K to obtain the trained target first feature extraction model and the target second feature extraction model, the following steps are also included: Get the search gallery and the gallery to be searched; Performing a third feature extraction process on all images in the search gallery according to the target first feature extraction model to obtain search gallery features; Performing a third feature mapping process on the retrieval gallery feature according to the first spatial mapping model to obtain a retrieval gallery mapping feature; Performing fourth feature extraction processing on all images in the to-be-searched gallery according to the target second feature extraction model to obtain features of the to-be-searched gallery; Performing fourth feature mapping processing on the features of the to-be-searched gallery according to the second spatial mapping model to obtain mapping features of the to-be-searched gallery; Constructing the retrieval gallery features and the index table of the retrieval gallery in an inverted index manner; Calculate the feature distance between the feature of the to-be-searched gallery and the feature of the to-be-searched gallery mapping according to the L2 distance algorithm; Determine, according to the feature distance, a preset number of target retrieval gallery mapping features with the closest feature distances corresponding to each of the to-be-retrieved gallery mapping features; A search image corresponding to the mapping feature of the target search library is obtained in the to-be-searched library according to the index table to obtain a target search result.

7. A model training device for asymmetric image retrieval, characterized in that: include: A training data acquisition module, used to acquire model training data, wherein the model training data includes an original image I and a similar image K corresponding to the original image I; A model calling module, used for calling an initial first feature extraction model and an initial second feature extraction model; The first feature extraction module is used to input the original image I and the similar image K into the initial first feature extraction model to perform first feature extraction processing to obtain the first original feature f G-I and the first similarity feature f G-k ; The second feature extraction module is used to input the original image I and the similar image K into the initial second feature extraction model to perform second feature extraction processing to obtain the second original feature f Q-I and the second similarity feature f Q-k ; The first feature mapping module is used to transform the first original feature f G-I And the first similar feature f G-K are respectively input into the first spatial mapping model for first feature mapping processing to obtain the first original mapping feature f TG-I And the first similarity mapping feature f TG-K ; The second feature mapping module is used to transform the second original feature f Q-I And the second similar feature f Q-K are respectively input into the second spatial mapping model for second feature mapping processing to obtain the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K ; A model training module is used to train the model according to the joint loss function and the first original feature f G-I , the first similar feature f G-K The second original feature f Q-I The second similar feature f Q-K , the first original mapping feature f TG-I , the first similarity mapping feature f TG-K , the second original mapping feature f TQ-I And the second similarity mapping feature f TQ-K Performing model training processing on the initial first feature extraction model and the initial second feature extraction model; The training end module is used to obtain the trained target first feature extraction model and the target second feature extraction model after the model training process traverses all the original images I and all the similar images K.

8. The model training device for asymmetric image retrieval according to claim 7, characterized in that: The device also includes: The gallery acquisition module is used to acquire the search gallery and the gallery to be searched; A third feature extraction module, used for performing a third feature extraction process on all images in the search gallery according to the target first feature extraction model to obtain search gallery features; A third feature mapping module, configured to perform a third feature mapping process on the retrieval gallery feature according to the first spatial mapping model to obtain a retrieval gallery mapping feature; A fourth feature extraction module, used for performing fourth feature extraction processing on all images in the to-be-searched gallery according to the target second feature extraction model to obtain features of the to-be-searched gallery; A fourth feature mapping module, used for performing a fourth feature mapping process on the features of the to-be-searched gallery according to the second spatial mapping model to obtain mapping features of the to-be-searched gallery; An index table construction module, used to construct the retrieval library features and the index table of the retrieval library according to an inverted index method; A feature distance calculation module, used for calculating the feature distance between the feature of the to-be-searched gallery and the feature of the to-be-searched gallery mapping according to an L2 distance algorithm; A target retrieval gallery mapping feature confirmation module is used to confirm, according to the feature distance, a preset number of target retrieval gallery mapping features that are closest in feature distance to each of the gallery mapping features to be retrieved; The retrieval result acquisition module is used to obtain the retrieval image corresponding to the mapping feature of the target retrieval library in the to-be-retrieved library according to the index table, and obtain the target retrieval result.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the model training method for asymmetric image retrieval as described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the model training method for asymmetric image retrieval as described in any one of claims 1 to 6.

Images