Feature model acquisition method, image processing method, device and computer equipment

Abstract

The present application discloses a method for acquiring a feature model, an image processing method, an apparatus, a computer device, and a storage medium, and belongs to the field of computer technology. The present application obtains a discriminative parameter for characterizing the degree of difference of the first image feature of sample images under different categories, so that the discriminative parameter can control the trained target feature model to have a better feature extraction capability, and obtains a compatibility parameter for characterizing the degree of similarity between the first image feature and the second image feature of the same sample image, so that the compatibility parameter can control the target feature model to have a better compatibility capability with the original first feature model. Therefore, under the joint constraints of the discriminative parameter and the compatibility parameter, the target feature model obtained by training the initial feature model not only has a better feature extraction capability, but also has a strong compatibility capability.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for acquiring a feature model, an image processing method, an apparatus, a computer device, and a storage medium. Background Art

[0002] With the development of computer technology, image retrieval systems have been used more and more widely. Usually, image retrieval systems use feature models to extract features of each original image, forming a feature library composed of each extracted feature. Then, when the user provides a query image, the feature model is used again to extract the features of the query image, and the features are compared with the features of each original image in the feature library. The sample that is most similar to the query image (which may be the original image, the index or link corresponding to the original image, etc.) is returned to the user.

[0003] In order to improve the performance of the image retrieval system, the feature model is usually upgraded during the optimization process, and the upgraded feature model is used to extract new features for each original image. The extracted new features are then used to replace the old features already in the feature library. However, this method is not applicable in the context of privacy protection. Due to the requirement of privacy protection, the server side can only store the features extracted from the original image, but cannot retain the original image. This means that even if the feature model is upgraded, the original image cannot be obtained to extract the corresponding new features. Therefore, there is an urgent need for a privacy-protected image retrieval system to obtain a new feature model with better performance. Summary of the invention

[0004] The embodiments of the present application provide a method for acquiring a feature model, an image processing method, an apparatus, a computer device, and a storage medium, which can improve the retrieval performance of an image retrieval system under the requirements of privacy protection.

[0005] The technical solution is as follows:

[0006] In one aspect, a method for acquiring a feature model is provided, the method comprising:

[0007] Based on the initial feature model, first image features are extracted from the plurality of sample images to obtain discriminative parameters, wherein the discriminative parameters represent the degree of difference of the first image features of the sample images under different categories;

[0008] Acquire a compatibility parameter based on the first feature model for extracting a second image feature and the first image feature from the plurality of sample images, wherein the compatibility parameter represents a degree of similarity between the first image feature and the second image feature of the same sample image;

[0009] Based on the discriminative parameter and the compatibility parameter, the parameters of the initial feature model are iteratively adjusted to obtain a target feature model.

[0010] In one aspect, an image processing method is provided, the method comprising:

[0011] In response to an image query request, obtaining a query image carried in the image query request;

[0012] Extracting query image features of the query image based on a target feature model, wherein the target feature model is obtained by iteratively adjusting parameters of an initial feature model based on a discriminative parameter and a compatibility parameter;

[0013] Determine, from a plurality of inventory upgrade features in a feature library, a target inventory feature that has the highest matching degree with the query image feature, wherein the inventory upgrade feature is obtained by processing the inventory image feature extracted by the first feature model by a feature upgrade model;

[0014] Returns image retrieval information associated with the target inventory feature.

[0015] In one aspect, a device for acquiring a feature model is provided, the device comprising:

[0016] A first acquisition module, configured to obtain a discriminant parameter based on the first image features extracted from the plurality of sample images based on the initial feature model, wherein the discriminant parameter represents the degree of difference between the first image features of the sample images of different categories;

[0017] A second acquisition module, configured to acquire a compatibility parameter based on the first feature model for extracting a second image feature from the plurality of sample images and the first image feature, wherein the compatibility parameter represents a degree of similarity between the first image feature and the second image feature of the same sample image;

[0018] The parameter adjustment module is used to iteratively adjust the parameters of the initial feature model based on the discriminative parameter and the compatibility parameter to obtain a target feature model.

[0019] In a possible implementation manner, the second acquisition module is used to:

[0020] For any sample image, obtaining a first similarity between a first image feature and a second image feature of the sample image;

[0021] The compatibility parameter is obtained based on the first similarities corresponding to each of the plurality of sample images.

[0022] In a possible implementation manner, the first acquisition module includes:

[0023] a determination unit, configured to determine, for any sample image, a predicted category obtained by the initial feature model by classifying the sample image based on a first image feature of the sample image;

[0024] A first acquisition unit, configured to acquire a discriminative loss of the sample image based on the predicted category and a reference category to which the sample image belongs, wherein the discriminative loss is used to characterize a degree of difference between the predicted category and the reference category;

[0025] The second acquisition unit is used to acquire the discriminative parameter based on the discriminative losses corresponding to each of the plurality of sample images.

[0026] In a possible implementation manner, the first acquiring unit is used to:

[0027] Acquire a feature angle between a first image feature of the sample image and a category feature of the reference category;

[0028] Adding the feature angle to the classification additive angle margin to obtain a feature correction angle, wherein the classification additive angle margin is used to increase the degree of inter-class difference between class features of different classes;

[0029] The discriminative loss of the sample image is obtained based on the cosine value of the feature correction angle and the prediction values ​​of classifying the first image feature of the sample image into respective ones of a plurality of candidate categories.

[0030] In a possible implementation manner, the parameter adjustment module is used to:

[0031] Determining the loss function value of the initial feature model for this iteration based on the discriminative parameter and the compatibility parameter;

[0032] When the number of iterations and the loss function value do not meet the first stop condition, iteratively adjust the parameters of the initial feature model, and perform the step of obtaining the loss function value again based on the initial feature model after adjusting the parameters;

[0033] When either the number of iterations or the loss function value meets the first stop condition, the target feature model is output.

[0034] In a possible implementation manner, the device further includes:

[0035] A first extraction module, used for inputting the plurality of sample images into the target feature model to obtain target image features of the plurality of sample images;

[0036] A second extraction module, configured to input the second image features of the plurality of sample images into an initial upgrade model to obtain third image features of the plurality of sample images;

[0037] A third acquisition module, configured to acquire a similarity parameter based on the third image feature and the target image feature, wherein the similarity parameter represents a degree of similarity between the third image feature and the target image feature of the same sample image;

[0038] The parameter adjustment module is further used to iteratively adjust the parameters of the initial upgrade model based on the similarity parameter to obtain a feature upgrade model, and the feature upgrade model is used to optimize the image features extracted by the first feature model.

[0039] In a possible implementation manner, the third acquisition module is used to:

[0040] For any sample image, obtaining a second similarity between a third image feature of the sample image and a target image feature;

[0041] The similarity parameter is obtained based on the second similarities corresponding to each of the plurality of sample images.

[0042] In a possible implementation manner, the device further includes:

[0043] a fourth acquisition module, configured to acquire a plurality of inventory image features in a feature library corresponding to the first feature model;

[0044] A feature upgrade module, used for inputting the plurality of inventory image features into the feature upgrade model to obtain a plurality of inventory upgrade features;

[0045] The feature replacement module is used to replace the plurality of inventory image features with the corresponding inventory upgrade features in the feature library.

[0046] In one aspect, an image processing device is provided, the device comprising:

[0047] An acquisition module, configured to, in response to an image query request, acquire a query image carried in the image query request;

[0048] An extraction module, configured to extract query image features of the query image based on a target feature model, wherein the target feature model is obtained by iteratively adjusting parameters of an initial feature model based on a discriminative parameter and a compatibility parameter;

[0049] a determination module, configured to determine, from a plurality of inventory upgrade features in a feature library, a target inventory feature that has the highest matching degree with the query image feature, wherein the inventory upgrade feature is obtained by processing the inventory image feature extracted by the first feature model by a feature upgrade model;

[0050] The returning module is used to return the image retrieval information associated with the target inventory feature.

[0051] On the one hand, a computer device is provided, which includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement a feature model acquisition method or an image processing method as described in any possible implementation method described above.

[0052] On the one hand, a storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement a method for acquiring a feature model or an image processing method as described in any possible implementation manner.

[0053] In one aspect, a computer program product or a computer program is provided, wherein the computer program product or the computer program comprises one or more program codes, wherein the one or more program codes are stored in a computer-readable storage medium. One or more processors of a computer device can read the one or more program codes from the computer-readable storage medium, and the one or more processors execute the one or more program codes, so that the computer device can perform the method for acquiring a feature model or the image processing method of any possible implementation manner described above.

[0054] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0055] By obtaining the discriminative parameters for characterizing the degree of difference between the first image features of sample images under different categories, the discriminative parameters can be used to control the trained target feature model to have better feature extraction capability. By obtaining the compatibility parameters for characterizing the degree of similarity between the first image features and the second image features of the same sample image, the compatibility parameters can be used to control the target feature model to have better compatibility with the original first feature model. Therefore, under the joint constraints of the discriminative parameters and the compatibility parameters, the target feature model trained by the initial feature model not only has better feature extraction capability, but also has strong compatibility capability.

[0056] Furthermore, after the target feature model is launched on the privacy-preserving image retrieval system, when a new query image is received, the target feature model optimizes the expression ability and improves the feature quality of the extracted query image features, and ensures that the query image features extracted by the target feature model can be directly calculated for similarity on the old feature library extracted by the original first feature model, thereby optimizing the retrieval performance of the entire image retrieval system by improving the feature quality of the query image without affecting the normal image retrieval function. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 It is a schematic diagram of an implementation environment of a method for acquiring a feature model provided in an embodiment of the present application;

[0059] Figure 2 is a flow chart of a method for acquiring a feature model provided in an embodiment of the present application;

[0060] Figure 3 is a flow chart of a method for acquiring a feature model provided in an embodiment of the present application;

[0061] Figure 4 It is a flow chart of a method for training a feature upgrade model provided in an embodiment of the present application;

[0062] Figure 5 It is a principle flow chart of a bidirectional compatible model upgrade solution provided in an embodiment of the present application;

[0063] Figure 6 is a flow chart of an image processing method provided by an embodiment of the present application;

[0064] Figure 7 It is a structural schematic diagram of a device for acquiring a feature model provided in an embodiment of the present application;

[0065] Figure 8 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0066] Fig. 9 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0068] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with basically the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on quantity and execution order.

[0069] In the present application, the term "at least one" means one or more, and the meaning of "plurality" means two or more. For example, a plurality of first positions means two or more first positions.

[0070] In this application, the term "including at least one of A or B" refers to the following situations: including only A, including only B, and including both A and B.

[0071] The user-related information (including but not limited to the user's device information, personal information, behavior information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application, when applied to specific products or technologies in the manner of the embodiments of this application, are all permitted, agreed, authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant information, data and signals must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the sample images used for model training involved in this application are all obtained with full authorization.

[0072] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0073] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0074] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D (3-Dimension) technology, virtual reality, augmented reality, simultaneous positioning and map construction, autonomous driving, smart transportation and other technologies.

[0075] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0076] The solution provided in the embodiments of the present application involves technologies such as artificial intelligence computer vision, and in particular, involves an image retrieval system that extracts features based on a feature model under computer vision technology to construct a feature library, thereby realizing an image retrieval function, which is explained in detail through the following embodiments.

[0077] The following is an explanation of the terms involved in the embodiments of the present application.

[0078] Image retrieval system: refers to a system where a user gives a query image and the system returns the sample most similar to the query image (which may be a similar image, the index, category or link corresponding to the similar image, etc.) to the user. In an image retrieval system, a feature model is usually used in advance to extract the features of each original image in the database to form a feature library consisting of the extracted features. When a user provides a query image, the feature model is used again to extract the features of the query image, and the features of the query image are compared with the features of each original image in the feature library to find the features of the stock image that are most similar to the features of the query image (i.e., find the features of the most similar original image in the feature library). At this time, the original image itself in the corresponding database (if saved), the index, category or link corresponding to the original image, etc. are returned to the user as the most similar sample according to the application requirements.

[0079] Feature model upgrade: A very effective way to improve the retrieval performance of the image retrieval system. Feature model upgrade means: training a new feature model with better feature extraction capabilities, using the new feature model to re-extract new features from each original image in the database, replacing the old feature library with a new feature library composed of these extracted new features, and then launching the new feature model and the new feature library to the image retrieval system. This allows the new feature model to be used to extract new features from subsequent query images, and to search the new feature library for the original image in the database that corresponds to the inventory image features that are most similar to the new features of the query image, thereby optimizing the performance of the image retrieval system.

[0080] Privacy-preserving image retrieval system: In the context of requiring privacy protection, the server side can only store the features of the original image in the feature library, but cannot retain the original image itself in the database. This means that the above-mentioned traditional feature model upgrade method cannot be applied to improve the performance of the privacy-preserving image retrieval system, because the server side cannot store the original image in the database, and thus cannot use the new feature model to extract new features of each original image in the database to form a new feature library.

[0081] Optionally, a privacy protection method is provided, where after the server extracts the features of the original image using the feature model, the server only saves the features of each original image in the feature library of the server, and does not save the original image itself in the database (that is, the original image will be deleted after the features are extracted using the feature model). Alternatively, another privacy protection method is provided, where the server sends the feature model to the terminal, and after the terminal extracts the features of the original image using the feature model, it only uploads the features of the original image to the server, that is, the server strictly does not touch the original image.

[0082] It can be seen that due to the requirement of privacy protection, the server will not save the original image in the database, so the traditional feature model upgrading method cannot be applied to the privacy-protected image retrieval system. In view of this, the embodiment of the present application provides a method for obtaining a feature model, which can be trained to obtain a backward-compatible new feature model, that is, the new features extracted by the trained new feature model (that is, the target feature model) can maintain compatibility with the old features extracted by the old feature model (that is, the first feature model) while improving its own quality, which means that the new features can directly calculate the similarity with the old features to ensure the query performance of the image retrieval system.

[0083] Furthermore, the trained backward-compatible new feature model can be directly put online in the image retrieval system. The features of the query image extracted using the new feature model can be directly retrieved on the old feature library without re-extracting the features of the original image and updating the feature library, which can meet the needs of the privacy-preserving image retrieval system. After the backward-compatible new feature model is put online in the image retrieval system, the improvement in the retrieval performance of the image retrieval system comes from the higher quality of the features of the query image extracted using the new feature model, that is, the features of the query image can improve their own quality while maintaining compatibility with the old feature library, thereby improving the system's retrieval accuracy.

[0084] Furthermore, the embodiment of the present application also proposes a bidirectionally compatible feature model upgrade solution, which not only trains the above-mentioned backward-compatible new feature model to ensure that the features output by the new feature model have backward-compatible characteristics and can be directly compared with the old feature library, so that the image retrieval system can be put online in real time, and the retrieval performance of the entire image retrieval system can be improved by improving the feature quality of the query image; in addition, a forward-compatible feature upgrade model is also trained. This feature upgrade model is used to upgrade the features of the old feature library. Since the input of the feature upgrade model is the old features of the original image rather than the original image, it can meet the privacy protection requirements. The feature upgrade model can try to update the old features forward to the new feature space corresponding to the above-mentioned new feature model, so as to further improve the retrieval performance of the entire image retrieval system by improving the feature quality of the old feature library.

[0085] The following describes the system architecture of the embodiment of the present application.

[0086] Figure 1 Schematic diagram of an implementation environment of a method for acquiring a feature model provided in an embodiment of the present application. Figure 1 , the implementation environment includes a terminal 101 and a server 102.

[0087] Terminal 101 is used to provide original images and / or query images. An application that supports image retrieval function is installed and run on terminal 101. Optionally, the application includes: at least one of: a social application, a content sharing application, a payment application, a short video application, a photo application, an audio and video application, a live broadcast application or a browser application.

[0088] The "original image" involved in the embodiment of the present application refers to the image provided to the feature model for extracting stock image features and constituting a feature library. The "query image" involved in the embodiment of the present application refers to the image used to query the most similar sample when providing image retrieval performance to the outside after the feature library is formed. In the privacy protection scenario, after the server uses the feature model to extract the stock image features of the original image, the original image will not be saved in the database, that is, the original image will be completely deleted from the storage medium such as memory, database or disk, and only the stock image features corresponding to the original image will be saved in the feature library.

[0089] In a privacy protection scenario, after the user uses the terminal 101 to take one (or more) original images, the original images are uploaded to the server 102 so that the server 102 can extract the features of the original images to form a feature library. After the server 102 uses the original images to extract the features, it will delete the original images, that is, the server 102 will not retain the original images in any database. After that, it is assumed that the user uses the terminal 101 again to take a query image and upload the query image to the server 102 to request the server 102 to return the sample most similar to the query image (which may be a similar image, an index, category or link corresponding to the similar image, etc.), or the server 102 directly returns the judgment result generated based on the most similar sample (such as whether the object in the query image matches the specified object in the similar image).

[0090] In another privacy protection scenario, the server 102 sends the feature model to the terminal 101. After the user uses the terminal 101 to take a picture (or multiple pictures) of the original image, the feature model is used on the terminal 101 to extract the features of the original image, and the extracted features are uploaded to the server 102 to form a feature library. The server 102 will not strictly touch the original image and cannot save the original image. After that, assuming that the user uses the terminal 101 again to take a query image, the query image features are extracted using the feature model and uploaded to the server 102 to request the server 102 to return the sample most similar to the query image, or the server 102 returns the judgment result generated based on the most similar sample, etc. Among them, the server 102 can also be a node of the blockchain.

[0091] The terminal 101 and the server 102 can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0092] The server 102 is used to provide background services for the above application, and the server 102 includes: at least one of a single server, multiple servers, a cloud computing platform, or a virtualization center. Optionally, the server 102 undertakes the main image processing work, and the terminal 101 undertakes the secondary image processing work; or, the server 102 undertakes the secondary image processing work, and the terminal 101 undertakes the main image processing work; or, the terminal 101 and the server 102 use a distributed computing architecture to collaboratively perform image processing work.

[0093] Optionally, server 102 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0094] Schematically, an image retrieval system is installed and run on the server 102. The image retrieval system is used to provide image retrieval functions externally. The image retrieval system includes a feature library, a feature model and a retrieval module. The feature library is used to store features extracted from each original image (also called stock image features), the feature model is used to extract features of the original image and / or the query image, and the retrieval module is used to find the most similar sample based on any query image.

[0095] Optionally, the terminal 101 generally refers to one of a plurality of terminals, and the device type of the terminal 101 includes, but is not limited to: a vehicle-mounted terminal, a television, a smart phone, a smart speaker, a smart watch, a tablet computer, an intelligent voice interaction device, a smart home appliance, an aircraft, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3) player, an MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) player, a laptop computer or a desktop computer. In the following embodiments, the terminal includes a smart phone for example.

[0096] Those skilled in the art will appreciate that the number of the terminals 101 may be more or less. For example, there may be only one terminal 101, or there may be dozens or hundreds of terminals 101, or a greater number. The embodiment of the present disclosure does not limit the number and device type of the terminals 101.

[0097] Figure 2is a flow chart of a method for obtaining a feature model provided in an embodiment of the present application. Figure 2 This embodiment is executed by a computer device and is described by taking the computer device as a server as an example. This embodiment includes the following steps:

[0098] 201. The server extracts first image features from a plurality of sample images based on an initial feature model, and obtains a discriminative parameter, where the discriminative parameter represents a degree of difference between the first image features of sample images of different categories.

[0099] The initial feature model involved in the embodiment of the present application refers to an untrained target feature model. In other words, the initial feature model and the target feature model have the same model architecture. After iteratively adjusting the model parameters (i.e., training), the initial feature model stops training when the first stopping condition is met to obtain the target feature model.

[0100] The sample images involved in the embodiments of the present application refer to images in a sample image set used to train an initial feature model to obtain a target feature model. This sample image set is a set of images that is similar in type to a data set consisting of original images but has been fully authorized and separately agreed to be used.

[0101] The discriminative parameter involved in the embodiment of the present application refers to a measure of the ability of the first image feature extracted by the initial feature model to distinguish sample images belonging to different categories. For example, when the discriminative parameter value is higher, the difference between the first image features extracted by the initial feature model representing sample images of different categories is greater, such as the larger the feature angle in the feature space, or the longer the Euclidean distance in the feature space, etc., which means that the ability to distinguish sample images belonging to different categories by the first image feature is stronger; conversely, when the discriminative parameter value is lower, the difference between the first image features extracted by the initial feature model representing sample images of different categories is smaller, such as the smaller the feature angle in the feature space, or the closer the Euclidean distance in the feature space, etc., which means that the ability to distinguish sample images belonging to different categories by the first image feature is weaker.

[0102] In some embodiments, the server obtains a sample image set, determines multiple sample images used for this training from the sample image set, then inputs the multiple sample images into an initial feature model, extracts features from the multiple sample images through the initial feature model, and outputs the first image features of each of the multiple sample images.

[0103] In some embodiments, the above-mentioned multiple sample images are all the images in the sample image set, or the images of the first target ratio in the sample image set are divided into a training sample set, and the remaining images of the second target ratio are divided into a test sample set. The multiple sample images in the training sample set are used as the multiple sample images used in the training stage, and the multiple sample images in the test sample set are used to test the feature extraction performance of the trained target feature model.

[0104] Optionally, the first target ratio and the second target ratio are both numerical values ​​greater than or equal to 0 and less than or equal to 100%, and the sum of the first target ratio and the second target ratio is equal to 100%. For example, the first target ratio is 80% and the second target ratio is 20%, or the first target ratio is 70% and the second target ratio is 30%, etc. The embodiments of the present application do not specifically limit this.

[0105] In some embodiments, the initial feature model is any machine learning model that can extract image features. The model architecture of the initial feature model includes: ResNet (residual network) -18, ResNet-34, ResNet-50, ResNet-101, ResNet-152, CNN (Convolutional Neural Network), DNN (Deep Neural Network), MLP (Multi-Layer Perceptron), etc. The embodiment of the present application does not specifically limit the model architecture of the initial feature model.

[0106] Schematically, taking ResNet-50 as the initial feature model as an example, the ResNet-50 model includes multiple serially connected residual blocks. The serial connection means that the output of the previous residual block is used as the input of the current residual block. Each residual block includes multiple convolutional layers. Adjacent convolutional layers in these convolutional layers are connected in series, and there are skip connections (Skip Connections) between non-adjacent convolutional layers, thereby forming a residual block with a residual structure. For example, a residual block in the ResNet-50 model contains 6 convolutional layers, in which in addition to the 6 convolutional layers being connected in series, there is a jump connection from the output of the previous residual block (i.e., the input of the first convolutional layer) to the input of the fourth convolutional layer, that is, after the output of the previous residual block is fused with the output of the third convolutional layer, the fused feature map is input to the fourth convolutional layer. In addition, there is a jump connection from the input of the fourth convolutional layer to the output of the sixth convolutional layer, that is, after the input of the fourth convolutional layer (referring to the feature map obtained by fusion of the output of the third convolutional layer and the output of the previous residual block) is fused with the output of the sixth convolutional layer, the fused feature map is used as the output of the current residual block and input to the next residual block. The internal structure of other residual blocks in the ResNet-50 model is similar to the above example and will not be repeated here.

[0107] When the initial feature model is a ResNet-50 model, for any sample image among the multiple sample images, the server inputs the sample image into the ResNet-50 model, performs convolution processing based on the residual structure on the sample image through the multiple residual blocks in the ResNet-50 model, inputs the feature map output by the last residual block into a pooling layer and a fully connected layer, and then the fully connected layer outputs the first image feature of the sample image that is finally extracted. The above operation is performed on each sample image to obtain the first image feature of each of the multiple sample images.

[0108] In some embodiments, after obtaining the first image features of each of the multiple sample images, optionally, for any sample image among the multiple sample images, the first image features of the sample image are input into an exponential normalization (Softmax) layer for exponential normalization processing, and the predicted probability that the sample image belongs to each candidate category (i.e., the predicted value of classifying the sample image into the corresponding candidate category) is output, and the candidate category with the highest predicted probability is used as the predicted category of the sample image. The above operation is performed on each sample image to obtain the predicted category of each of the multiple sample images. Next, using the predicted categories of each of the multiple sample images and the reference categories of each of the multiple sample images (i.e., the categories to which the sample images actually belong), the cross entropy loss of this iteration is obtained as the above discriminative parameter.

[0109] In some embodiments, by further introducing a classification additive angular margin to Softmax, a stricter discriminative loss (ArcFace Loss) is obtained for each sample image, and the discriminative parameter is obtained based on the weighted sum of the discriminative losses corresponding to the multiple sample images. How to obtain the discriminative loss will be described in detail in this embodiment and will not be repeated here.

[0110] 202. The server obtains a compatibility parameter based on the first feature model and the second image feature extracted from the multiple sample images, where the compatibility parameter represents the degree of similarity between the first image feature and the second image feature of the same sample image.

[0111] The first feature model involved in the embodiment of the present application refers to an old feature model in the image retrieval system that has not been upgraded, and the target feature model obtained by training the initial feature model in the following step 203 refers to a new feature model used to replace the old feature model. In other words, the optimization goal of the embodiment of the present application is to obtain the target feature model through training, so as to use the target feature model in the image retrieval system to replace the original first feature model, thereby improving the performance of the image retrieval system.

[0112] In some embodiments, the server inputs the multiple sample images into a first feature model, extracts features from the multiple sample images through the first feature model, and outputs second image features corresponding to each of the multiple sample images.

[0113] In some embodiments, the first feature model is any machine learning model that can extract image features. The model architecture of the first feature model includes: ResNet-18, ResNet-34, ResNet-50, ResNet-101, ResNet-152, CNN, DNN, MLP, etc. The embodiments of the present application do not specifically limit the model architecture of the first feature model.

[0114] In some embodiments, in order to ensure that the target feature model has better discriminative parameters than the first feature model, so as to achieve the goal of feature model upgrading, schematically, the feature model is optimized in the following manner: A) the sample capacity of the training sample set used for initial feature model training is made larger than the sample capacity of the training sample set corresponding to the first feature model. Since the sample capacity is expanded, the model performance can be improved; B) the number of categories in the training sample set of the initial feature model is made larger than the number of categories in the training sample set of the first feature model. Since the number of categories is expanded, the model performance can be improved; C) the model structure of the initial feature model is made more complex than the model structure of the first feature model, for example, the neural network of the initial feature model is made At least one of the width, depth or convolution kernel size of the network is greater than that of the first feature model, the above-mentioned "width" refers to the number of convolution kernels contained in each layer of the neural network, "depth" refers to the number of convolution layers contained in the neural network, and "convolution kernel size" refers to the size of the convolution kernel used in each layer of the neural network; D) the loss function construction of the initial feature model is better than the loss function construction of the first feature model. For example, the initial feature model uses ArcFace loss as a discriminative parameter and combines other compatibility parameters to jointly construct the loss function, while the first feature model only uses cross entropy loss as the loss function, etc. It should be noted that only a few ways of optimizing feature models are given here as illustrative illustrations, but they should not constitute a limitation on the way of optimizing feature models.

[0115] In some embodiments, the first feature model is taken as a ResNet-18 model as an example for explanation. The ResNet-18 model is similar to the ResNet-50 model introduced in the above step 201, except that the ResNet-18 model contains 18 convolutional layers, which is less than the 50 convolutional layers contained in the ResNet-50 model, and is not described in detail here.

[0116] Optionally, for any sample image among the multiple sample images, the server inputs the sample image into a ResNet-18 model, performs convolution processing based on the residual structure on the sample image through multiple residual blocks in the ResNet-18 model, inputs the feature map output by the last residual block into a pooling layer and a fully connected layer, and then the fully connected layer outputs the second image features of the sample image that are finally extracted. The above operation is performed on each sample image to obtain the second image features of each of the multiple sample images.

[0117] In some embodiments, for any sample image, a first image feature can be extracted using the initial feature model, and a second image feature can be extracted using the first feature model. Then, the first similarity between the first image feature and the second image feature of the sample image can be obtained. In other words, this first similarity can characterize the feature similarity between the first image feature and the second image feature of the same sample image. Then, the first similarities corresponding to each of the multiple sample images are weighted and summed to obtain the final compatibility parameter.

[0118] 203. The server iteratively adjusts the parameters of the initial feature model based on the discriminative parameter and the similarity parameter to obtain a target feature model.

[0119] In some embodiments, since the goal of the trained target feature model is to replace the first feature model originally existing in the image retrieval model, for the target feature model, not only is it required that the discriminative parameters obtained in the above step 201 are better than the first feature model, which means that the target feature model has stronger expression ability and stronger discriminativeness for the features extracted from the same image than the first feature model, but it is also required that the compatibility parameters obtained in the above step 202 also meet the compatibility conditions. The compatibility conditions refer to that the features extracted by the target feature model can be directly calculated for similarity with the features originally extracted by the first feature model. In this way, without obtaining the original image required for privacy protection, the features of the query image extracted by the target feature model can be directly used to realize the calculation of feature similarity from the feature library composed of the stock image features extracted from the original image by the original first feature model, so as to provide external image retrieval functions normally. Therefore, the discriminative parameters of the above target feature model that are better than the first feature model and the compatibility parameters that meet the compatibility conditions can be collectively referred to as optimization conditions for the first feature model.

[0120] In some embodiments, the server obtains the loss function value of this iteration based on the discriminative parameters and compatibility parameters of this iteration. Then, the server determines whether the first stop condition is met, and the first stop condition refers to the condition for stopping the training of the initial feature model. Optionally, the first stop condition includes at least one of the following: the number of iterations is greater than the first number threshold, the first number threshold is any integer greater than 0, for example, the first number threshold is 30, 50, 100, etc., and the first number threshold in the embodiment of the present application is not specifically limited; or, the loss function value is less than the first loss threshold, the first loss threshold is any value greater than or equal to 0 and less than or equal to 1, for example, the first loss threshold is 0.1, 0.15, 0.2, etc.; or, the difference between the loss function value of this iteration and the loss function value of the previous iteration is less than the first difference threshold, the first difference threshold is any value greater than or equal to 0 and less than or equal to 1, for example, the first difference threshold is 0.01, 0.02, 0.03, etc.

[0121] In some embodiments, when both the number of iterations and the loss function value do not meet the above-mentioned first stop condition, the server adjusts the model parameters of the initial feature model based on the back propagation algorithm, and uses the initial feature model again to extract the first image features of each sample image, and then executes the above steps 201-202 to respectively obtain the discriminative parameters and compatibility parameters of the next iteration, and then obtains the loss function value of the next iteration. The above operations are iteratively performed until either the number of iterations or the loss function value meets the above-mentioned first stop condition, then the iteration is stopped, and the initial feature model used in the last iteration is output as the target feature model.

[0122] In the above process, the target feature model trained by controlling the discriminative parameters has better feature extraction ability than the first feature model, and the new features extracted by the target feature model trained by controlling the compatibility parameters can be compatible with the old features that have been extracted by the first feature model. This means that in a privacy-preserving image retrieval system, the target feature model does not need to use the original image to extract new features to replace the original old features in the feature library. Instead, it optimizes the expression ability and improves the quality of the features of the extracted query image, and ensures that the new features of the query image extracted by the target feature model can directly perform similarity calculations on the feature library formed by the old features extracted from each original image by the first feature model. Therefore, the retrieval performance of the entire image retrieval system is optimized by improving the feature quality of the query image without affecting the normal image retrieval function.

[0123] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0124] The method provided in the embodiment of the present application obtains a discriminative parameter for characterizing the degree of difference of the first image features of sample images under different categories, so that the discriminative parameter can control the trained target feature model to have a better feature extraction capability, and obtains a compatibility parameter for characterizing the degree of similarity between the first image features and the second image features of the same sample image, so that the compatibility parameter can control the target feature model to have a better compatibility with the original first feature model. Therefore, under the joint constraints of the discriminative parameter and the compatibility parameter, the target feature model obtained by training the initial feature model not only has a better feature extraction capability, but also has a strong compatibility capability.

[0125] Furthermore, after the target feature model is launched on the privacy-preserving image retrieval system, when a new query image is received, the target feature model optimizes the expression ability and improves the feature quality of the extracted query image features, and ensures that the query image features extracted by the target feature model can be directly calculated for similarity on the old feature library extracted by the original first feature model, thereby optimizing the retrieval performance of the entire image retrieval system by improving the feature quality of the query image without affecting the normal image retrieval function.

[0126] In the above embodiment, the training process of the target feature model is briefly introduced. In the embodiment of the present application, the training process of the target feature model is introduced in detail by taking the case where the discriminative parameter is the weighted sum of the discriminative losses (ArcFace Loss) of each sample image as an example, which is explained below.

[0127] Figure 3 is a flow chart of a method for obtaining a feature model provided in an embodiment of the present application. Figure 3 This embodiment is executed by a computer device and is described by taking the computer device as a server as an example. This embodiment includes the following steps:

[0128] 301. The server inputs a plurality of sample images into an initial feature model, and obtains first image features of each of the plurality of sample images through extraction using the initial feature model.

[0129] In some embodiments, the server obtains a sample image set, determines multiple sample images used for this training from the sample image set, inputs the multiple sample images into an initial feature model, extracts features from the multiple sample images through the initial feature model, and outputs the first image features of each of the multiple sample images.

[0130] In some embodiments, the above-mentioned multiple sample images are all the images in the sample image set, or the images of the first target ratio in the sample image set are divided into a training sample set, and the remaining images of the second target ratio are divided into a test sample set. The multiple sample images in the training sample set are used as the multiple sample images used in the training stage, and the multiple sample images in the test sample set are used to test the feature extraction performance of the trained target feature model.

[0131] Optionally, the first target ratio and the second target ratio are both numerical values ​​greater than or equal to 0 and less than or equal to 100%, and the sum of the first target ratio and the second target ratio is equal to 100%. For example, the first target ratio is 80% and the second target ratio is 20%, or the first target ratio is 70% and the second target ratio is 30%, etc. The embodiments of the present application do not specifically limit this.

[0132] In some embodiments, the initial feature model is any machine learning model that can extract image features. The model architecture of the initial feature model includes: ResNet-18, ResNet-34, ResNet-50, ResNet-101, ResNet-152, CNN, DNN, MLP, etc. The embodiments of the present application do not specifically limit the model architecture of the initial feature model.

[0133] Schematically, the ResNet-50 model is taken as an example of the initial feature model. The ResNet-50 model includes multiple serially connected residual blocks. The serial connection means that the output of the previous residual block is used as the input of the current residual block. Each residual block includes multiple convolutional layers. Adjacent convolutional layers in these convolutional layers are serially connected, and there are jump connections between non-adjacent convolutional layers, thereby forming a residual block with a residual structure. For example, a residual block in the ResNet-50 model contains 6 convolutional layers, in which in addition to the 6 convolutional layers being connected in series, there is a jump connection from the output of the previous residual block (i.e., the input of the first convolutional layer) to the input of the fourth convolutional layer, that is, after the output of the previous residual block is fused with the output of the third convolutional layer, the fused feature map is input to the fourth convolutional layer. In addition, there is a jump connection from the input of the fourth convolutional layer to the output of the sixth convolutional layer, that is, after the input of the fourth convolutional layer (referring to the feature map obtained by fusion of the output of the third convolutional layer and the output of the previous residual block) is fused with the output of the sixth convolutional layer, the fused feature map is used as the output of the current residual block and input to the next residual block. The internal structure of other residual blocks in the ResNet-50 model is similar to the above example and will not be repeated here.

[0134] In the case where the initial feature model is a ResNet-50 model, for any sample image among the multiple sample images, the server inputs the sample image into the ResNet-50 model, and performs convolution processing based on the residual structure on the sample image through the multiple residual blocks in the ResNet-50 model. After the feature map output by the last residual block is input into a pooling layer and a fully connected layer, the fully connected layer outputs the first image feature of the sample image that is finally extracted. The above operation is performed on each sample image to obtain the first image feature of each of the multiple sample images. Optionally, the feature dimension of the first image feature output by the ResNet-50 model can also be specified, for example, the feature dimension of the output first image feature is set to 512.

[0135] 302. The server inputs the multiple sample images into a first feature model, and extracts second image features of each of the multiple sample images through the first feature model.

[0136] In some embodiments, the server inputs the multiple sample images into a first feature model, extracts features from the multiple sample images through the first feature model, and outputs second image features corresponding to each of the multiple sample images.

[0137] In some embodiments, the first feature model is any machine learning model that can extract image features. The model architecture of the first feature model includes: ResNet-18, ResNet-34, ResNet-50, ResNet-101, ResNet-152, CNN, DNN, MLP, etc. The embodiments of the present application do not specifically limit the model architecture of the first feature model.

[0138] In some embodiments, the first feature model is taken as a ResNet-18 model as an example for explanation. The ResNet-18 model is similar to the ResNet-50 model introduced in the above step 301, except that the ResNet-18 model contains 18 convolutional layers, which is less than the 50 convolutional layers contained in the ResNet-50 model, and is not described in detail here.

[0139] Optionally, for any sample image among the multiple sample images, the server inputs the sample image into a ResNet-18 model, performs convolution processing based on the residual structure on the sample image through multiple residual blocks in the ResNet-18 model, inputs the feature map output by the last residual block into a pooling layer and a fully connected layer, and then the fully connected layer outputs the second image features of the sample image that are finally extracted. The above operation is performed on each sample image to obtain the second image features of each of the multiple sample images.

[0140] 303. The server obtains a discriminative parameter of this iteration based on the first image features of each of the multiple sample images, where the discriminative parameter represents the degree of difference of the first image features of the sample images in different categories.

[0141] In some embodiments, for any sample image among the multiple sample images, the first image feature of the sample image is input into the exponential normalization (Softmax) layer of the initial feature model for exponential normalization processing, and the predicted probability that the sample image belongs to each candidate category (i.e., the predicted value of classifying the sample image into the corresponding candidate category) is output, and the candidate category with the highest predicted probability is used as the predicted category obtained by the initial feature model for classifying the sample image. Then, based on the predicted category of the sample image and the reference category to which the sample image actually belongs, the discriminative loss of the sample image in this iteration is obtained, and the discriminative loss is used to characterize the degree of difference between the predicted category of the sample image and the reference category; then, based on the discriminative losses corresponding to each of the multiple sample images, the discriminative parameter is obtained.

[0142] In some embodiments, for each sample image, the cross entropy between the predicted category and the reference category of the sample image is used as the discriminative loss. The discriminative loss in this case is also called Softmax Loss, and then the weighted sum of the discriminative losses of each sample image is used as the final discriminative parameter, such as taking the arithmetic mean of the discriminative losses of each sample image as the discriminative parameter, or taking the weighted average of the discriminative losses of each sample image as the discriminative parameter, etc. The embodiments of the present application do not specifically limit the discriminative parameters.

[0143] In other embodiments, for each sample image, a classification additive angular margin can be introduced on the basis of Softmax to obtain the ArcFace Loss (additive loss) of the sample image as the discriminative loss, and then the weighted sum of the discriminative losses of each sample image is used as the final discriminative parameter, such as taking the arithmetic mean of the discriminative losses of each sample image as the discriminative parameter, or taking the weighted average of the discriminative losses of each sample image as the discriminative parameter, etc. The embodiments of the present application do not specifically limit the discriminative parameters.

[0144] In some embodiments, the server can obtain the ArcFaceLoss of each sample image in the following manner: obtain the feature angle between the first image feature of the sample image and the category feature of the reference category, for example, first obtain the cosine similarity between the first image feature and the category feature of the reference category, this cosine similarity represents the cosine value of the above-mentioned feature angle, and then use the inverse cosine function to obtain the cosine similarity to obtain the feature angle; then, add the feature angle to the classification additive angle margin to obtain the feature correction angle, and the classification additive angle margin is used to increase the degree of inter-class difference between category features of different categories. Since the feature angle between the first image feature and the category feature of the reference category is increased by the classification additive angle margin, the loss can be reduced by adjusting parameters during the iteration process, so that the feature angle is optimized smaller during the optimization process than when the classification additive angle margin is not used, which promotes the optimization of the same category. The intra-class compactness of the first image features of each sample image is more significant, so that the first image features of each sample image of different categories are easier to distinguish in the feature space, that is, the degree of inter-class difference or inter-class discriminability is more significant; then, based on the cosine value of the feature correction angle and the prediction value of classifying the first image feature of the sample image into multiple candidate categories, the discriminative loss of the sample image is obtained. The prediction value represents the prediction probability of classifying the first image feature into the corresponding candidate category. The prediction probability also refers to the cosine similarity between the first image feature and the category feature of the corresponding candidate category (representing the cosine value of the feature angle between the first image feature and the category feature of the corresponding candidate category). Therefore, it is equivalent to obtaining the cosine value of the feature angle between the first image feature and the category feature of the corresponding candidate category, and then obtaining the cosine value of the feature correction angle, and using the cosine values ​​corresponding to these categories to calculate the Softmax Loss as the final discriminative loss.

[0145] In the above process, it is equivalent to increasing the feature angle between the first image feature and the category feature of the reference category through the classification additive angle margin, while keeping the feature angle between the first image feature and the category features of the remaining candidate categories except the reference category unchanged (that is, no feature angle increase is performed). This is equivalent to only applying resistance to the feature angle between the first image feature and the category feature of the reference category. During the iteration process, this feature angle is optimized to be smaller (equivalent to being one classification additive angle margin smaller than the feature angle when the optimization stops under normal circumstances). This makes the intra-class compactness of the first image features of each sample image under the same category more significant, thereby making the first image features of each sample image of different categories easier to distinguish in the feature space, that is, making the degree of inter-class difference more significant, so that the first image features have stronger discriminability and stronger expressiveness.

[0146] 304. The server obtains a compatibility parameter of this iteration based on the first image feature and the second image feature of each of the multiple sample images. The compatibility parameter represents the similarity between the first image feature and the second image feature of the same sample image.

[0147] In some embodiments, for any sample image, a first image feature can be extracted using the initial feature model, and a second image feature can be extracted using the first feature model. Then, the first similarity between the first image feature and the second image feature of the sample image can be obtained. In other words, this first similarity can characterize the feature similarity between the first image feature and the second image feature of the same sample image. Then, the first similarities corresponding to each of the multiple sample images are weighted and summed to obtain the final compatibility parameter.

[0148] In some embodiments, for each sample image, the cosine similarity between the first image feature and the second image feature of the sample image is used as the first similarity, or the inverse of the Euclidean distance between the first image feature and the second image feature of the sample image is used as the first similarity, or other methods are used to characterize the first similarity, so that when the first image feature and the second image feature are more similar, the first similarity value is larger, and when the first image feature and the second image feature are more dissimilar, the first similarity value is smaller.

[0149] In some embodiments, for each sample image, when the cosine similarity between the first image feature and the second image feature of the sample image is used as the first similarity, the difference obtained by subtracting the cosine similarity of the sample image from 1 can be used as the cosine feature loss of the sample image, and then the cosine feature losses of each sample image are added together to obtain a compatibility parameter, or the arithmetic mean of the cosine feature losses of each sample image is used as the compatibility parameter, or the weighted average of the cosine feature losses of each sample image is used as the compatibility parameter, etc. The embodiments of the present application do not specifically limit this.

[0150] 305. The server obtains the loss function value of the initial feature model for this iteration based on the discriminative parameters and compatibility parameters of this iteration.

[0151] In some embodiments, the server obtains the loss function value of this iteration based on the discriminative parameters and compatibility parameters of this iteration, and then, if both the number of iterations and the loss function value do not meet the first stop condition, the server proceeds to the following step 306; if either the number of iterations or the loss function value meets the first stop condition, the server proceeds to the following step 307.

[0152] In some embodiments, the server determines a preset hyperparameter as a balancing factor, multiplies the balancing factor and the compatibility parameter to obtain a balanced compatibility parameter, and adds the balanced compatibility parameter and the discriminative parameter to obtain the loss function value of this iteration.

[0153] Schematically, assume that the first characteristic model is represented by Φ old , the initial feature model is represented by Φ new , the discriminative parameter of the initial feature model is expressed as l base (Φ new ), the compatibility parameter between the initial feature model and the first feature model is expressed as l compatible (Φ new , Φ old ), the hyperparameter balance factor is denoted as λ, then the loss function value of the initial feature model in this iteration is L BCT (Φ new ) can be expressed as the following formula:

[0154] L BCT (Φ new )=l base (Φ new )+λl compatible (Φ new , Φ old )

[0155] The balance factor λ is a hyperparameter preset by the technicians. The balance factor λ can be any value greater than 0. For example, the balance factor λ = 2c

[0156] It can be seen that in the above loss function formula, l base Responsible for constraining the initial feature model Φ new The discriminant parameter of the constrained initial feature model Φ new The first image feature extracted has discriminativeness (it has the ability to distinguish image features of sample images of different categories), for example, l base The traditional Softmax Loss or the ArcFace Loss involved in the embodiments of the present application can be used; in addition, compatible It is called the backward compatible loss function, which ensures that the feature model is old To the initial feature model Φ new During the forward iteration of the trained target feature model, the initial feature model Φ is constrained new The extracted first image feature and the first feature model Φ oldThe extracted second image features have a certain compatibility (i.e. similarity), so that it can be ensured that the second image features (i.e. new features) can be directly searched on the first image features (i.e. old features), ensuring that the image retrieval performance remains normal after the feature model is upgraded.

[0157] In schematic form, the sum of the cosine feature losses of each sample image is used as the compatibility parameter l compatible , where the cosine feature loss refers to the difference between 1 and the cosine similarity between the first image feature and the second image feature, that is, the first similarity. Then the compatibility parameter l compatible It can be expressed as the following formula:

[0158]

[0159] Where D refers to the initial feature model Φ new The training sample set is composed of multiple sample images, i refers to the sequence number of the sample image in the training sample set D (i.e., the i-th sample image), Φ new (i) refers to the initial feature model Φ new The first image feature extracted from the i-th sample image, Φ old (i) refers to the first characteristic model Φ old For the second image feature extracted from the i-th sample image, cos(Φ new (i), Φ old (i)) refers to the first image feature Φ of the i-th sample image new (i) and the second image feature Φ old The cosine similarity between (i) is the first similarity.

[0160] It can be seen that the above compatibility parameter l compatible The definition formula constrains that for each sample image i∈D, the first image feature Φ of the sample image is guaranteed new (i) and the second image feature Φ old (i) As similar as possible, so as to ensure that the initial feature model Φ new After the trained target feature model is put online in the image retrieval system, the target feature model Φ is used for any query image. new The extracted new features can be directly combined with the first feature model Φ old The extracted old features are directly used for similarity calculation to provide better retrieval performance.

[0161] 306. When the number of iterations and the loss function value do not meet the first stopping condition, the server adjusts the parameters of the initial feature model, and iteratively executes the above steps 301-305 based on the initial feature model with adjusted parameters.

[0162] The first stopping condition refers to the condition for stopping training the initial feature model.

[0163] Optionally, the first stop condition includes at least one of the following: the number of iterations is greater than a first number threshold, the first number threshold is any integer greater than 0, for example, the first number threshold is 30, 50, 100, etc., and the first number threshold is not specifically limited in the embodiment of the present application; or, the loss function value is less than a first loss threshold, the first loss threshold is any value greater than or equal to 0 and less than or equal to 1, for example, the first loss threshold is 0.1, 0.15, 0.2, etc.; or, the difference between the loss function value of this iteration and the loss function value of the previous iteration is less than a first difference threshold, the first difference threshold is any value greater than or equal to 0 and less than or equal to 1, for example, the first difference threshold is 0.01, 0.02, 0.03, etc.

[0164] In some embodiments, when both the number of iterations and the loss function value do not meet the above-mentioned first stop condition, the server transmits back the model parameters of the adjusted initial feature model based on the back propagation algorithm, and again uses the initial feature model with adjusted parameters to extract the first image features of each sample image, and then executes the above steps 301-305 to obtain the discriminative parameters and compatibility parameters of the next iteration respectively, and then obtain the loss function value of the next iteration, and iteratively executes the above operations until either the number of iterations or the loss function value meets the above-mentioned first stop condition, and then enters the following step 307.

[0165] 307. When either the number of iterations or the loss function value meets the first stopping condition, the server outputs the initial feature model of the last iteration as the target feature model.

[0166] In some embodiments, when either the number of iterations or the loss function value meets the first stop condition described above, the server stops iterating and outputs the initial feature model of the last iteration as a target feature model, wherein the discriminative parameter and the compatibility parameter of the target feature model meet the optimization conditions for the first feature model.

[0167] In some embodiments, since the training target feature model aims to replace the first feature model originally existing in the image retrieval model, the target feature model is not only required to have a discriminative parameter better than the first feature model in the above step 303, which means that the target feature model has stronger expression ability and stronger discriminative power than the first feature model for the features extracted from the same image, but also required to have a compatibility parameter obtained in the above step 304 that also satisfies the compatibility condition, which means that the features extracted by the target feature model can be directly similar to the features originally extracted by the first feature model. In this way, without obtaining the original image required for privacy protection, the features of the query image extracted by the target feature model can be directly used to calculate the feature similarity from the feature library composed of the features of the original image extracted by the first feature model, so as to provide external image retrieval function normally. Therefore, the discriminant parameters of the above-mentioned target feature model are better than the first feature model and the compatibility parameters meet the compatibility conditions, which can be collectively referred to as the optimization conditions for the first feature model, and the loss function value is jointly constructed by using the discriminant parameters and the compatibility parameters, so that the loss function value can take into account the above-mentioned optimization conditions for the first feature model at the same time.

[0168] In an illustrative manner, the initial feature model is trained under the constraint of the loss function formula provided in the above step 305. On the one hand, the compatibility parameter l compatible Under the constraint of maintaining the first characteristic model Φ old The backward compatibility of the extracted old features (i.e., the features of the original image) is achieved, so that when a new query image comes, the trained target feature model Φ is used. new The new features extracted from the query image can be directly retrieved from the feature library composed of old features. On the other hand, due to the target feature model Φ new The advantages of improving the network structure or expanding the training sample set are as follows: base Under the constraint of old New features that are more discriminative.

[0169] In schematic form, when the difference between two consecutive iterations of the loss function value is observed to be less than the first difference threshold, indicating that the loss function value no longer decreases significantly, the training can be stopped to obtain the target feature model Φ with backward compatibility characteristics. new (i.e., new feature model), this target feature model Φ new Can immediately launch the current image retrieval system to replace the first feature model Φ old When the server receives an image query request, it obtains the query image Q carried in the image query request and uses the trained target feature model Φ newTo extract the features Φ of the query image Q new (Q), and using Φ new (Q) Directly in the old feature library Φ composed of the old features of each original image G old (G) to retrieve similar samples, thereby returning the most similar samples retrieved. new Improve the features Φ of the query image Q new (Q) quality, thereby improving the retrieval accuracy and retrieval performance of the image retrieval system.

[0170] In the above process, the target feature model trained by controlling the discriminative parameters has better feature extraction ability than the first feature model, and the new features extracted by the target feature model trained by controlling the compatibility parameters can be compatible with the old features that have been extracted by the first feature model. This means that in a privacy-preserving image retrieval system, the target feature model does not need to use the original image to extract new features to replace the original old features in the feature library. Instead, it optimizes the expression ability and improves the quality of the features of the extracted query image, and ensures that the new features of the query image extracted by the target feature model can directly perform similarity calculations on the feature library formed by the old features extracted from each original image by the first feature model. Therefore, the retrieval performance of the entire image retrieval system is optimized by improving the feature quality of the query image without affecting the normal image retrieval function.

[0171] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0172] The method provided in the embodiment of the present application obtains a discriminative parameter for characterizing the degree of difference of the first image features of sample images under different categories, so that the discriminative parameter can control the trained target feature model to have a better feature extraction capability, and obtains a compatibility parameter for characterizing the degree of similarity between the first image features and the second image features of the same sample image, so that the compatibility parameter can control the target feature model to have a better compatibility with the original first feature model. Therefore, under the joint constraints of the discriminative parameter and the compatibility parameter, the target feature model obtained by training the initial feature model not only has a better feature extraction capability, but also has a strong compatibility capability.

[0173] Furthermore, after the target feature model is launched on the privacy-preserving image retrieval system, when a new query image is received, the target feature model optimizes the expression ability and improves the feature quality of the extracted query image features, and ensures that the query image features extracted by the target feature model can be directly calculated for similarity on the old feature library extracted by the original first feature model, thereby optimizing the retrieval performance of the entire image retrieval system by improving the feature quality of the query image without affecting the normal image retrieval function.

[0174] In the above embodiment, it is described in detail how to train a target feature model with backward compatibility to replace the original first feature model, thereby improving the retrieval performance of the image retrieval system from the aspect of improving the feature quality of the query image. In the embodiment of the present application, the process of training a feature upgrade model with forward compatibility based on the trained target feature model will be introduced, which will be explained below.

[0175] Figure 4 is a flow chart of a method for training a feature upgrade model provided in an embodiment of the present application. Figure 4 This embodiment is executed by a computer device and is described by taking the computer device as a server as an example. This embodiment includes the following steps:

[0176] 401. The server inputs a plurality of sample images into a target feature model, and obtains target image features of each of the plurality of sample images through extraction of the target feature model.

[0177] The above step 401 is similar to the above step 301, except that the multiple sample images are again input into the target feature model trained based on the initial feature model, so as to extract the target image features of the multiple sample images respectively. Since the target feature model and the initial feature model have the same model structure, they will not be elaborated here.

[0178] 402. The server inputs the multiple sample images into a first feature model, and extracts second image features of each of the multiple sample images through the first feature model.

[0179] The above step 402 is similar to the above step 302 and will not be described in detail here.

[0180] 403. The server inputs the second image features of each of the multiple sample images into an initial upgrade model to obtain the third image features of each of the multiple sample images.

[0181] In some embodiments, the initial upgrade model is any machine learning model that can further extract new features with stronger expressive power based on existing old features. The model architecture of the initial upgrade model includes: MLP, CNN, DNN, ResNet, etc. The embodiment of the present application does not specifically limit the model architecture of the initial upgrade model.

[0182] Schematically, the initial upgrade model is taken as an example of an MLP model. The MLP model includes at least one basic module, each of which includes a fully connected (FC) layer, a batch normalization (BN) layer, and a nonlinear activation (ReLU) layer connected in series. In other words, each basic module can be expressed as: [FC→BN→ReLU]. In one example, three series-connected basic modules are set in the MLP model, the dimension of the fully connected layer is set to 4096, and an output FC layer is added after the last basic module to form the final initial upgrade model ψ.

[0183] Furthermore, for any sample image among the multiple sample images, the server inputs the second image feature extracted by the first feature model for the sample image into at least one basic module of the initial upgrade model. For each basic module, the FC layer in the basic module is first used to fully connect the input feature, and then the BN layer in the basic module is used to batch normalize the output feature of the FC layer. Then, the ReLU layer in the basic module is used to add nonlinear mapping to the output feature of the BN layer, and finally the output feature of the ReLU layer in the current basic module is input into the next basic module. After the last basic module is processed, the output feature of the last basic module is input into the output FC layer for fully connected output, and the output FC layer outputs the third image feature finally extracted from the sample image.

[0184] In the above process, it is equivalent to constructing the upgraded new features (third image features) of the sample image using only the old features (second image features) of the sample image, without the need for the initial upgraded model to perceive the sample image itself. In this way, after the initial upgraded model is trained, it has the function of directly upgrading the old features to new features with stronger expression capabilities. In addition, this upgrading process does not need to use the original image itself at all, so that the trained feature upgrading model can be used to move towards the target feature model Φ trained in the previous embodiment. new The feature space of the original first feature model Φ old The feature library composed of old features extracted from each original image is forward updated. Under the condition of not saving the original image under the condition of satisfying privacy protection, the disadvantage that the old feature library is fixed when only the backward-compatible target feature model is trained, resulting in limited improvement in system retrieval performance, is improved.

[0185] 404. The server obtains a similarity parameter based on the third image feature and the target image feature of each of the multiple sample images, where the similarity parameter represents the degree of similarity between the third image feature and the target image feature of the same sample image.

[0186] In some embodiments, for any sample image, a target image feature can be extracted using a target feature model, and the second image feature originally extracted by the first feature model can be upgraded using an initial upgraded model to obtain a corresponding third image feature. Then, a second similarity between the third image feature of the sample image and the target image feature can be obtained. The second similarity can characterize the feature similarity between the third image feature and the target image feature of the same sample image. Based on the second similarities corresponding to each of the multiple sample images, a similarity parameter of the initial upgraded model in this iteration is obtained. For example, a weighted sum is performed on the second similarities corresponding to each of the multiple sample images to obtain the similarity parameter.

[0187] In some embodiments, for each sample image, the cosine similarity between the third image feature of the sample image and the target image feature is used as the second similarity, or the inverse of the Euclidean distance between the third image feature of the sample image and the target image feature is used as the second similarity, or other methods are used to characterize the second similarity, so that when the third image feature and the target image feature are more similar, the second similarity value is larger, and when the third image feature and the target image feature are less similar, the second similarity value is smaller.

[0188] In some embodiments, for each sample image, when the cosine similarity between the third image feature of the sample image and the target image feature is used as the second similarity, the difference obtained by subtracting the cosine similarity of the sample image from 1 can be used as the cosine feature loss of the sample image, and then the cosine feature losses of each sample image are added together to obtain a similarity parameter, or the arithmetic mean of the cosine feature losses of each sample image is used as the similarity parameter, or the weighted average of the cosine feature losses of each sample image is used as the similarity parameter, etc. The embodiments of the present application do not specifically limit this.

[0189] In one example, assuming that the initial upgraded model is denoted as Ψ, the loss function value of the initial upgraded model is denoted as L FCT (Ψ), the first characteristic model is represented by Φ old , the target feature model is represented by Φ new , then the above similarity parameter can be expressed as l compatible (Ψ(Φ old ), Φ new), assuming that the similarity parameter is used as the loss function value of the initial upgrade model, then the loss function value L of the initial upgrade model FCT (Ψ) can be expressed as the following formula:

[0190] L FCT (Ψ)=l compatible (Ψ(Φ old ), Φ new )

[0191] Among them, Ψ(Φ old ) indicates that the input of the initial upgrade model Ψ is the first feature model Φ old The output features of (i.e., the second image features, also called old features), l compatible It also represents the third image feature and target feature model Φ after the initial upgrade model Ψ is optimized new The similarity parameter between the extracted target image features is expressed by the similarity parameter l compatible To constrain the feature Ψ(Φ old (i)) To match the target feature model Φ new The extracted features Φ new (i) Be as similar as possible.

[0192] In schematic form, the sum of the cosine feature losses of each sample image is used as the similarity parameter l compatible , where the cosine feature loss refers to the difference between 1 and the cosine similarity between the third image feature and the target image feature, that is, the second similarity. Then the similarity parameter l compatible It can be expressed as the following formula:

[0193]

[0194] Where D refers to the training sample set consisting of multiple sample images of the initial upgraded model ψ (for example, new Using the same training sample set), i refers to the sequence number of the sample image in the training sample set D (i.e., the i-th sample image), Φ new (i) refers to the target feature model Φ new For the target image features extracted from the i-th sample image, Φ old (i) refers to the first characteristic model Φ old For the second image feature extracted from the i-th sample image, Ψ(Φ old (i)) refers to the second image feature Φ of the i-th sample image for the initial upgraded model Ψ old (i) The third image feature obtained by upgrading, cos(Ψ(Φ old (i)), Φ new(i)) refers to the third image feature Ψ(Φ old (i)) and the target image feature Φ new The cosine similarity between (i) is the second similarity.

[0195] 405. When the number of iterations and the similarity parameter do not meet the second stop condition, the server adjusts the parameters of the initial upgrade model, and iteratively executes the above steps 401-404 based on the initial upgrade model after the adjustment of the parameters.

[0196] The second stopping condition refers to the condition for stopping training of the initial upgraded model.

[0197] In the embodiment of the present application, the similarity parameter is taken as the loss function value of the initial upgrade model for explanation. At this time, after the similarity parameter is obtained through the above step 404 in this iteration, when the number of iterations and the similarity parameter do not meet the second stop condition, the server adjusts the model parameters of the initial upgrade model based on the back propagation algorithm, and executes the above steps 401-404 again based on the initial upgrade model after adjusting the parameters to obtain the similarity parameters of the next iteration. The above operations are iteratively performed until either the number of iterations or the similarity parameter meets the above second stop condition, and then enters the following step 406.

[0198] Optionally, the second stop condition includes at least one of the following: the number of iterations is greater than a second number threshold, the second number threshold is any integer greater than 0, for example, the second number threshold is 30, 50, 100, etc., and the second number threshold in the embodiment of the present application is not specifically limited; or, the similarity parameter is less than a second loss threshold, the second loss threshold is any value greater than or equal to 0 and less than or equal to 1, for example, the second loss threshold is 0.1, 0.15, 0.2, etc.; or, the difference between the similarity parameter of this iteration and the similarity parameter of the previous iteration is less than a second difference threshold, the second difference threshold is any value greater than or equal to 0 and less than or equal to 1, for example, the second difference threshold is 0.01, 0.02, 0.03, etc.

[0199] 406. When either the number of iterations or the similarity parameter meets the second stop condition, the server outputs the initial upgraded model of the last iteration as the feature upgraded model.

[0200] In some embodiments, when either the number of iterations or the similarity parameter meets the second stop condition, the server stops iterating and outputs the initial upgraded model of the last iteration as a feature upgraded model, wherein the similarity parameter of the feature upgraded model meets the optimization condition for the image features extracted from the first feature model.

[0201] In the above process, it is equivalent to iteratively adjusting the parameters of the initial upgrade model based on the similarity parameter to obtain the feature upgrade model, which is used to optimize the image features extracted by the first feature model. Since the goal of the trained feature upgrade model is to upgrade the feature library composed of the old features extracted by the first feature model from each original image in the image retrieval system, so that the old features can be gradually updated from the original feature space to the new feature space corresponding to the target feature model, but under the privacy protection requirement, since the original image is not available, it is impossible to directly use the target feature model to re-extract the new features of each original image to replace the entire feature library. Therefore, by training the feature upgrade model of the embodiment of the present application, it is possible to directly use the old features of the original image (i.e., the second image features) as input without obtaining the original image, and extract the upgraded new features (i.e., the third image features) through the feature upgrade model, and impose constraints through the similarity parameters so that the upgraded third image features are as similar as possible to the target image features extracted by the target feature model, thereby further improving the retrieval performance of the image retrieval system from the perspective of optimizing the old features in the original feature library.

[0202] In schematic form, when it is observed that the difference between two consecutive iterations of the similarity parameter is less than the second difference threshold, which means that the similarity parameter no longer decreases significantly, the training can be stopped to obtain a feature upgrade model Ψ with forward compatibility characteristics. The trained feature upgrade model Ψ can also be used to update the features of each original image in the original feature library, thereby achieving forward compatibility optimization of the entire feature library.

[0203] In some embodiments, the server obtains the first feature model Φ old Corresponding to multiple stock image features in the feature library Φ old (G), the stock image feature Φ old (G) refers to the first feature model Φ already in the feature library old The old inventory features extracted from the original image G are then converted into the multiple inventory image features Φ old (G) Input the feature upgrade model Ψ to obtain multiple inventory upgrade features Ψ (Φ old (G)), inventory upgrade feature Ψ(Φ old (G)) refers to the feature Φ in the stock image old (G) is used to obtain new features by upgrading and optimizing the features. Then, in the feature library, the multiple stock image features Φ old (G) (i.e. old features) are replaced by their corresponding inventory upgrade features Ψ(Φ old (G)) (i.e., the new features obtained by upgrading) can achieve forward compatibility optimization of the entire feature library:

[0204] The first characteristic model Φ involved in the embodiment of the present application old , target feature model Φ new The feature spaces of the three are compatible with each other, which is equivalent to training the target feature model Φ with backward compatibility characteristics on the training sample set D. new And a feature upgrade model Ψ with forward compatibility, thus providing a bidirectional compatible model upgrade solution. Since the training sample set D and the original images required by privacy protection are of the same type but completely non-overlapping image datasets (that is, although the original images and sample images are of the same type of data, there is no intersection in any specific data), and the training sample set D is put into the training process only after the corresponding user has fully authorized and agreed separately, it can fully meet the privacy protection requirements. In addition, the feature upgrade model Ψ is used to upgrade the feature Φ of the stock image in the feature library. old The upgrade process of (G) can be carried out online in real time and in batches, without the need to update the entire feature library offline and then upload it to the image retrieval system for coverage. This means that there is no need to shut down for maintenance during feature upgrades, which greatly reduces the cost and loss of feature upgrades of the image retrieval system. Moreover, when the entire feature library is upgraded online in real time, the retrieval accuracy of the image retrieval system will also be gradually improved in real time. Finally, all inventory image feature Φ old (G) will be upgraded online, and the two-way compatible model upgrade process is completed.

[0205] Optionally, the above-mentioned batch-by-batch online model upgrade refers to: gradually using the feature upgrade model Ψ to upgrade some inventory image features Φ of the current batch in the feature library old (G) to upgrade. After performing the above upgrade operation in multiple batches, the entire feature library can be upgraded, thereby gradually using the inventory upgrade feature Ψ(Φ old (G)) Replace the original stock image feature Φ old (G).

[0206] Figure 5 is a principle flow chart of a bidirectional compatible model upgrade solution provided by an embodiment of the present application, such as Figure 5 As shown, in the bidirectional compatible model upgrade solution, for each sample image 501 in the training sample set D, the sample image is input into the old feature model 502 (i.e., the first feature model Φ old ) and the new feature model 503 (i.e., the target feature model Φ new ), the old feature model 502 outputs the old feature 5011 (ie, the second image feature) of the sample image 501, and the new feature model 503 outputs the new feature 5012 (ie, the target image feature) of the sample image 501.

[0207] On the one hand, a new feature model 503 with backward compatibility is provided through backward compatible training (BCT), so that the new feature model 503 is used to replace the old feature model 502, so that the image retrieval system can extract features of the query image, which can improve the feature quality of the query image.

[0208] On the other hand, forward compatible training (FCT) is used to provide a feature upgrade model 504 (i.e., feature upgrade model Ψ) with forward compatibility characteristics, and the old features 5011 extracted from the old feature model 502 are input into the feature upgrade model 504. The feature upgrade model 504 outputs new features 5013 that are upgraded from the old features 5012. The feature upgrade model 504 can thus be used to improve the old feature library in the image retrieval system, thereby improving the feature quality of the inventory image features.

[0209] The new feature model 503 combined with backward compatible training and the feature upgrade model 504 combined with forward compatible training can be collectively referred to as a feature model upgrade scheme of bidirectional compatible training (BiCT). This bidirectional compatible model upgrade scheme can fully meet the privacy protection scenario. It can complete the upgrade of the feature model and the upgrade of the stock image features without obtaining the original images corresponding to the stock image features in the feature library, thereby effectively improving the retrieval accuracy and retrieval performance of the entire image retrieval system.

[0210] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0211] The method provided in the embodiment of the present application provides a feature upgrade model with forward compatibility characteristics. It can achieve feature upgrade by only using stock image features without obtaining the original images corresponding to the stock image features in the feature library, so as to obtain upgraded stock upgrade features. Since constraints are imposed by similarity parameters when training the feature upgrade model, the stock upgrade features and the target image features extracted by the target feature model belong to the same feature space, thereby ensuring that the target feature model can also perform similar sample retrieval on the new feature library composed of the stock upgrade features, thereby meeting the optimization needs of instantly updating the feature library under privacy protection requirements.

[0212] Furthermore, the feature upgrade model with forward compatibility provided in the embodiment of the present application is combined with the target feature model with backward compatibility provided in the previous embodiment, which is equivalent to forming a bidirectional compatible model upgrade solution, which can not only optimize the feature quality of the query image through the target feature model, but also update the inventory image features in the old feature library through the feature upgrade model, so that the upgraded inventory upgraded features are as close as possible to the new feature space corresponding to the target feature model (since the feature quality is better in the new feature space, this process is called feature upgrading). Since the feature upgrade model only upgrades the old features to new features without using the original images corresponding to the old features, it can well solve the dilemma of needing to use the original images when upgrading the old feature library, but not storing the original images under privacy protection requirements.

[0213] Furthermore, the bidirectional compatible model upgrade solution provided in the embodiment of the present application can be applied to various image retrieval systems involving privacy protection, such as object authentication (finding the most similar object in the object feature library and determining whether it is the same as the query object), infringement detection (detecting whether the query sample is plagiarized in the original database), etc. In these privacy-protected image retrieval systems, feature model upgrades and inventory old feature upgrades can be achieved without storing the original image, thereby effectively improving the retrieval accuracy of the image retrieval system.

[0214] Taking the image retrieval system that provides object authentication function as an example, since most object-related image data are usually sensitive privacy data, after the server collects the original image containing the object for the first time and extracts the inventory image features and stores them in the feature library, the image retrieval system only retains the inventory image features in the feature library and the corresponding object identifier (such as object number, object identification code, etc.), but does not retain the original image containing the object. Alternatively, the server provides the user with a feature model (referring to the old feature model) so that the user can use the feature model on the terminal side to extract the inventory image features and then upload the inventory image features to the image retrieval system on the server side. In this way, the server side always contacts the inventory image features instead of the original image containing the object. This kind of situation has a stronger protection for privacy data. After that, the user provides a query image to the image retrieval system, and the image retrieval system extracts the features of the query image (or the user extracts the features of the query image and uploads them to the image retrieval system) to compare with the various inventory image features in the object feature library, returns the object identifier corresponding to the most similar sample, and verifies whether the query image and the most similar sample correspond to the same object based on whether the feature similarity between the two features is greater than the set threshold.

[0215] For this type of privacy-protected retrieval system, the embodiments of the present application can provide a bidirectionally compatible feature model upgrade method and a stock image feature upgrade method in the absence of an original image containing an object, thereby efficiently improving the object retrieval performance of the image retrieval system. First, the trained new feature model (i.e., the target feature model) has backward compatibility and can be directly compared with the old feature library, so that the image retrieval system can be put online in real time, and the retrieval performance of the entire retrieval system can be improved by improving the features of the query image; secondly, the trained feature upgrade model can upgrade the stock image features of the old feature library, so that it can be updated as much as possible to the new feature space corresponding to the above-mentioned new feature model. Since the feature library has been updated, although the upgraded stock upgrade features and the target image features extracted by the new feature model are not exactly the same, they can be kept as close as possible, so the feature quality of the feature library has also been improved and improved to a certain extent, thereby further improving the retrieval performance of the entire image retrieval system.

[0216] Below, the test performance of the bidirectional compatible model upgrade solution provided by the embodiment of the present application is described in detail. During the test, three commonly used retrieval data sets were used: GLDv2 (Google Landmark Datasetv2), ROxford (Revisited Oxford) and RParis (Revisited Paris) as test sample sets, and four possible model upgrade methods (expressed in the form of "old feature model training setting → new feature model setting") were used: data volume expansion of the training sample set (25% data → 100% data, i.e., data volume expansion by 4 times), data category expansion of the training sample set (25% class → 100% class, i.e., data category expansion by 4 times), feature model structure upgrade (ResNet-18 → ResNet-50, i.e., deepening the number of model layers) and feature model discriminative loss improvement (Softmax Loss → ArcFace Loss, replacing the discriminative parameter from Softmax loss to ArcFace loss) to verify the effectiveness of this solution.

[0217] It should be noted that the test uses the mAP (mean Average Precision) indicator to measure the retrieval performance. The larger the mAP value, the better the retrieval performance, and the smaller the mAP value, the worse the retrieval performance. The test results of the above four model upgrade methods on three test sample sets are shown in Table 1.

[0218] Table 1

[0219]

[0220]

[0221] Among them, M o2o refers to the retrieval performance of the old feature model (i.e., the first feature model) itself, M BCT It refers to the retrieval performance after the backward-compatible new feature model (i.e., target feature model) is put into the image retrieval system. FCT It refers to the retrieval performance after the backward-compatible new feature model is put online in the image retrieval system and the forward-compatible feature upgrade model is used to upgrade the old features in the feature library. n2n refers to the retrieval performance of the new feature model itself. However, due to the requirement of privacy protection, it is impossible to use the original image to re-extract new features, so M n2n The search performance of is not achievable, and is listed here only for reference. In addition, the symbol “↑” represents the difference between M o2o The symbol “↓” represents the improvement of retrieval performance compared with M o2o The degree of retrieval performance degradation.

[0222] From the data in Table 1, we can see that, in addition to the data category expansion experiment on the ROxford test sample set, the retrieval performance M after the backward compatible new feature model is launched on the image retrieval system is BCT Compared with the old feature model's own retrieval performance M o2o Furthermore, based on the backward-compatible new feature model on-line image retrieval system, the forward-compatible feature upgrade model is used to complete the retrieval performance M of the upgraded old features in the feature library. FCT , and compared with the old feature model's own retrieval performance M o2o The consistency has been significantly improved in various test sample sets and model upgrade methods. In addition, compared with the retrieval performance of the new feature model that is only backward compatible, BCT It also achieved general improvement, which illustrates the importance of forward-compatible feature upgrade after backward-compatible training, and it also proves that the two-way compatible model upgrade scheme has a stronger system improvement advantage than the backward-compatible model upgrade scheme alone.

[0223] In addition, another set of tests was conducted on the test sample set GLDv2, in which two consecutive model upgrades (i.e., two-generation model upgrades) were performed on the same old feature model, involving three possible model upgrade methods: data volume expansion of the training sample set (25%→50%→100%, i.e., the data volume was expanded by 2 times each time), data category expansion of the training sample set (25%→50%→100%, i.e., the data category was expanded by 2 times each time), and feature model structure upgrade (ResNet-18→ResNet-34→ResNet-101, i.e., the number of model layers continued to increase). The test results are shown in Table 2.

[0224] Table 2

[0225]

[0226]

[0227] The BCT method means that only backward compatible training is performed, and the backward compatible new feature model is upgraded twice in succession (that is, the query image feature quality is improved twice, and the inventory features are not updated). The "n / a" in the table means not applicable. Since the BCT method does not combine forward compatible training, the retrieval performance M of the feature upgrade combined with forward compatible training is FCT Naturally not applicable.

[0228] The BiCT method means that: on the basis of backward compatibility training, forward compatibility training is also performed to obtain a feature upgrade model to upgrade the old inventory features in the feature library (that is, the query image features and inventory features are improved twice). Therefore, BiCT refers to a two-way compatible model upgrade solution. BCT represents the retrieval performance after backward-compatible training improves query image features, M FCT Represents the retrieval performance after improving inventory features through bidirectional compatibility training.

[0229] From the data in Table 2, it can be seen that when only the backward compatible BCT mode is used for training, the improvement brought by upgrading the feature model for multiple generations has certain limitations. However, when the bidirectional compatible BiCT mode is used for training in the embodiment of the present application, after the backward compatible feature model is upgraded, the forward compatible feature upgrade model can be used to timely update the old features in the feature library, so that the feature library can also be upgraded forward. Therefore, higher benefits can be obtained in the same generation model upgrade, that is, the new feature model is applied to the improvement of the query image feature quality and guides the upgrade of the old features in the inventory. Therefore, in the same generation model upgrade, there are M FCT >M BCTIn addition, since the constraint that the new feature model needs to be compatible with the old inventory features extracted by the oldest feature model is weakened in the subsequent model upgrade (the old inventory features are also forward upgraded in the BiCT method, and the new feature model only needs to be compatible with the upgraded new inventory features), better model upgrade retrieval performance is also obtained in the subsequent second-generation model upgrade, that is, starting from the second-generation model upgrade, the retrieval performance M under the BiCT method is BCT and M FCT The retrieval performance of the BCT method is better than that of the BCT high.

[0230] In the above-mentioned embodiments, the training process of the target feature model, the training process of the feature upgrade model, and the test performance improvement of the feature model upgrade and the upgrade of the old inventory features through a two-way compatible training method are respectively introduced in detail. In the embodiments of the present application, it will be introduced in detail how the image retrieval system utilizes the upgraded target feature model and the upgraded inventory upgrade features to realize the image retrieval function, which is explained below.

[0231] Figure 6 is a flow chart of an image processing method provided by an embodiment of the present application. Figure 6 This embodiment is executed by a computer device and is described by taking the computer device as a server as an example. This embodiment includes the following steps:

[0232] 601. The server responds to an image query request and obtains a query image carried in the image query request.

[0233] In some embodiments, the server receives any request sent by the terminal, parses the header field of the request, and when the header field carries an image query identifier, determines that the request is an image query request, and then parses the data field of the image query request to obtain the query image carried in the image query request.

[0234] 602. The server extracts query image features of the query image based on a target feature model, where the target feature model is obtained by iteratively adjusting parameters of an initial feature model based on a discriminative parameter and a compatibility parameter.

[0235] In some embodiments, the server inputs the query image into a trained target feature model, extracts features of the query image through the target feature model, and outputs query image features of the query image. The training method of the target feature model can be found in the above-mentioned embodiments and will not be repeated here.

[0236] The above step 602 is similar to step 401 in the previous embodiment, except that the sample image is replaced by the query image, which will not be described in detail here.

[0237] 603. The server determines, from among a plurality of inventory upgrade features in the feature library, a target inventory feature that has the highest matching degree with the query image feature, the inventory upgrade feature being obtained by processing the inventory image feature extracted by the first feature model by the feature upgrade model.

[0238] In some embodiments, feature upgrade is implemented for each inventory image feature in the feature library through the feature upgrade model trained in the previous embodiment, and the upgraded inventory upgrade feature overwrites the original corresponding inventory image feature. Then, from the multiple inventory upgrade features in the feature library, the feature similarity (for example, cosine similarity) between each inventory upgrade feature and the query image feature is determined. Then, the inventory upgrade feature with the highest feature similarity is selected as the target inventory feature with the highest matching degree with the query image feature.

[0239] 604. The server returns image retrieval information associated with the target inventory feature.

[0240] In some embodiments, the server returns image retrieval information associated with the target inventory feature to the terminal. For example, the image retrieval information includes but is not limited to: the target inventory feature itself, the object identifier associated with the target inventory feature, the index associated with the target inventory feature, the link associated with the target inventory feature, etc., which is not specifically limited in the embodiments of the present application.

[0241] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0242] The method provided in the embodiment of the present application utilizes a trained target feature model to extract query image features of a query image, and directly performs similar sample retrieval on the upgraded inventory upgrade features. This not only improves the performance of the query image features through the target feature model, but also greatly improves the retrieval performance of the image retrieval system because the inventory upgrade features are also better than the original inventory image features. This improves the retrieval accuracy of the retrieved target inventory features, and satisfies the need to improve the retrieval performance of the image retrieval system under privacy protection requirements.

[0243] Figure 7 is a schematic diagram of a structure of a feature model acquisition device provided in an embodiment of the present application, such as Figure 7 As shown, the device comprises:

[0244] A first acquisition module 701 is used to obtain a discriminant parameter based on the first image features extracted from the plurality of sample images based on the initial feature model, where the discriminant parameter represents the degree of difference of the first image features of the sample images under different categories;

[0245] A second acquisition module 702 is used to acquire a compatibility parameter based on the first feature model for extracting the second image feature and the first image feature from the plurality of sample images, wherein the compatibility parameter represents the degree of similarity between the first image feature and the second image feature of the same sample image;

[0246] The parameter adjustment module 703 is used to iteratively adjust the parameters of the initial feature model based on the discriminative parameter and the compatibility parameter to obtain a target feature model.

[0247] The device provided in the embodiment of the present application obtains a discriminant parameter for characterizing the degree of difference of the first image features of sample images under different categories, so that the discriminant parameter can control the trained target feature model to have a better feature extraction capability, and obtains a compatibility parameter for characterizing the degree of similarity between the first image features and the second image features of the same sample image, so that the compatibility parameter can control the target feature model to have a better compatibility with the original first feature model. Therefore, under the joint constraints of the discriminant parameter and the compatibility parameter, the target feature model obtained by training the initial feature model not only has a better feature extraction capability, but also has a strong compatibility capability.

[0248] Furthermore, after the target feature model is launched on the privacy-preserving image retrieval system, when a new query image is received, the target feature model optimizes the expression ability and improves the feature quality of the extracted query image features, and ensures that the query image features extracted by the target feature model can be directly calculated for similarity on the old feature library extracted by the original first feature model, thereby optimizing the retrieval performance of the entire image retrieval system by improving the feature quality of the query image without affecting the normal image retrieval function.

[0249] In a possible implementation manner, the second acquisition module 702 is used to:

[0250] For any sample image, obtaining a first similarity between a first image feature and a second image feature of the sample image;

[0251] The compatibility parameter is obtained based on the first similarities corresponding to each of the plurality of sample images.

[0252] In one possible implementation, based on Figure 7 The first acquisition module 701 includes:

[0253] A determination unit, configured to determine, for any sample image, based on a first image feature of the sample image, a predicted category into which the initial feature model classifies the sample image;

[0254] A first acquisition unit, configured to acquire a discriminative loss of the sample image based on the predicted category and a reference category to which the sample image belongs, wherein the discriminative loss is used to characterize a degree of difference between the predicted category and the reference category;

[0255] The second acquisition unit is used to acquire the discriminative parameter based on the discriminative losses corresponding to each of the multiple sample images.

[0256] In a possible implementation manner, the first acquisition unit is used to:

[0257] Acquire a feature angle between a first image feature of the sample image and a category feature of the reference category;

[0258] The feature angle is added to the classification additive angle margin to obtain the feature correction angle. The classification additive angle margin is used to increase the degree of inter-class difference between class features of different classes.

[0259] Based on the cosine value of the feature correction angle and the prediction values ​​of classifying the first image feature of the sample image into respective ones of a plurality of candidate categories, the discriminative loss of the sample image is obtained.

[0260] In a possible implementation manner, the parameter adjustment module 703 is used to:

[0261] Based on the discriminative parameter and the compatibility parameter, determining the loss function value of the initial feature model for this iteration;

[0262] When the number of iterations and the loss function value do not meet the first stop condition, iteratively adjust the parameters of the initial feature model, and perform the step of obtaining the loss function value again based on the initial feature model after adjusting the parameters;

[0263] When either the number of iterations or the loss function value meets the first stopping condition, the target feature model is output.

[0264] In one possible implementation, based on Figure 7 The device is composed of:

[0265] A first extraction module, used for inputting the plurality of sample images into the target feature model to obtain target image features of the plurality of sample images;

[0266] A second extraction module is used to input the second image features of the plurality of sample images into the initial upgrade model to obtain third image features of the plurality of sample images;

[0267] A third acquisition module, used to acquire a similarity parameter based on the third image feature and the target image feature, wherein the similarity parameter represents the degree of similarity between the third image feature and the target image feature of the same sample image;

[0268] The parameter adjustment module 703 is further used to iteratively adjust the parameters of the initial upgrade model based on the similarity parameter to obtain a feature upgrade model, and the feature upgrade model is used to optimize the image features extracted by the first feature model.

[0269] In a possible implementation manner, the third acquisition module is used to:

[0270] For any sample image, obtaining a second similarity between a third image feature of the sample image and a target image feature;

[0271] The similarity parameter is obtained based on the second similarities corresponding to each of the plurality of sample images.

[0272] In one possible implementation, based on Figure 7 The device is composed of:

[0273] A fourth acquisition module, used for acquiring a plurality of inventory image features in a feature library corresponding to the first feature model;

[0274] A feature upgrade module, used for inputting the plurality of inventory image features into the feature upgrade model to obtain a plurality of inventory upgrade features;

[0275] The feature replacement module is used to replace the multiple stock image features with the corresponding stock upgrade features in the feature library.

[0276] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0277] It should be noted that: the feature model acquisition device provided in the above embodiment only uses the division of the above functional modules as an example when acquiring the target feature model. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the feature model acquisition device provided in the above embodiment and the feature model acquisition method embodiment belong to the same concept. The specific implementation process is detailed in the feature model acquisition method embodiment, which will not be repeated here.

[0278] Figure 8 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application, such as Figure 8 As shown, the device comprises:

[0279] The acquisition module 801 is used to respond to an image query request and acquire a query image carried in the image query request;

[0280] An extraction module 802 is used to extract query image features of the query image based on a target feature model, wherein the target feature model is obtained by iteratively adjusting parameters of an initial feature model based on a discriminative parameter and a compatibility parameter;

[0281] A determination module 803 is used to determine, from among a plurality of inventory upgrade features in the feature library, a target inventory feature that has the highest matching degree with the query image feature, the inventory upgrade feature being obtained by processing the inventory image feature extracted by the first feature model by the feature upgrade model;

[0282] The returning module 804 is used to return the image retrieval information associated with the target inventory feature.

[0283] The device provided in the embodiment of the present application utilizes a trained target feature model to extract query image features of a query image, and directly performs similar sample retrieval on the upgraded inventory upgrade features. This not only improves the performance of the query image features through the target feature model, but also greatly improves the retrieval performance of the image retrieval system because the inventory upgrade features are also better than the original inventory image features. This improves the retrieval accuracy of the retrieved target inventory features, and satisfies the need to improve the retrieval performance of the image retrieval system under privacy protection requirements.

[0284] It should be noted that: the image processing device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when processing the query image. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the image processing device provided in the above embodiment and the image processing method embodiment belong to the same concept, and the specific implementation process is detailed in the image processing method embodiment, which will not be repeated here.

[0285] Fig. 9 It is a structural diagram of a computer device provided in an embodiment of the present application. The computer device 900 may have relatively large differences due to different configurations or performances. The computer device 900 includes one or more processors (Central Processing Units, CPU) 901 and one or more memories 902, wherein the memory 902 stores at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 901 to implement the feature model acquisition method or image processing method provided in the above-mentioned various embodiments. Optionally, the computer device 900 also has components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The computer device 900 also includes other components for realizing device functions, which will not be repeated here.

[0286] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including at least one computer program, and the at least one computer program can be executed by a processor in a terminal to complete the method for acquiring a feature model or the image processing method in each of the above embodiments. For example, the computer-readable storage medium includes a ROM (Read-Only Memory), a RAM (Random-Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, and an optical data storage device.

[0287] In an exemplary embodiment, a computer program product or a computer program is also provided, including one or more program codes, which are stored in a computer-readable storage medium. One or more processors of a computer device can read the one or more program codes from the computer-readable storage medium, and the one or more processors execute the one or more program codes, so that the computer device can execute to complete the method for acquiring a feature model or the image processing method in the above-mentioned embodiment.

[0288] A person of ordinary skill in the art will understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by instructing related hardware through a program. Optionally, the program is stored in a computer-readable storage medium. Optionally, the above-mentioned storage medium is a read-only memory, a disk or an optical disk, etc.

[0289] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for acquiring a feature model, characterized in that: The method comprises: Based on the initial feature model, first image features are extracted from the plurality of sample images to obtain discriminative parameters, wherein the discriminative parameters represent the degree of difference of the first image features of the sample images under different categories; Obtaining a compatibility parameter based on the first feature model for extracting a second image feature and the first image feature from the plurality of sample images, wherein the compatibility parameter represents a degree of similarity between the first image feature and the second image feature of the same sample image; Determining the loss function value of the initial feature model for this iteration based on the discriminative parameter and the compatibility parameter; When the number of iterations and the loss function value do not meet the first stop condition, iteratively adjust the parameters of the initial feature model, and perform the step of obtaining the loss function value again based on the initial feature model after adjusting the parameters; When either the number of iterations or the loss function value meets the first stop condition, the target feature model is output.

2. The method according to claim 1, characterized in that The obtaining of the compatibility parameter based on the first feature model extracting the second image feature and the first image feature from the plurality of sample images comprises: For any sample image, obtaining a first similarity between a first image feature and a second image feature of the sample image; The compatibility parameter is obtained based on the first similarities corresponding to each of the plurality of sample images.

3. The method according to claim 1, characterized in that The first image feature extracted from the plurality of sample images based on the initial feature model to obtain the discriminative parameter comprises: For any sample image, based on the first image feature of the sample image, determining a predicted category obtained by the initial feature model for classifying the sample image; Based on the predicted category and the reference category to which the sample image belongs, obtaining a discriminative loss of the sample image, wherein the discriminative loss is used to characterize a degree of difference between the predicted category and the reference category; The discriminative parameters are obtained based on the discriminative losses corresponding to the multiple sample images.

4. The method according to claim 3, characterized in that The obtaining the discriminative loss of the sample image based on the predicted category and the reference category to which the sample image belongs includes: Acquire a feature angle between a first image feature of the sample image and a category feature of the reference category; Adding the feature angle to the classification additive angle margin to obtain a feature correction angle, wherein the classification additive angle margin is used to increase the degree of inter-class difference between class features of different classes; The discriminative loss of the sample image is obtained based on the cosine value of the feature correction angle and the prediction values ​​of classifying the first image feature of the sample image into respective ones of a plurality of candidate categories.

5. The method according to claim 1, characterized in that The method further comprises: Inputting the plurality of sample images into the target feature model to obtain target image features of the plurality of sample images; Inputting the second image features of the plurality of sample images into the initial upgrading model to obtain third image features of the plurality of sample images; Based on the third image feature and the target image feature, obtaining a similarity parameter, wherein the similarity parameter represents a degree of similarity between the third image feature and the target image feature of the same sample image; Based on the similarity parameter, the parameters of the initial upgrade model are iteratively adjusted to obtain a feature upgrade model, where the feature upgrade model is used to optimize the image features extracted by the first feature model.

6. The method according to claim 5, characterized in that The acquiring the similarity parameter based on the third image feature and the target image feature comprises: For any sample image, obtaining a second similarity between a third image feature of the sample image and a target image feature; The similarity parameter is obtained based on the second similarities corresponding to each of the plurality of sample images.

7. The method according to claim 5 or 6, characterized in that: The method further comprises: Acquire a plurality of inventory image features in a feature library corresponding to the first feature model; Inputting the plurality of inventory image features into the feature upgrade model to obtain a plurality of inventory upgrade features; In the feature library, the plurality of stock image features are replaced with respective corresponding stock upgrade features.

8. An image processing method, characterized in that: The method comprises: In response to an image query request, obtaining a query image carried in the image query request; Extracting query image features of the query image based on a target feature model, wherein the target feature model is obtained by iteratively adjusting parameters of an initial feature model based on a discriminative parameter and a compatibility parameter; Determine, from a plurality of inventory upgrade features in the feature library, a target inventory feature that has the highest matching degree with the query image feature, wherein the inventory upgrade feature is obtained by processing the inventory image feature extracted by the first feature model by the feature upgrade model; Returning image retrieval information associated with the target inventory feature; Wherein, when the parameters of the initial feature model are iteratively adjusted based on the discriminative parameter and the compatibility parameter to obtain the target feature model, the following steps are included: Determining the loss function value of the initial feature model for this iteration based on the discriminative parameter and the compatibility parameter; When the number of iterations and the loss function value do not meet the first stop condition, iteratively adjust the parameters of the initial feature model, and perform the step of obtaining the loss function value again based on the initial feature model after adjusting the parameters; When either the number of iterations or the loss function value meets the first stop condition, the target feature model is output.

9. A device for acquiring a feature model, characterized in that: The device comprises: A first acquisition module, configured to obtain a discriminant parameter based on the first image features extracted from the plurality of sample images based on the initial feature model, wherein the discriminant parameter represents the degree of difference between the first image features of the sample images of different categories; A second acquisition module, configured to acquire a compatibility parameter based on the first feature model for extracting second image features and the first image features from the plurality of sample images, wherein the compatibility parameter represents a degree of similarity between the first image feature and the second image feature of the same sample image; A parameter adjustment module is used to determine the loss function value of the current iteration of the initial feature model based on the discriminative parameter and the compatibility parameter; when the number of iterations and the loss function value do not meet the first stopping condition, iteratively adjust the parameters of the initial feature model, and based on the initial feature model after the adjustment of the parameters, execute the step of obtaining the loss function value again; when either the number of iterations or the loss function value meets the first stopping condition, output the target feature model.

10. The device according to claim 9, characterized in that The second acquisition module is used for: For any sample image, obtaining a first similarity between a first image feature and a second image feature of the sample image; The compatibility parameter is obtained based on the first similarities corresponding to each of the plurality of sample images.

11. The device according to claim 9, characterized in that The first acquisition module includes: a determination unit, configured to determine, for any sample image, a predicted category obtained by the initial feature model by classifying the sample image based on a first image feature of the sample image; A first acquisition unit, configured to acquire a discriminative loss of the sample image based on the predicted category and a reference category to which the sample image belongs, wherein the discriminative loss is used to characterize a degree of difference between the predicted category and the reference category; The second acquisition unit is used to acquire the discriminative parameter based on the discriminative losses corresponding to each of the plurality of sample images.

12. The device according to claim 11, characterized in that The first acquisition unit is used for: Acquire a feature angle between a first image feature of the sample image and a category feature of the reference category; Adding the feature angle to the classification additive angle margin to obtain a feature correction angle, wherein the classification additive angle margin is used to increase the degree of inter-class difference between class features of different classes; The discriminative loss of the sample image is obtained based on the cosine value of the feature correction angle and the prediction values ​​of classifying the first image feature of the sample image into respective ones of a plurality of candidate categories.

13. The device according to claim 9, characterized in that The device also includes: A first extraction module, used for inputting the plurality of sample images into the target feature model to obtain target image features of the plurality of sample images; A second extraction module, configured to input the second image features of the plurality of sample images into an initial upgrade model to obtain third image features of the plurality of sample images; A third acquisition module, configured to acquire a similarity parameter based on the third image feature and the target image feature, wherein the similarity parameter represents a degree of similarity between the third image feature and the target image feature of the same sample image; The parameter adjustment module is further used to iteratively adjust the parameters of the initial upgrade model based on the similarity parameter to obtain a feature upgrade model, and the feature upgrade model is used to optimize the image features extracted by the first feature model.

14. The device according to claim 13, characterized in that The third acquisition module is used for: For any sample image, obtaining a second similarity between a third image feature of the sample image and a target image feature; The similarity parameter is obtained based on the second similarities corresponding to each of the plurality of sample images.

15. The device according to claim 13 or 14, characterized in that The device also includes: a fourth acquisition module, configured to acquire a plurality of inventory image features in a feature library corresponding to the first feature model; A feature upgrade module, used for inputting the plurality of inventory image features into the feature upgrade model to obtain a plurality of inventory upgrade features; The feature replacement module is used to replace the plurality of inventory image features with the corresponding inventory upgrade features in the feature library.

16. An image processing device, characterized in that: The device comprises: An acquisition module, configured to, in response to an image query request, acquire a query image carried in the image query request; An extraction module, configured to extract query image features of the query image based on a target feature model, wherein the target feature model is obtained by iteratively adjusting parameters of an initial feature model based on a discriminative parameter and a compatibility parameter; a determination module, configured to determine, from a plurality of inventory upgrade features in a feature library, a target inventory feature that has the highest matching degree with the query image feature, wherein the inventory upgrade feature is obtained by processing the inventory image feature extracted by the first feature model by the feature upgrade model; A return module, used for returning image retrieval information associated with the target inventory feature; Wherein, when the parameters of the initial feature model are iteratively adjusted based on the discriminative parameter and the compatibility parameter to obtain the target feature model, the following steps are included: Determining the loss function value of the initial feature model for this iteration based on the discriminative parameter and the compatibility parameter; When the number of iterations and the loss function value do not meet the first stop condition, iteratively adjust the parameters of the initial feature model, and perform the step of obtaining the loss function value again based on the initial feature model after adjusting the parameters; When either the number of iterations or the loss function value meets the first stop condition, the target feature model is output.

17. A computer device, characterized in that: The computer device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement the feature model acquisition method as described in any one of claims 1 to 7 or the image processing method as described in claim 8.

18. A storage medium, characterized in that: The storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the method for acquiring a feature model as described in any one of claims 1 to 7 or the image processing method as described in claim 8.

19. A computer program product, characterized in that The computer program product includes at least one computer program, and the at least one computer program is loaded and executed by a processor to implement the method for acquiring a feature model as described in any one of claims 1 to 7 or the image processing method as described in claim 8.

Images