Deep metric learning image retrieval system optimization method, system, device, and medium

Abstract

The application belongs to the field of deep learning models, and discloses a deep metric learning image retrieval system optimization method, system, device and medium, which comprises the following steps: obtaining a picture triple batch used for current training, judging the disturbance mode used by the picture batch according to a preset condition, generating an anchor disturbance and a candidate disturbance corresponding to the current batch of triplets according to the preset disturbance mode, the former comprising a variable direction attack and a collapse correlation adaptive distance, and the latter comprising a distance proportion attack loss function and a ranking relationship regularization term, then adjusting the intensity of the anchor disturbance according to the existing round number, forming an adversarial triplet, and training the deep metric learning model using the loss function corresponding to the disturbance mode, which can solve the model collapse problem and greatly increase the benign sample recall rate of the model, and can also solve the problem of insufficient attack intensity and improve the robustness of the model.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of deep learning models, and relates to a deep metric learning image retrieval system optimization method, system, device and medium. BACKGROUND

[0002] The image retrieval task refers to giving a query picture, querying all pictures similar to the query picture in a database, and sorting them according to their similarity, and finally returning one or more samples in the front. With the current demand for retrieval tasks having developed from traditional text detection to multi-modal and even cross-modal retrieval, image retrieval has attracted increasing attention. Deep metric learning, as one of the main implementation methods of image retrieval systems, is also widely used. The principle is to use the learning ability of a deep neural network to enable the model to learn a reasonable embedding space. In this space, similar pictures are closer in distance, and different pictures are farther apart. However, with the discovery of adversarial attacks, the vulnerability and robustness of deep metric learning have gradually emerged. Adversarial attacks usually refer to attackers adding tiny perturbations that are difficult for the human eye to detect to benign samples, so that the deep neural network makes a huge error in judging the sample.

[0003] For image retrieval systems based on deep metric learning, adversarial attacks mainly disturb the distribution position of the query picture in the embedding space learned by the model, so that it is farther away from the main distribution of similar samples and closer to the main distribution of different samples. Unlike adversarial attacks for image classification, according to the attack method, it can be divided into query-based attacks, inference-based attacks, etc., and black-box attacks are the most common, which is caused by the systematic security problem of the retrieval task itself. For retrieval tasks that return multiple results, the query result inevitably leaks part of the training data set each time, which makes query-based attacks can obtain a subset of the victim model training set by a certain number of queries, and then use it to construct an adversarial attack; even for retrieval tasks that return a single result, similar to black-box attacks for image classification, the image retrieval system can be attacked. In addition, unlike the measurement standard of image classification adversarial attack effect, the adversarial attack effect for image retrieval is mainly judged by recall rate, that is, the probability that the correct sample is retrieved and returned to the retrieval list given the query sample. Image retrieval adversarial attack is to add a tiny perturbation to the query sample using the characteristics of adversarial attack, thereby greatly reducing the recall rate of the system for the query sample.

[0004] The existing deep metric learning image retrieval system does not have resistance to the above-mentioned adversarial attack, and the recall rate thereof will be greatly reduced after the attack, which will cause hidden troubles after the deep metric learning image retrieval is widely applied in the industry. The effective way to resist the adversarial attack at present is to add the adversarial sample in the training process of the neural network, so that the adversarial robustness of the model is improved. However, there is no effective and feasible adversarial training method to improve the image detection system, and the existing method has the following limitations: first, the adversarial training causes the model to collapse, so that the model cannot normally perform image retrieval. This is due to the legacy problem of deep metric learning itself, which can only avoid the model collapse problem by avoiding the use of too difficult triplets, but this problem cannot be avoided in adversarial training. Second, the adversarial training is insufficient, so that the adversarial robustness of the deep metric model obtained after the adversarial training is limited even if a large number of recall rates for benign samples are lost. This is because the adversarial training of deep metric learning needs a more appropriate training method, and due to the limitation of model collapse, the strength of adversarial training should not be too large. In addition, the adversarial training of the deep metric learning image retrieval system also has problems such as long training time and large operation cost. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art, such as the model collapse during training of the deep metric learning image retrieval system, insufficient adversarial training strength, and high training cost, and to provide a deep metric learning image retrieval system optimization method, system, device and medium.

[0006] To achieve the above purpose, the following technical solutions are adopted:

[0007] In a first aspect, the present application provides a deep metric learning image retrieval system optimization method, comprising:

[0008] S1: obtaining triplets and a perturbation amplitude of the current training round of the image retrieval system, and obtaining a perturbation mode of the current training round according to the number of the current training round and the perturbation mode of the last training round;

[0009] S2: when the perturbation mode of the current training round is candidate perturbation, iteratively perturbing the triplets based on the perturbation amplitude until a first preset iteration number or a preset function value of the candidate perturbation objective function meets a preset requirement, to obtain adversarial triplets; when the perturbation mode of the current training round is anchor point perturbation, iteratively perturbing the triplets based on the perturbation amplitude until a second preset iteration number or a preset function value of the anchor point perturbation objective function meets a preset requirement, to obtain adversarial triplets;

[0010] S3: training the deep metric learning model of the image retrieval system through the adversarial triplets; when the number of the current training round is the preset maximum number of rounds, obtaining the optimized image retrieval system; otherwise, repeating S1-S3 with the next training round as the current training round.

[0011] Optionally, the obtaining the triplets of the current training round of the image retrieval system comprises:

[0012] obtaining a plurality of training samples of the image retrieval system, and randomly selecting one training sample as an anchor sample;

[0013] repeating the selecting step until the number of correct samples and the number of incorrect samples both reach a preset value:

[0014] the selecting step: randomly selecting one training sample of the same category as the anchor sample as a correct sample, and selecting one training sample from the plurality of training samples as an incorrect sample based on the correct sample;

[0015] wherein the correct sample and the incorrect sample satisfy: the Euclidean distance between the anchor sample and the incorrect sample is greater than the Euclidean distance between the anchor sample and the correct sample, and the Euclidean distance between the anchor sample and the incorrect sample and the Euclidean distance between the anchor sample and the correct sample differ by no more than wherein m is the number of the current training round, and M is the preset maximum number of rounds;

[0016] combining the anchor sample and all correct samples and incorrect samples to obtain the triplets of the current training round.

[0017] Optionally, the obtaining the perturbation amplitude of the current training round of the image retrieval system comprises:

[0018] obtaining a perturbation amplitude increasing step of the image retrieval system;

[0019] multiplying the number of the current training round of the image retrieval system by the perturbation amplitude increasing step to obtain the perturbation amplitude of the current training round of the image retrieval system.

[0020] Optionally, the obtaining the perturbation mode of the current training round according to the number of the current training round and the perturbation mode of the previous training round comprises:

[0021] when , or and the perturbation mode of the previous training round is anchor perturbation, the perturbation mode of the current training round is candidate perturbation; otherwise, the perturbation mode of the current training round is anchor perturbation.

[0022] Optionally, the candidate perturbation objective function l CAP is: l CAP = φ(H) + γ(H).

[0023] wherein H is the difficulty of the triplet, H = d(A, P) - d(A, N), A is the anchor sample of the triplet, P is the correct sample set of the triplet, N is the error sample set of the triplet, d(A, P) is the sum of the Euclidean distances between A and each correct sample in P, d(A, N) is the sum of the Euclidean distances between A and each error sample in N;

[0024] when H > 0, φ(H) = d(A, P) - d(A, N); when H < 0, φ(H) = d(A, N) - d(A, P);

[0025] when H > ξ, φ(H) = φ(H) - δ * B; when H < -ξ, φ(H) = φ(H) + δ * B; when -ξ < H < ξ, γ(H) = γ t-1 ; wherein ξ is a preset model collapse threshold, γ t-1 is the value of γ(H) at the last iteration disturbance, δ is a preset distance change step, and B is the number sum of correct samples and error samples in the triplet.

[0026] Optionally, the anchor point disturbance target function l CAP is: l CAP = l dw + Δ TR ;

[0027] wherein, ReLU is a rectified linear function; A is the anchor sample of the triplet, P is the correct sample set of the triplet, and N is the error sample set of the triplet; n j is the jth error sample in N, d(A, n j ) is the Euclidean distance between A and n j , p i is the ith correct sample in P, d(A, p i ) is the Euclidean distance between A and p i , d max (A, N) is the maximum value of the Euclidean distances between A and each error sample in N, d min (A, N) is the minimum value of the Euclidean distances between A and each error sample in N; d max (A, P) is the maximum value of the Euclidean distances between A and each correct sample in P, d min (A, P) is the minimum value of the Euclidean distances between A and each correct sample in P;

[0028] N s is an error sample subset, and is obtained by selecting all error samples with a Euclidean distance less than δ from A in N.​​ is obtained by combining all the error samples in N with A, and d(A, N) is the sum of the Euclidean distances between A and each error sample in N. is the average of the Euclidean distances between A and each error sample in N, and A' is the anchor sample after the perturbation.

[0029] Optionally, when training the deep metric learning model of the image retrieval system by using the adversarial triplets, the following loss function l is used:

[0030] When the perturbation manner of the adversarial triplets is the candidate perturbation, the following loss function l is used: triplet Training the deep metric learning model of the image retrieval system: wherein A is an anchor sample of the triplets, is a set of perturbed correct samples of the adversarial triplets, is a set of perturbed error samples of the adversarial triplets; is the sum of the Euclidean distances between A and each perturbed correct sample in P, is the average of the Euclidean distances between A and each perturbed correct sample in P, is the sum of the Euclidean distances between A and each perturbed error sample in N, is the average of the Euclidean distances between A and each perturbed error sample in N.

[0031] When the perturbation manner of the adversarial triplets is the anchor perturbation, the following loss function l is used: triplet Training the deep metric learning model of the image retrieval system: wherein P is a set of correct samples of the triplets, N is a set of error samples of the triplets, N s is a subset of error samples, which is obtained by selecting all the error samples with a Euclidean distance less than d(A, N) from N, is obtained by combining all the error samples in N with A, and d(A, N) is the sum of the Euclidean distances between A and each error sample in N. is the average of the Euclidean distances between A and each error sample in N, P s is a subset of correct samples, which is obtained by selecting all the correct samples with a Euclidean distance less than d(A, P) from P, is obtained by combining all the correct samples in P with A, and d(A, P) is the sum of the Euclidean distances between A and each correct sample in P. is the average of the Euclidean distances between A and each correct sample in P, is the anchor sample after the perturbation in the adversarial triplets; is the sum of the Euclidean distances between A and each correct sample in P, is the sum of the Euclidean distances between A and each correct sample in P, is the sum of the Euclidean distances between A and each error sample in N, is the sum of the Euclidean distances between A and each error sample in N, d(A, P s ) is the sum of the Euclidean distances between A and each correct sample in P, S is the sum of the Euclidean distances between A and each error sample in N. s S

[0032] ​​In a second aspect, the present application provides a deep metric learning image retrieval system optimization system, comprising:

[0033] a disturbance information determination module configured to obtain a triplet and a disturbance amplitude of a current training round of the image retrieval system, and obtain a disturbance mode of the current training round according to a round number of the current training round and a disturbance mode of a previous training round;

[0034] a disturbance generation module configured to, when the disturbance mode of the current training round is a candidate disturbance, iteratively generate an adversarial triplet based on the disturbance amplitude until a first preset iteration number is reached or a function value of a preset candidate disturbance objective function meets a preset requirement, to obtain the adversarial triplet; and when the disturbance mode of the current training round is an anchor point disturbance, iteratively generate an adversarial triplet based on the disturbance amplitude until a second preset iteration number is reached or a function value of a preset anchor point disturbance objective function meets a preset requirement, to obtain the adversarial triplet;

[0035] a model training module configured to train a deep metric learning model of the image retrieval system by using the adversarial triplet; when the round number of the current training round is a preset maximum round number, obtain an optimized image retrieval system; otherwise, take a next training round as the current training round, and sequentially trigger the disturbance information determination module, the disturbance generation module and the model training module.

[0036] In a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the deep metric learning image retrieval system optimization method described above when executing the computer program.

[0037] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement the steps of the deep metric learning image retrieval system optimization method described above.

[0038] Compared with the prior art, the present application has the following beneficial effects:

[0039] The deep metric learning image retrieval system optimization method of the present application actively adjusts the intensity and direction of adversarial training based on a candidate disturbance objective function and an anchor point disturbance objective function, realizes adaptive adversarial training based on collapse degree, and gradually reduces the learning difficulty of the triplet, which effectively solves the model collapse problem and makes the deep metric learning model not collapse in the whole training process. Meanwhile, by alternately interfering, the adversarial training is split into candidate disturbance and anchor point disturbance based on the triplet, which greatly increases the embedding displacement generated by each disturbance, thereby generating stronger adversarial disturbance and increasing the intensity of adversarial training. Finally, due to the solution of the above limitations, the time and computational cost of the overall adversarial training are also reduced. Attached Figure Description

[0040] Figure 1 This is a flowchart of the optimization method for a depth metric learning image retrieval system according to an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram illustrating the principle of the depth metric learning image retrieval system optimization method according to an embodiment of the present invention.

[0042] Figure 3 This is a block diagram of the optimized system structure of the depth metric learning image retrieval system according to an embodiment of the present invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0044] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0045] The present invention will now be described in further detail with reference to the accompanying drawings:

[0046] See Figure 1 and 2 In one embodiment of the present invention, an optimization method for a deep metric learning image retrieval system is provided, specifically a training method for a deep metric learning image retrieval system with adversarial robustness. This method mainly addresses the two major limitations of deep metric adversarial training by using collapse-based adaptive adversarial training and alternating interference, thereby obtaining a deep metric learning model with adversarial robustness.

[0047] Specifically, the optimization method for this deep metric learning image retrieval system includes the following steps:

[0048] S1: Obtain a triplet and a perturbation amplitude of a current training round of an image retrieval system, and obtain a perturbation mode of the current training round according to a round number of the current training round and a perturbation mode of a previous training round.

[0049] S2: When the perturbation mode of the current training round is candidate perturbation, iteratively perturb the triplet based on the perturbation amplitude until a first preset iteration number is reached or a function value of a preset candidate perturbation objective function meets a preset requirement, to obtain an adversarial triplet; when the perturbation mode of the current training round is anchor point perturbation, iteratively perturb the triplet based on the perturbation amplitude until a second preset iteration number is reached or a function value of a preset anchor point perturbation objective function meets a preset requirement, to obtain an adversarial triplet.

[0050] S3: Train a deep metric learning model of the image retrieval system through the adversarial triplet; when the round number of the current training round is a preset maximum round number, obtain an optimized image retrieval system; otherwise, repeat S1-S3 with a next training round as the current training round.

[0051] The triplet includes an anchor point sample, a correct sample, and an error sample.

[0052] The selected perturbation objective function adjusts the optimization direction and strength of the adversarial perturbation after generating the adversarial perturbation according to the collapse of the deep metric learning model: if the deep metric learning model has not reached a collapse threshold, the adversarial perturbation pushes the correct sample away from the anchor point sample and pulls the error sample towards the anchor point sample, and a constant parameter of the adversarial training loss function is adjusted, thereby increasing the difficulty of the triplet and reducing the probability of retrieving the correct sample. If the deep metric learning model has reached the collapse threshold, the adversarial perturbation pulls the correct sample towards the anchor point sample and pushes the error sample away from the anchor point sample, and the constant parameter of the adversarial training loss function is adjusted, thereby reducing the difficulty of the triplet and increasing the probability of retrieving the correct sample.

[0053] Optionally, the candidate perturbation objective function is defined as the sum of a variable direction attack function and a collapse associated adaptive distance function, which are respectively responsible for adjusting the direction of the candidate perturbation and the strength of the candidate perturbation. The variable direction attack function adjusts the optimization direction of the current adversarial attack through the difficulty of the current triplet, and the collapse associated adaptive distance function adjusts the strength of the current adversarial attack through the current collapse degree of the deep metric learning model.

[0054] The anchor point disturbance target function gives corresponding proportions to all correct samples and error samples according to the distances between all correct samples and error samples and the anchor point sample. The closer a sample is to the anchor point sample, the closer the proportion of the sample is to 1. The farther the sample is from the anchor point sample, the closer the proportion of the sample is to 0. After obtaining the proportions of all correct samples and error samples, the proportion distance between the correct sample, the error sample and the anchor point sample is calculated, and then the corresponding anchor point sample is obtained according to the preset adversarial training loss function. The adversarial disturbance specifically includes two stages: in the first stage, the anchor point sample is simultaneously pushed away from the correct sample and pushed towards the error sample according to the proportions of all correct samples and error samples, until the proportion distance between the anchor point sample and the correct sample and the error sample is equal; in the second stage, the anchor point sample is further pushed towards all error samples preselected to have a distance less than an average value from the anchor point sample, and the displacement of the anchor point sample after the adversarial disturbance is maximized.

[0055] To sum up, the deep metric learning image retrieval system optimization method actively adjusts the intensity and direction of adversarial training based on the candidate disturbance target function and the anchor point disturbance target function, realizes adaptive adversarial training based on the collapse degree, gradually reduces the learning difficulty of the triplets, effectively solves the model collapse problem, and makes the deep metric learning model not collapse in the whole training process. Meanwhile, by alternately interfering, the adversarial training is split into candidate disturbance and anchor point disturbance in the way of triplets, which greatly increases the embedding displacement generated by each disturbance, thereby generating stronger adversarial disturbance and increasing the intensity of adversarial training. Finally, due to the solution of the above limitations, the time and computational cost of the overall adversarial training are also reduced.

[0056] In a possible implementation, the acquiring the triplets of the current training round of the image retrieval system includes: acquiring a plurality of training samples of the image retrieval system, and randomly selecting one training sample as an anchor point sample; repeating the selecting step until the number of correct samples and error samples reaches a preset value: the selecting step: randomly selecting one training sample of the same category as the anchor point sample as a correct sample, and selecting one training sample from the plurality of training samples as an error sample based on the correct sample; wherein the correct sample and the error sample satisfy: the Euclidean distance between the anchor point sample and the error sample is greater than the Euclidean distance between the anchor point sample and the correct sample, and the Euclidean distance between the anchor point sample and the error sample and the Euclidean distance between the anchor point sample and the correct sample differ by no more than wherein m is the number of the current training round, and M is a preset maximum number of rounds; the anchor point sample and all correct samples and error samples are combined to obtain the triplets of the current training round.

[0057] Specifically, according to the current training round m and the total training round M, and given the total sample quantity B in the mini-batch, the sampling of the triplets is called round progressive difficulty sampling, which specifically includes the following steps: a sample is randomly selected as the anchor sample A of the triplet; a correct sample P belonging to the same category as A is randomly selected; an error sample N belonging to a different category from A is randomly selected, which satisfies condition 1 and condition 2: condition 1: the Euclidean distance d(A, N) between N and A is greater than the Euclidean distance d(A, P) between P and A, that is, the difficulty H of the triplet is less than 0, where H = d(A, P) - d(A, N); condition 2: the difference between the Euclidean distance between N and A and the distance between P and A does not exceed If no sample satisfies condition 2, an N satisfying condition 1 is randomly selected. P and N are repeatedly selected until the number of the mini-batch reaches B, where the number of P and N is

[0058] In a possible implementation, the obtaining the perturbation amplitude of the current training round of the image retrieval system includes: obtaining a perturbation amplitude increasing step of the image retrieval system; and taking the product of the number of the current training round of the image retrieval system and the perturbation amplitude increasing step as the perturbation amplitude of the current training round of the image retrieval system.

[0059] Specifically, according to the current training round m and the total training round M, the strength of the adversarial perturbation of the candidate perturbation or the anchor point perturbation is gradually increased from 0, and the specific manner is that before each round, the perturbation amplitude of the adversarial triplet is adjusted.

[0060] In a possible implementation, the obtaining the perturbation mode of the current training round according to the number of the current training round and the perturbation mode of the last training round includes: when , or and the perturbation mode of the last training round is the anchor point perturbation, the perturbation mode of the current training round is the candidate perturbation; otherwise, the perturbation mode of the current training round is the anchor point perturbation.

[0061] Specifically, the candidate perturbation and the anchor point perturbation are implemented in the form of rounds, and can also be implemented in other forms of rotation, mainly to realize the splitting of the candidate perturbation and the anchor point perturbation.

[0062] In a possible implementation, the candidate perturbation target function l CAP is: l CAP= φ(H) + γ(H); wherein, H is the difficulty of the triplet, H = d(A, P) - d(A, N), A is the anchor sample of the triplet, P is the correct sample set of the triplet, N is the error sample set of the triplet, d(A, P) is the sum of the Euclidean distances between A and each correct sample in P, d(A, N) is the sum of the Euclidean distances between A and each error sample in N; when H > 0, φ(H) = d(A, P) - d(A, N); when H < 0, φ(H) = d(A, N) - d(A, P); when H > ξ, when H < -ξ, when -ξ < H < ξ, γ(H) = γ t-1 ; wherein, ξ is a preset model collapse threshold, γ t-1 is the value of γ(H) at the last iteration disturbance, δ is a preset distance change step, and B is the sum of the number of correct samples and error samples in the triplet.

[0063] Specifically, the target function of the candidate disturbance is defined as l CAP = φ(H) + γ(H), wherein φ(H) is a variable direction attack function, and γ(H) is a collapse correlation adaptive distance function, which are respectively responsible for adjusting the direction of the candidate disturbance and the strength of the candidate disturbance. The variable direction attack function φ(H) adjusts the optimization direction of the current adversarial attack through the difficulty H of the current triplet, and the algorithm process is as follows: when H < 0, the embedding space of the deep metric model can correctly distinguish P and N, at this time, the adversarial attack should increase the difficulty H of the current triplet, that is, φ(H) = d(A, N) - d(A, P); when H > 0, the embedding space of the deep metric model cannot correctly distinguish P and N, at this time, the model is in a model collapse state, and the adversarial attack should reduce the difficulty H of the current triplet, that is, φ(H) = d(A, P) - d(A, N). The collapse correlation adaptive distance function γ(H) automatically adjusts the strength of the current adversarial attack through the current model collapse degree. Specifically, the greater γ(H) is, the greater the adversarial attack strength is, and vice versa, and the algorithm process is as follows: given the model collapse threshold ξ, the value of this round γ(H) is γ t , the value of the last round γ(H) is γ t-1 , the distance change step δ, when H > ξ, γ(H) judges that the model is in an uncollapsed state, at this time, the model can continue to withstand the adversarial training, so γ t can be increased, that is, when H < -ξ, γ(H) judges that the model is in a collapsed state, at this time, the model shows signs of collapse due to too large adversarial training difficulty, so γ t needs to be reduced, that is,

[0064] In one possible implementation, the anchor point disturbance target function l CAP is: l CAP = ldw + Δ TR .

[0065] where, ReLU is the rectified linear function; A is the anchor sample of the triplet, P is the correct sample set of the triplet, N is the error sample set of the triplet; n j is the jth error sample in N, d(A, n j ) is the Euclidean distance between A and n j , p i is the ith correct sample in P, d(A, p i ) is the Euclidean distance between A and p i , d max (A, N) is the maximum value of the Euclidean distance between A and each error sample in N, d min (A, N) is the minimum value of the Euclidean distance between A and each error sample in N; d max (A, P) is the maximum value of the Euclidean distance between A and each correct sample in P, d min (A, P) is the minimum value of the Euclidean distance between A and each correct sample in P; N s is the error sample subset, which is obtained by selecting all error samples from N whose Euclidean distance with A is less than , is the average value of the Euclidean distance between A and each error sample in N, A' is the anchor sample after perturbation, d(A, A') is the Euclidean distance between A and A'.

[0066] Specifically, the anchor perturbation objective function is defined as l CAP = l dw + Δ TR , wherein l dw is the distance proportion adversarial attack loss function, Δ TR is the ranking relationship regularization term, which is responsible for guiding the adversarial attack to push the anchor away from P as far as possible while pushing it towards N as far as possible.

[0067] The distance proportion adversarial attack loss function l dw is responsible for the first phase of anchor perturbation, that is, to quickly push the anchor away from the correct sample P. l dw Based on the triplet difficulty H, the specific calculation process is as follows: after calculating the distance between the correct sample P, the error sample N and the anchor sample A, the corresponding sample proportion is assigned according to the sample-anchor distance d, the larger the distance d, the smaller the sample proportion ω, and the smaller the distance d, the larger the sample proportion ω. For any correct sample p i , its proportion​​ For For any error sample n j , the proportion of For Finally, the distance proportion adversarial attack loss function is obtained by calculating the weighted average, that is, where ReLU is a rectified linear unit.

[0068] The ranking relationship regular term Δ TR is responsible for the second phase of anchor point disturbance, that is, further pulling the anchor point to the error sample N. Δ TR contains a self-adjusting coefficient λ and the ranking relationship regular itself, and the specific operation process is as follows: from the error sample N, all error samples with a distance d less than the average value between the anchor sample A and the error sample are selected as the error sample subset N S , that is, all error samples more similar to the anchor sample A are selected as the subset, and the average distance between all error samples in N S and A is calculated again, that is, Subsequently, to maximize the intensity of anchor point disturbance, the distance between the anchor point before and after disturbance is added as a regular term to ensure that the embedding displacement generated by anchor point disturbance is maximum. Finally, according to the value of the distance proportion adversarial attack loss function l dw , is calculated Finally, the current regular term Δ is obtained TR is small when l dw is dominant, at which time the anchor point disturbance mainly pushes the anchor sample A away from the nearby correct sample P; when the anchor sample A is far away from the correct sample P, l dw is close to 0, and Δ TR begins to dominate, at which time the anchor point disturbance mainly pulls the anchor sample as far as possible to the error sample subset N s closest to the anchor sample A.

[0069] In a possible implementation, when training the deep metric learning model of the image retrieval system by using the adversarial triplets, the following loss function l triplet is used to train the deep metric learning model of the image retrieval system when the disturbance mode of the adversarial triplets is candidate disturbance: where A is an anchor sample of the triplet, is a disturbed correct sample set of the adversarial triplet, is a disturbed error sample set of the adversarial triplet; is the sum of the Euclidean distances between A and each disturbed correct sample in , and is the sum of the Euclidean distances between A and each disturbed error sample in the sum of the Euclidean distances between each perturbation error sample.

[0070] When the perturbation mode of the adversarial triplet is anchor point perturbation, the following loss function l is used triplet Training a deep metric learning model of an image retrieval system: Wherein, P is the correct sample set of the triplet, N is the error sample set of the triplet, N s is the error sample subset, which is obtained by selecting all error samples from N with the Euclidean distance between A and the error sample less than , is the average of the Euclidean distances between A and each error sample in N, P s is the correct sample subset, which is obtained by selecting all correct samples from P with the Euclidean distance between A and the correct sample less than , is the average of the Euclidean distances between A and each correct sample in P, is the anchor point sample after perturbation in the adversarial triplet; is the sum of the Euclidean distances between A and each correct sample in P, is the sum of the Euclidean distances between A and each error sample in N, d(A, P s ) is the sum of the Euclidean distances between A and each correct sample in P s , d(A, N s ) is the sum of the Euclidean distances between A and each error sample in N s .

[0071] Specifically, the adjustment of the adversarial perturbation intensity is realized in a gradual difficulty adjustment mode, and the amplitude of the adversarial interference produced by the adjustment of the candidate interference and the anchor point interference is adjusted according to the training process, which helps the deep metric model to balance the learning between the adversarial samples and the benign samples in the adversarial training process, so that the model can obtain adversarial robustness while maintaining the recall rate for benign samples. The specific process is as follows:

[0072] If the adversarial attack η is generated by the candidate interference, given the current round number m and the total training round number M, the correct sample P and the error sample N are disturbed at this time, and the corresponding adversarial triplet is Then, the deep metric learning loss function corresponding to the triplet is used as the triplet loss to train the deep metric model. Finally, if m

[0073] If the adversarial attack η is generated by the anchor point interference, given the current round number m and the total training round number M, only the anchor point sample A is disturbed at this time, and the corresponding adversarial triplet is Subsequently, the corresponding deep metric learning loss function of the triplet is used as the triplet loss The deep metric model is trained, wherein P s , N s are the correct sample subset and the error sample subset screened in the manner described in S3. Finally, if m < M-1 at this time, S1 is performed, otherwise the training is ended.

[0074] Through experimental demonstration, the training method of the deep metric learning image retrieval system with adversarial robustness can obtain better robustness even if a relatively simple neural network ResNet18 is used, and the recall rate can still maintain a relatively high level, indicating that the deep metric learning image retrieval system optimization method can effectively improve the adversarial robustness while avoiding model collapse, and maintain a certain benign sample recall rate.

[0075] In summary, the deep metric learning image retrieval system optimization method obtains a triplet batch of pictures used for current training, judges the perturbation mode used by the current triplet batch according to a preset condition, generates an adversarial perturbation corresponding to the current triplet batch according to the preset perturbation mode, that is, a candidate perturbation and an anchor point perturbation, the former includes a variable direction adversarial attack and a collapse correlation adaptive distance, and the latter includes a distance proportion adversarial attack loss function and a ranking relationship regularization term, then adjusts the strength of the adversarial perturbation according to the existing round number to form an adversarial triplet, uses the loss function corresponding to the perturbation mode to train the deep metric learning model, which on the one hand solves the problem of deep metric learning model collapse, greatly increases the benign sample recall rate of the deep metric learning model, and on the other hand solves the problem of insufficient adversarial attack strength, so that the robustness of the deep metric learning model is improved.

[0076] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the apparatus embodiment, please refer to the method embodiment of the present application.

[0077] Referring to Figure 3 In another embodiment of the present application, a deep metric learning image retrieval system optimization system is provided, which can be used to implement the deep metric learning image retrieval system optimization method described above. Specifically, the deep metric learning image retrieval system optimization system includes a perturbation information determination module, a perturbation generation module, and a model training module.

[0078] The disturbance information determination module is configured to obtain a triple of a current training round of the image retrieval system and a disturbance amplitude, and obtain a disturbance mode of the current training round according to a round number of the current training round and a disturbance mode of a previous training round; the disturbance generation module is configured to, when the disturbance mode of the current training round is candidate disturbance, iteratively disturb the triple based on the disturbance amplitude until a first preset iteration number is reached or a function value of a preset candidate disturbance objective function meets a preset requirement, to obtain an adversarial triple; when the disturbance mode of the current training round is anchor point disturbance, iteratively disturb the triple based on the disturbance amplitude until a second preset iteration number is reached or a function value of a preset anchor point disturbance objective function meets a preset requirement, to obtain the adversarial triple; the model training module is configured to train the deep metric learning model of the image retrieval system by using the adversarial triple; when the round number of the current training round is a preset maximum round number, an optimized image retrieval system is obtained; otherwise, a next training round is taken as the current training round, and the disturbance information determination module, the disturbance generation module and the model training module are triggered in sequence.

[0079] In a possible implementation, the obtaining the triple of the current training round of the image retrieval system comprises: obtaining a plurality of training samples of the image retrieval system, and randomly selecting one training sample as an anchor point sample; repeating a selection step until the number of correct samples and the number of incorrect samples both reach preset values; the selection step comprises: randomly selecting one training sample of the same category as the anchor point sample as a correct sample, and selecting one training sample from the plurality of training samples as an incorrect sample based on the correct sample; wherein the correct sample and the incorrect sample satisfy: the Euclidean distance between the anchor point sample and the incorrect sample is greater than the Euclidean distance between the anchor point sample and the correct sample, and the Euclidean distance between the anchor point sample and the incorrect sample and the Euclidean distance between the anchor point sample and the correct sample differ by no more than wherein m is the round number of the current training round, and M is a preset maximum round number; the anchor point sample is combined with all the correct samples and the incorrect samples to obtain the triple of the current training round.

[0080] In a possible implementation, the obtaining the disturbance amplitude of the current training round of the image retrieval system comprises: obtaining a disturbance amplitude increasing step; and taking the product of the round number of the current training round of the image retrieval system and the disturbance amplitude increasing step as the disturbance amplitude of the current training round of the image retrieval system.

[0081] In a possible implementation, the obtaining the disturbance mode of the current training round according to the round number of the current training round and the disturbance mode of the previous training round comprises: when , or and the disturbance mode of the previous training round is anchor point disturbance, the disturbance mode of the current training round is candidate disturbance; otherwise, the disturbance mode of the current training round is anchor point disturbance.

[0082] In a possible implementation, the candidate perturbation objective function l CAP is: l CAP = φ(H) + γ(H); where H is the difficulty of the triplet, H = d(A, P) - d(A, N), A is the anchor sample of the triplet, P is the correct sample set of the triplet, N is the error sample set of the triplet, d(A, P) is the sum of the Euclidean distances between A and each correct sample in P, d(A, N) is the sum of the Euclidean distances between A and each error sample in N; when H > 0, φ(H) = d(A, P) - d(A, N); when H < 0, φ(H) = d(A, N) - d(A, P); when H > ξ, when H < -ξ, when -ξ < H < ξ, γ(H) = γ t-1 ; where ξ is a preset model collapse threshold, γ t-1 is the value of γ(H) at the last iteration perturbation, δ is a preset distance change step, and B is the sum of the number of correct samples and error samples in the triplet.

[0083] In a possible implementation, the anchor perturbation objective function l CAP is: l CAP = l dw + Δ TR ; where, ReLU is a rectified linear function; A is the anchor sample of the triplet, P is the correct sample set of the triplet, and N is the error sample set of the triplet; n j is the jth error sample in N, d(A, n j ) is the Euclidean distance between A and n j , p i is the ith correct sample in P, d(A, p i ) is the Euclidean distance between A and p i , d max (A, N) is the maximum value of the Euclidean distances between A and each error sample in N, d min (A, N) is the minimum value of the Euclidean distances between A and each error sample in N; d max (A, P) is the maximum value of the Euclidean distances between A and each correct sample in P, d min (A, P) is the minimum value of the Euclidean distances between A and each correct sample in P; ​​Let A be the average Euclidean distance between each erroneous sample in A and N, A′ be the anchor point sample after the current perturbation, and d(A,A′) be the Euclidean distance between A and A′.

[0084] In one possible implementation, when training the deep metric learning model of the image retrieval system using adversarial triples: when the perturbation mode of the adversarial triples is obtained as a candidate perturbation, the following loss function is used: triplet Training a deep metric learning model for an image retrieval system: 0.2; where A is the anchor point sample of the triple, To counteract the perturbation of triples in the correct sample set. To combat the perturbation error sample set of triples; For A and The sum of Euclidean distances between each perturbed correct sample. For A and The sum of Euclidean distances between each perturbation error sample.

[0085] When the perturbation method for the adversarial triple is obtained as anchor point perturbation, the following loss function is used: triplet Training a deep metric learning model for an image retrieval system: Where P is the set of correct triples, and N is the set of incorrect triples. s The erroneous sample subset is determined by selecting all samples from N whose Euclidean distance to A is less than 1. The incorrect sample combination is obtained. P is the average Euclidean distance between each erroneous sample in A and N. s To obtain the correct sample subset, select all samples from P whose Euclidean distance to A is less than 1. The correct combination of samples is obtained. Let be the average Euclidean distance between all correct samples in A and P. To counteract the perturbation of anchor point samples in triplet; for The sum of the Euclidean distances between each correct sample in P, for The sum of the Euclidean distances between each erroneous sample in N, d(A,P) s ) for A and P s The sum of Euclidean distances between all correct samples, d(A,N) s ) for A and N S The sum of the Euclidean distances between each erroneous sample.

[0086] All the related contents of the steps involved in the foregoing embodiment of the deep metric learning image retrieval system optimization method can be cited to the function description of the function modules corresponding to the deep metric learning image retrieval system optimization system in the embodiment of the present application, and will not be repeated here.

[0087] The division of the modules in the embodiments of the present application is illustrative, and is only a logical function division. In actual implementation, there can be another division manner. In addition, each function module in each embodiment of the present application can be integrated in one processor, or can be a separate physical existence, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software function module.

[0088] In another embodiment of the present application, a computer device is provided, which includes a processor and a memory. The memory is configured to store a computer program, and the computer program includes program instructions. The processor is configured to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor is the computing core and control core of the terminal, and is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in the computer storage medium to implement a corresponding method flow or a corresponding function. The processor in the embodiments of the present application can be used for the operation of the deep metric learning image retrieval system optimization method.

[0089] In still another embodiment of the present application, the present application also provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in a computer device, used for storing programs and data. It can be understood that the computer readable storage medium here can include the built-in storage medium in the computer device, and of course can also include the extended storage medium supported by the computer device. The computer readable storage medium provides a storage space, which stores the operating system of the terminal. Moreover, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium here can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory. One or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to realize the corresponding steps of the deep metric learning image retrieval system optimization method in the above embodiment.

[0090] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0091] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the flow Figure 1 The function specified in one or more flows and / or blocks Figure 1 The means for performing the function specified in one or more flows or blocks.

[0092] These computer program instructions can also be stored in a computer readable memory capable of directing the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction means, which implements the flow Figure 1 The function specified in one or more flows and / or blocksFigure 1 the function specified in the one or more blocks.

[0093] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processing, so that the instructions executed on the computer or other programmable data processing devices provide a process for implementing the flow Figure 1 the flow or flows and / or blocks Figure 1 the steps of the function specified in the one or more blocks.

[0094] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it, although the above embodiments of the present application have been described in detail, those skilled in the art should understand: the specific embodiments of the present application can be modified or replaced by the same, without departing from the spirit and scope of the present application, any modification or equivalent replacement, which should be covered within the scope of protection of the claims of the present application.

Claims

1. An optimization method for a deep metric learning image retrieval system, characterized in that, include: S1: Obtain the triples and perturbation amplitude of the current training round of the image retrieval system, and obtain the perturbation method of the current training round based on the round number of the current training round and the perturbation method of the previous training round. S2: When the perturbation method of the current training round is candidate perturbation, iterate the adversarial triples based on the perturbation amplitude until the first preset number of iterations is reached or the function value of the preset candidate perturbation objective function meets the preset requirements, and then obtain the adversarial triples; when the perturbation method of the current training round is anchor perturbation, iterate the adversarial triples based on the perturbation amplitude until the second preset number of iterations is reached or the function value of the preset anchor perturbation objective function meets the preset requirements, and then obtain the adversarial triples. S3: Train the deep metric learning model of the image retrieval system by using adversarial triples; when the number of rounds in the current training round is the preset maximum number of rounds, the optimized image retrieval system is obtained; otherwise, the next training round is used as the current training round and S1~S3 are repeated. The perturbation method for the current training round, derived from the round number of the current training round and the perturbation method of the previous training round, includes: when Sometimes, or Furthermore, if the perturbation method in the previous training round was anchor point perturbation, the perturbation method in the current training round is candidate perturbation; otherwise, the perturbation method in the current training round is anchor point perturbation. The candidate perturbation objective function for: ; in, The difficulty level is for triples. , A Anchor point samples for triples, This is the correct sample set for triples. This is the set of erroneous samples for triples. for A and The sum of the Euclidean distances between all correct samples in the sample. for A and The sum of Euclidean distances between each erroneous sample; when When >0, ;when When <0, ; when > hour, ;when < hour, ;when < ;in, To set a preset model collapse threshold, When the perturbation was in the previous iteration The value of , To preset the distance variation step size, B The sum of the number of correct and incorrect samples in a triple; The anchor point perturbation objective function for: ; in, ), ReLU To correct the linear function; A Anchor point samples for triples, This is the correct sample set for triples. The set of erroneous samples for triples; for The Middle j One error sample, for A and European distance, for The Middle i One correct sample, for A and European distance, for , for A and The maximum Euclidean distance between each erroneous sample in the dataset. for A and The minimum Euclidean distance between each erroneous sample in the dataset; for , for A and The maximum Euclidean distance between all correct samples. for A and The minimum Euclidean distance between all correct samples in the dataset; , , For the subset of erroneous samples, by from N Select all that are related to A The Euclidean distance between them is less than The incorrect sample combination is obtained. for A and The average Euclidean distance between each erroneous sample in the dataset. This is a sample of anchor points after the current disturbance. for A and The Euclidean distance between them.

2. The optimization method for a deep metric learning image retrieval system according to claim 1, characterized in that, The triples for the current training round of the image retrieval system include: Obtain several training samples from the image retrieval system, and randomly select one training sample as the anchor sample; Repeat the selection process until both the number of correct and incorrect samples reach the preset values: Selection steps: Randomly select a training sample of the same category as the anchor sample as the correct sample, and select a training sample from several training samples as the incorrect sample based on the correct sample; Among them, the correct and incorrect samples satisfy the following conditions: the Euclidean distance between the anchor sample and the incorrect sample is greater than the Euclidean distance between the anchor sample and the correct sample, and the difference between the Euclidean distance between the anchor sample and the incorrect sample and the Euclidean distance between the anchor sample and the correct sample does not exceed [a certain value]. ;in, m This is the epoch number of the current training round. M The maximum number of rounds is preset; Combine the anchor samples with all correct and incorrect samples to obtain the triples for the current training round.

3. The optimization method for a deep metric learning image retrieval system according to claim 1, characterized in that, The perturbation amplitude of the current training round of the image retrieval system includes: The perturbation amplitude of the image retrieval system is increased by a step size; The perturbation magnitude of the current training round of the image retrieval system is obtained by multiplying the current training round number of the image retrieval system by the perturbation magnitude increment step.

4. The optimization method for a deep metric learning image retrieval system according to claim 1, characterized in that, When training the deep metric learning model of the image retrieval system using adversarial triples: When the perturbation method for the adversarial triple is obtained as a candidate perturbation, the following loss function is used. Training a deep metric learning model for an image retrieval system: ;in, A Anchor point samples for triples, To counteract the perturbation of triples in the correct sample set. To combat the perturbation error sample set of triples; for A and The sum of Euclidean distances between each perturbed correct sample. for A and The sum of Euclidean distances between each perturbation error sample; When the perturbation method for the adversarial triple is obtained as anchor point perturbation, the following loss function is used. Training a deep metric learning model for an image retrieval system: ;in, This is the correct sample set for triples. This is the set of erroneous samples for triples. For the subset of erroneous samples, by from N Select all that are related to A The Euclidean distance between them is less than The incorrect sample combination is obtained. for A and The average Euclidean distance between each erroneous sample in the dataset. For the correct sample subset, by from Select all that are related to A The Euclidean distance between them is less than The correct combination of samples is obtained. for A and The average Euclidean distance between each correct sample in the dataset. To counteract the perturbation of anchor point samples in triplet; for and The sum of the Euclidean distances between all correct samples in the sample. for and The sum of the Euclidean distances between each erroneous sample in the sample. for A and The sum of the Euclidean distances between all correct samples in the sample. for A and The sum of the Euclidean distances between each erroneous sample.

5. A deep metric learning image retrieval system optimization system based on the deep metric learning image retrieval system optimization method of claim 1, characterized in that, include: The perturbation information determination module is used to obtain the triples and perturbation magnitude of the current training round of the image retrieval system, and to obtain the perturbation method of the current training round based on the round number of the current training round and the perturbation method of the previous training round. The perturbation generation module is used to iterate adversarial triples based on the perturbation amplitude when the perturbation method of the current training round is a candidate perturbation, until a first preset number of iterations is reached or the function value of the preset candidate perturbation objective function meets the preset requirements, thus obtaining an adversarial triple; when the perturbation method of the current training round is an anchor point perturbation, it iterates adversarial triples based on the perturbation amplitude until a second preset number of iterations is reached or the function value of the preset anchor point perturbation objective function meets the preset requirements, thus obtaining an adversarial triple. The model training module is used to train the deep metric learning model of the image retrieval system through adversarial triples. When the number of training rounds is the preset maximum number of rounds, an optimized image retrieval system is obtained; otherwise, the next training round is used as the current training round, and the perturbation information determination module, perturbation generation module, and model training module are triggered in sequence.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the depth metric learning image retrieval system optimization method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the depth metric learning image retrieval system optimization method as described in any one of claims 1 to 4.

Images