Learning device, learning method, and program
Patent Information
- Application Number
- JP2024504337
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Priority Date
- 2022-03-04
- Filing Date
- 2022-03-04
- Publication Date
- 2024-11-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing content-based image search systems are vulnerable to adversarial samples, particularly when the attack method is unknown, as they struggle to verify the influence of adversarial perturbations on search results effectively.
A learning device and method that utilizes a feature extractor to ensure the upper and lower bounds of distance in the feature space between images are close, allowing for robustness verification against adversarial samples by calculating the infinite norm and using Interval Bound Propagation (IBP) to determine the impact of noise on feature extraction models.
Enables effective verification of the degree of influence on search results when adversarial samples are applied, ensuring that the ranking of images remains stable even with noise, thereby enhancing the robustness of content-based image search systems.
Abstract
Description
Learning device, learning method, and recording medium
[0001] The present invention relates to a learning device, a learning method, and a recording medium.
[0002] Adversarial training (AX) has been proposed as a countermeasure against attacks using adversarial samples. Adversarial training involves including adversarial samples in the training data when training a feature extraction model. By using a feature extraction model that has undergone adversarial training, it is expected that the output results will be less affected even if adversarial samples are input. For example, Non-Patent Document 1 experimentally demonstrates that adversarial training is effective against attacks using adversarial samples against content-based image retrieval.
[0003] Mo Zhou, 3 others, “Adversarial Ranking Attack and Defense”, The 2020 European Conference on Computer Vision (ECCV 2020), 2020.
[0004] Non-Patent Document 1 describes adversarial training that depends on an attack method when the attack method, such as a method for generating adversarial samples, is known. On the other hand, there may be cases where an attack method using adversarial samples against content-based image retrieval is unknown, and it is desirable to address this. It is desirable to be able to verify the degree of impact on search results when adversarial samples are given and to understand the impact when the attack method is unknown.
[0005] An example of an object of the present invention is to provide a learning device, a learning method, and a recording medium that can solve the above-mentioned problems.
[0006] According to a first aspect of the present invention, the learning device includes a learning means for learning the feature extractor f so that the upper and lower limits of the distance in feature space between images when the feature extractor f is used become close to the distance.
[0007] According to a second aspect of the present invention, the training method includes a step of training the feature extractor f so that the upper and lower limits of the distance in feature space between images when using the feature extractor f are close to the distance.
[0008] According to a third aspect of the present invention, a recording medium is a recording medium having recorded thereon a program for causing a computer to execute a step of training a feature extractor f so that the upper and lower limits of the distance in feature space between images when the feature extractor f is used become close to the distance in question.
[0009] According to the above-described learning device, learning method, and recording medium, it is possible to verify the degree of influence on search results when adversarial samples with adversarial perturbations are given in content-based image retrieval.
[0010] 9 is a schematic block diagram showing an example of the functional configuration of a content-based image retrieval device 900. ∞1 is a diagram showing the range of image x+δ in which noise δ of radius ε or less is superimposed. A diagram showing a specific example of (α, β)-robustness verification. A schematic block diagram showing an example of the functional configuration of a robustness verification device according to the first embodiment. A flowchart showing an example of the processing procedure of a robustness verification device according to the first embodiment. A schematic block diagram showing an example of the functional configuration of a robustness verification device according to the second embodiment. A flowchart showing an example of the processing procedure of a robustness verification device according to the second embodiment. A diagram showing minimization of a TripletLoss loss function. A schematic block diagram showing an example of the functional configuration of a learning device according to the third embodiment. A flowchart showing an example of the processing procedure of a learning device according to the third embodiment. A schematic block diagram showing an example of the functional configuration of a learning device according to the fourth embodiment. A flowchart showing an example of the processing procedure of a learning device according to the fourth embodiment. A schematic block diagram showing an example of the functional configuration of a learning device according to the fifth embodiment. A flowchart showing an example of the processing procedure of a learning device according to the fifth embodiment. A schematic block diagram showing an example of the functional configuration of a learning device according to the sixth embodiment. A flowchart showing an example of the processing procedure of a learning device according to the sixth embodiment. A flowchart showing an example of the processing procedure of a learning device according to the sixth embodiment. A schematic block diagram showing an example of the functional configuration of a learning device according to the seventh embodiment. A flowchart showing an example of the processing procedure of a learning device according to the seventh embodiment. A schematic block diagram showing the configuration of a computer according to at least one embodiment.
[0011] The following describes embodiments of the present invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.
[0012] <Content-Based Image Retrieval Device 900> First, an example of a content-based image retrieval device 900 that is the subject of robustness verification by the robustness verification device 100 (200) will be described.
[0013] In the real world, content-based image retrieval (CBIR) is used in medical image retrieval systems, similar product retrieval systems, face recognition systems, etc. In content-based image retrieval, when an input image q∈χ is given as a search query, a set of candidate images C={c i ∈χ} (i = 1 to N), select an image c with high similarity to q. i ∈C, where χ represents the input space of images.
[0014] In content-based image retrieval, a feature extraction model f trained by a machine learning model such as deep metric learning (DML) is used. For example, the feature extraction model f is a function f:χ→R that converts an input space of an image into an n-dimensional vector space of real numbers representing features. n Deep distance learning learns a feature extraction function f so that features can be calculated that result in a short distance between images with high similarity and a long distance between images with low similarity.
[0015] Content-based image retrieval outputs results based on the Euclidean distance dist(f(q), f(c)) of the features between an input image q and an arbitrary candidate image c∈C. For example, content-based image retrieval outputs the top k candidate images c∈C with the smallest distances from the input image q as similar images to q.
[0016] 1 is a schematic block diagram showing an example of the configuration of a content-based image retrieval device 900. When an input image q∈χ is given as a search query, the content-based image retrieval device 900 retrieves a set of candidate images C={c i ∈χ} (i=1 to N), where χ represents an input space of images. The content-based image retrieval device 900 includes an image storage unit 902, a feature extraction unit 904, and a ranking calculation unit 906.
[0017] The image storage unit 902 stores a group of candidate images (hereinafter also referred to as a group of candidate images) C={c i ∈χ} (i=1 to N). iThe candidate images C may be input to the content-based image retrieval device 900 without being stored in the storage unit.
[0018] The feature extraction unit 904 uses a feature extractor f to extract the input image q and the image c acquired from the image storage unit 902. i ∈C (i=1 to N). The feature extractor f extracts features from, for example, an image input space χ to a real n-dimensional vector space (hereinafter also referred to as a feature space) representing features as a function f:χ→R n This feature extractor f is a function that has been trained in advance using a deep learning model such as deep metric learning. Deep metric learning trains the feature extractor f so that it can calculate features that result in a short distance between images with high similarity and a long distance between images with low similarity.
[0019] The rank calculation unit 906 calculates the extracted feature value f(q) and each f(c i ) (i = 1 to N) and the Euclidean distance dist(f(q), f(c i Then, the ranking calculation unit 906 calculates the distances of a predetermined number of images c i is output as an image similar to the input image q. The image that is j-th similar to the input image q (having the j-th smallest distance) is called IR(q, C). j IR stands for Image Retrieval.
[0020] Note that the content-based image retrieval device that the robustness verification device 100 (200) targets for robustness verification is not limited to the content-based image retrieval device 900, as long as it can rank the candidate image group C for the input image q using the feature extractor f. The robustness verification device 100 (200) has, as part of its input, the input image q, the candidate image group C, and the feature extractor f, which are parameters of the content-based image retrieval device 900.
[0021] <Noise> Next, we will explain noise, which is a minute adversarial perturbation intentionally added to an image, assumed by the robustness verification device 100 (200). Adversarial examples (AX) are known to be a serious problem for the safety of machine learning models. Adversarial examples are data created by intentionally adding minute perturbations that cause machine learning models such as feature extraction models to make erroneous decisions. Perturbations added by adversarial examples are called adversarial perturbations. When an adversarial example is input, a machine learning model may output a different class or value compared to data without adversarial perturbations.
[0022] Content-based image retrieval using feature extraction models can also be subject to attacks using adversarial samples. In this case, the adversarial perturbation is noise added to the image, etc. Two potential threats to content-based image retrieval using adversarial samples are the query attack and the candidate attack. A query attack involves inputting an adversarial sample as the input image, which is the search query, to manipulate the output of content-based image retrieval. A candidate attack involves inputting an adversarial sample as the candidate image to manipulate the output of content-based image retrieval. Both attacks are achieved by manipulating the output of a feature extraction model with an adversarial sample.
[0023] Examples of attacks using adversarial samples include preferentially recommending one's own products in a similar product search system used in online sales using content-based image search, or impersonating someone else's face in a face recognition system using content-based image search.
[0024] When the input space of images is χ, the robustness verification device 100 (200) calculates the infinity norm L ∞ Even if noise δ∈χ of radius ε or less is given, the j-th closest image IR(q, C) in the input image q and the candidate image group C is jThat is, the robustness verification device 100 (200) verifies that the rank of ∀δ∈{δ∈χ||δ|| ∞ Even if image x is an image x+δ with noise for {≦ε}, image IR(q, C) j The image x to which noise is added is the input image q in the robustness verification device 100, and is an arbitrary candidate image c in the robustness verification device 200. i ∈C.
[0025] FIG. 2 shows the infinity norm L ∞ This figure shows the range of image x + δ with noise δ of radius ε or less in the infinity norm. As shown in Figure 2, in the case of the infinity norm, this range becomes a hypersphere of radius ε with x as the center. Image x + δ is an element within this hypersphere. Note that in the infinity norm L ∞ is an example, and the norm is not limited to this.
[0026] First Embodiment Next, a first embodiment of the present invention will be described. The first embodiment is a robustness verification device 100 that verifies the robustness of a content-based image retrieval device 900 against a query image attack.
[0027] An input image that is a query of the content-based image retrieval device 900 is assumed to be q, and an input image to which noise δ of a size equal to or less than ε is added is assumed to be q+δ. When the input image q is input, the content-based image retrieval device 900 generates a set of candidate images C={c i ∈χ} (i=1 to N), the j-th image similar to q is searched for as IR(q,C) j In this case, even if an input image q+δ is input to the content-based image retrieval device 900, the robustness verification device 100 determines that IR(q, C) j Specifically, the robustness verification device 100 verifies whether the ranking of IR(q, C) does not change significantly even when an input image q+δ is input to the content-based image retrieval device 900. j The ranking of IR(q, C) j and a group of target images C with low similarity βWe verify whether the fluctuation is at most α.
[0028] <(α, β)-Robustness Verification Against Query Image Attacks> First, (α, β)-Robustness Verification, which is a basic concept when the robustness verification device 100 verifies robustness, will be described.
[0029] The (α, β)-robustness verification against query image attacks is defined as follows: α is a natural number greater than or equal to 0, and β is a real number greater than or equal to 0. In this case, For IR(q,C) j is verified as (α, β)-robust, where, is.
[0030] Equation (1) represents the range of noise δ added to the input image. χ represents the input space of the image. "δ∈χ" indicates that δ is also an element of the input space of the image. "||δ|| p " is the infinity norm of δ L ∞ represents ||δ|| p ≦ε” is the infinity norm L of δ ∞ This indicates that the magnitude of the p-norm is equal to or less than ε. This is shown in Figure 2. Note that the p-norm is not limited to the ∞-norm, and may be any of the 1-, 2-, or p-norm.
[0031] Equation (3) is a set of images (hereinafter also referred to as a target image group) C that are the subject of robustness verification in equation (2). β It represents "{IR(q, C) j}” is IR(q,C) j (the j-th candidate image similar to q that the content-based image retrieval device 900 retrieves from the candidate image group C when the input image q is input) is defined as C β It represents the inclusion of
[0032] “{c|c∈C,β≦||f(c)−f(IR(q,C) j ) || qRegarding "{β≦∥f(c)−f(IR(q,C)}", first, "c∈C" indicates the condition that c is an element of the candidate image group C. "f(c)" indicates the feature extracted by the feature extractor f for image c. This feature extractor f is the feature extractor of the content-based image retrieval device 900. "{β≦∥f(c)−f(IR(q,C) j ) || q}" is the image IR(q,C) j This represents the condition that the magnitude of the difference in q norm between the feature amount of C and the feature amount of image c is equal to or greater than β. β β is a parameter for determining candidate images that take into consideration changes in ranking. β indicates that images with a distance difference of less than β are included in C β By not including it in IR(q,C) j This means that fluctuations in the ranking of similar images are allowed. The larger β is, the easier it is to verify (α, β)-robustness. Note that the q-norm can be 1, 2, p, or ∞ norm.
[0033] Equation (2) expresses the specific conditions for (α, β)-robustness verification. Rank(q, c, C)) expresses the ranking of image c in the candidate image group C in terms of similarity using feature extractor f when input image q is used. Therefore, "Rank(q + δ, IR(q, C)" j , C β )) is the target image group C obtained by equation (3). β In this case, the image IR(q, C) when the input image is q+δ is j It also represents the ranking of "Rank(q,IR(q,C) j , C β )) is the target image group C obtained by equation (3). β In this case, the image IR(q, C) when the input image is q is j Therefore, the formula (2) expresses the ranking of the target image group C obtained by the formula (3). β In this case, the image IR(q, C) when the input image is q+δ is j The ranking of the image IR(q, C) when the input image is q jThis condition states that the ranking of the (α, β) is allowed to vary by at most α. α is a parameter that indicates the amount of variation in the ranking that is allowed, and indicates that a ranking variation of at most α is allowed. The larger α is, the easier it is to verify (α, β)-robustness.
[0034] 3 is a diagram showing a specific example of (α, β)-robustness verification. The (α, β)-robustness verification is a method for verifying whether IR(q, C) is robust when a noise δ of a magnitude equal to or less than ε is added to an input image q. j The ranking of IR(q, C) j and a group of target images C with low similarity β We verify whether the fluctuation is at most α in
[0035] An example of the feature space created by the feature extractor f is shown in Fig. 3. The circle shown in Fig. 3 has a radius β and its center is the image IR(q, C). j The point f(IR(q, C) represents the feature value obtained by the feature extractor f. j ) where f(q) and f(q+δ) are points that represent the feature values obtained by the feature extractor f for the input image q and the input image q+δ with noise δ added, respectively. 1 ), f(c 2 ), f(c 3 ), f(c 4 ), f(c 5 ) are images c included in the candidate image group C, respectively. 1 , c 2 , c 3 , c 4 , c 5 The points represent the feature quantities calculated by the feature quantity extractor f. The distance between the points represented by the feature quantities is the distance on the plane shown in FIG.
[0036] As shown in FIG. 3, when a parameter β is used that takes into account the change in ranking, β≦∥f(c)−f(IR(q,C)) in equation (3) is satisfied. j ) || q The image that satisfies c is outside the circle. 3 , c 4 , c 5 Therefore, C β = {IR(q, C) j , c 3 , c 4 , c5}.
[0037] The ranking by the content-based image retrieval device 900 for the input image q before adding noise is C β The images included in the list are ranked in order of proximity to f(q), so 1st place: IR(q, C) j 2nd place: c 4 3rd place: c 3 4th place: c 5 On the other hand, the ranking by the content-based image retrieval apparatus 900 for the input image q+δ after adding noise δ is C β The images included in the list are ranked in order of proximity to f(q+δ), so 1st place: c 3 2nd place: c 4 3rd place: IR (q, C) j 4th place: c 5 This becomes:
[0038] Therefore, in equation (2), Rank(q+δ, IR(q, C) j , C β ) = 3, and Rank(q,IR(q,C) j , C β ) = 1, so equation (2) becomes 1 - α ≦ 3 ≦ 1 + α. Therefore, if α ≧ 2, then IR(q, C) j is (α, β)-robust verified, and if α=0, 1, then IR(q, C) j is (α, β) - Robustness has not been verified (Not Verified).
[0039] <Robustness Verification Device 100> The robustness verification device 100 performs the above-mentioned (α, β)-robustness verification, but accurate calculation of the (α, β)-robustness verification is difficult due to the amount of calculation involved. In other words, it is difficult for the robustness verification device 100 to verify equation (2) for any δ that satisfies (1). Therefore, the robustness verification device 100 performs the following: Here, q is an input image, and c is an element of the target image group C.
[0040] That is, the robustness verification device 100 Here, "d(f(q+δ), f(c))" represents the Euclidean distance d(f(q+δ), f(c)) between the feature value f(q+δ) of input image q with noise δ and the feature value f(c) of image c. q (f(q), f(c)) is the upper bound of d(f(q + δ), f(c)) for any δ that satisfies equation (4). q " q indicates that noise is included in q. q "_d(f(q), f(c))" is the lower bound of d(f(q+δ), f(c)) for any δ that satisfies equation (4). q " indicates that noise is included in q.
[0041] The robustness verification device 100 performs the calculation using, for example, the well-known technique Interval Bound Propagation (IBP) described in the following non-patent document: IBP is a technique for solving the problem of noise δ∈{δ|||δ|| ∞ ≦ε}, when an image q + δ with noise δ is input, the upper and lower limits of each element i of the feature f(q + δ) are calculated sequentially. Here, if the feature is an n-dimensional vector, i represents the i-th element (1≦i≦n).
[0042] <Non-patent literature> Sven Gowal and 8 others, "On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models", The 2019 International Conference on Computer Vision (ICCV 2019), 2019.
[0043] The robustness verification device 100 uses IBP to calculate f(q+δ) i Upper limit of f(q) i and lower bound f(q) iwhere i represents the i-th element of the n-dimensional vector. Then, the robustness verification device 100 calculates the upper limit d of d(f(q+δ), f(c)) using the upper and lower limits. q (f(q), f(c)) and lower bound d q (f(q), f(c)) are calculated using equations (6) and (7), respectively.
[0044]
[0045] In equation (6), "|  ̄ f(q) i -f(c) i | 1 " represents the absolute value of the difference between the upper limit of the i-th element of the feature amount of the input image q and the i-th element of the feature amount of the image c. i -_f(q) i | 1 " represents the absolute value of the difference between the i-th element of the feature of image c and the lower limit of the i-th element of the feature of input image q. The right-hand side of equation (6) represents the square of the larger of these values, summed over all n-dimensional elements i. This value is the upper limit d of d(f(q+δ), f(c)). q (f(q), f(c)).
[0046] In equation (7), "f(q) i -f(c) i " represents the difference between the upper limit of the i-th element of the feature amount of the input image q and the i-th element of the feature amount of the image c. i -_f(q) i " represents the difference between the i-th element of the feature of image c and the lower limit of the i-th element of the feature of input image q. The right-hand side of equation (7) represents the square of the smaller of these values and 0, and the sum of the squares for all n-dimensional elements i. This value is the lower limit_d of d(f(q+δ), f(c)). q (f(q), f(c)).
[0047] When the robustness verification device 100 calculates the upper and lower bounds of equation (5) using the above-mentioned IBP calculation method, the norm of equation (4) is the infinity norm. The robustness verification device 100 may also calculate the upper and lower bounds of d(f(q+δ), f(c)) using a calculation method other than IBP, in which case the norm of equation (4) is not limited to the infinity norm.
[0048] The robustness verification device 100 calculates the upper limit d(f(q+δ), f(c)) q (f(q), f(c)) and lower bound d q (α, β)-robustness verification is performed using (f(q), f(c)).
[0049] 4 is a schematic block diagram showing an example of the functional configuration of the robustness verification device 100 according to the first embodiment. The robustness verification device 100 includes a similar image identification unit 102, a comparison image calculation unit 104, an upper / lower limit calculation unit 106, and a rank verification unit 108. The rank verification unit 108 includes a rank calculation unit 110 and a rank counting unit 112. The robustness verification device 100 performs a robustness verification using an input image q∈χ, which is a query, and a set of candidate images C={c i ∈χ} (i=1 to N), a feature extractor f, a perturbation size ε, parameters α and β, and a rank j are input. Note that the robustness verification device 100 may be provided with a storage unit that stores the candidate image group C, rather than receiving the candidate image group C as input.
[0050] The similar image identification unit 102 receives an input image q, a group of candidate images C, a feature extractor f, and a rank j, and identifies an image IR(q, C) that is j-th similar to the input image q in the group of candidate images C. j Specifically, the similar image identification unit 102 uses a feature extractor f to extract the input image q and each candidate image c i ∈C features f(q), f(c i ) (i=1 to N). Then, the similar image identification unit 102 calculates the feature amount f(q) and each f(c i ) and the Euclidean distance dist(f(q), f(c i Then, the similar image identification unit 102 calculates the image that is the jth most similar (has the jth smallest distance) to the input image q as IR(q, C). jThe similar image identification unit 102 corresponds to the search performed by the content-based image search device 900.
[0051] The comparison image calculation unit 104 calculates IR(q, C) j The candidate image group C, the feature extractor f, and the parameter β are input, and a target image group C is a set of images to be subjected to the robustness verification shown in equation (8). β Calculate.
[0052]
[0053] Specifically, the comparison image calculation unit 104 calculates IR(q, C) j C β The comparison image calculation unit 104 also calculates IR(q, C) j For each target image c in the candidate image group C, a feature extractor f extracts a feature f(IR(q, C) j Then, the comparison image calculation unit 104 calculates the magnitude of the difference in the feature amount in the q-norm, ∥f(c)−f(IR(q,C) j ) || q If it is greater than or equal to β, the comparison image calculation unit 104 calculates the target image c as C β β is a parameter for determining candidate images that take into consideration changes in ranking. β is a parameter for determining candidate images that take into consideration changes in ranking. β By not including it in IR(q,C) j This means that fluctuations in the ranking of similar images are allowed. The larger β is, the easier it is to verify (α, β)-robustness. Note that the q-norm can be 1, 2, p, or ∞ norm.
[0054] The upper and lower limit calculation unit 106 calculates the target image group C β The input image q, the feature extractor f, and the perturbation size ε are input, and each target image c∈C β For any δ that satisfies the above-mentioned equation (4), the upper limit d of d(f(q+δ), f(c)) that satisfies the equation (5) is q (f(q), f(c)) and lower bound d q Calculate (f(q), f(c)).
[0055] Specifically, the upper and lower limit calculation unit 106 calculates the average value of each target image c∈C using the above-mentioned IBP (Interval Bound Propagation). β On the other hand, the upper limit d of d(f(q+δ), f(c)) shown in equation (6) is q (f(q), f(c)), and the lower limit d(f(q+δ), f(c)) shown in equation (7) q (f(q), f(c)). The upper and lower limit calculation unit 106 corresponds to an example of upper and lower limit calculation means. Note that the method by which the upper and lower limit calculation unit 106 calculates the upper and lower limits of d(f(q+δ), f(c)) is not limited to IBP, and other methods may be used.
[0056] The ranking verification unit 108 compares the input image q with the image IR(q, C) j and a group of target images C to be subjected to robustness verification. β and each target image c∈C β The upper bound of d(f(q+δ), f(c)) for q (f(q), f(c) and lower bound_d q The rank verification unit 108 receives (f(q), f(c)) and a parameter α as input. Then, the rank verification unit 108 performs (α, β)-robustness verification, that is, the rank verification of the target image group C β In this case, the image IR(q, C) when the input image is q+δ is j The ranking of the image IR(q, C) when the input image is q j The condition for the (α, β)-robustness verification performed by the ranking verification unit 108 is not based on the definition of equation (2), but is a condition using the upper and lower limits of d(f(q+δ), f(c)). Specifically, the ranking verification unit 108 verifies whether the following equations (9) and (10) are satisfied.
[0057]
[0058] The rank calculation unit 110 of the rank verification unit 108 calculates "Rank(q,IR(q,C)" in the formulas (9) and (10). j , C β "Rank(q,IR(q,C)" is calculated.j , C β )) is the target image group C obtained by equation (8). β In this case, the image IR(q, C) when the input image is q is j Specifically, the ranking calculation unit 110 uses a feature extractor f to calculate the ranking of q,IR(q,C) j , ∀c∈C β Each feature f(q), f(IR(q, C) j The ranking is calculated by finding the Euclidean distance between f(c) and f(c).
[0059] The rank counting unit 112 of the rank verification unit 108 calculates the right side of equation (9) and the left side of equation (10). The rank counting unit 112 first calculates the right side of equation (9). q (f(q), f(c)) is the upper bound of d(f(q+δ), f(c)), and "_d q (f(q), f(IR(q,C) j ))' is d(f(q+δ), f(IR(q,C) j The rank counting unit 112 calculates the rank of the q (f(q), f(c))<_d q (f(q), f(IR(q,C) j )))]" is counted as 1 when the upper limit is less than the lower limit. β to IR(q, C) j For all elements c except for , "1 [ d q (f(q), f(c))<_d q (f(q), f(IR(q,C) j )))] and add 1 to calculate the value.
[0060] The rank counting unit 112 then calculates the left side of equation (10). q (f(q), f(IR(q,C) j ))' is d(f(q+δ), f(IR(q,C) j )) is the upper limit of "_d q (f(q), f(c)) is the lower limit of d(f(q+δ), f(c)). q (f(q), f(IR(q,C)j ))<_d q (f(q), f(c))]" is counted as 1 when the upper limit is less than the lower limit. β to IR(q, C) j For all elements c except for , "1 [ d q (f(q), f(IR(q,C) j ))<_d q (f(q), f(c))]" and C β Number of elements in |C β Calculate the value by subtracting the counted value from |.
[0061] The ranking verification unit 108 determines whether the calculated value of the right side of the formula (9) is "Rank(q, IR(q, C) j , C β )-α" or more, and the calculated value of the left side of the equation (10) is "Rank(q,IR(q,C) j , C β ) + α". Here, α is a parameter indicating the amount of fluctuation in ranking that is allowed, and indicates that a fluctuation in ranking of at most α is allowed. The larger α is, the easier it is to verify (α, β)-robustness. If the condition is met, the ranking verification unit 108 outputs that the (α, β)-robustness verification has been performed (Verified), and if the condition is not met, the ranking verification unit 108 outputs that the (α, β)-robustness verification has not been performed (Not Verified).
[0062] If the conditions of equations (9) and (10) verified by the ranking verification unit 108 are satisfied, it is known that the condition of equation (2) for (α, β)-robustness verification is satisfied (sufficient condition). Therefore, the conditions of equations (9) and (10) are satisfied for the target image group C β In this case, the image IR(q, C) when the input image is q+δ is j The ranking of the image IR(q, C) when the input image is q jThis means that the order of f(q+δ), f(c) fluctuates by at most α. Note that in the robustness verification device 100, the upper and lower limit calculation unit 106 uses the upper and lower limits of d(f(q+δ), f(c)), so even if an input is originally verified in the definition of (α, β)-robustness verification, it may be deemed not verified by the robustness verification device 100.
[0063] Next, the operation of the robustness verification device 100 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of the processing procedure when the robustness verification device 100 performs (α, β)-robustness verification.
[0064] First, the robustness verification device 100 calculates an input image qεχ, which is a query, and a set of candidate images C={c i ∈χ} (i=1 to N), a feature extractor f, a perturbation size ε, parameters α and β, and a rank j are received as input (step S101).
[0065] Next, the similar image identification unit 102 identifies the jth image IR(q, C) from the candidate image group C that is similar to the input image q. j Specifically, the similar image identification unit 102 uses a feature extractor f to identify the input image q and each candidate image c i ∈C features f(q), f(c i ) (i = 1 to N), and calculate the feature value f(q) and each f(c i ) and the Euclidean distance dist(f(q), f(c i Then, the similar image identification unit 102 calculates the image having the j-th smallest distance from the input image q as IR(q, C). j (step S102).
[0066] Next, the comparison image calculation unit 104 calculates a target image group C, which is a set of images to be subjected to the robustness verification. β Specifically, the comparison image calculation unit 104 selects IR(q, C) j C β The comparison image calculation unit 104 also calculates IR(q, C) j For each target image c in the candidate image group C, the feature value f(IR(q, C)j Then, the comparison image calculation unit 104 calculates the magnitude of the difference in the feature amount in the q-norm, ∥f(c)−f(IR(q,C) j ) || q If " is greater than or equal to β, the target image c is β (step S103).
[0067] Next, the upper and lower limit calculation unit 106 calculates the target image group C β For each target image c, for any δ that satisfies equation (4), the upper limit d of d(f(q+δ), f(c)) that satisfies equation (5) is q (f(q), f(c)) and lower bound d q (f(q), f(c)) is calculated (step S104).
[0068] Next, the rank calculation unit 110 of the rank verification unit 108 calculates Rank(q, IR(q, C) j , C β Specifically, the ranking calculation unit 110 calculates q,IR(q,C) using a feature extractor f. j , ∀c∈C β Each feature f(q), f(IR(q, C) j The ranking is calculated by finding the Euclidean distance between f(c) and f(c) (step S105).
[0069] Next, the rank counting unit 112 of the rank verification unit 108 calculates the right side of the equation (9). That is, the rank counting unit 112 calculates the number of images in the target image group C β to IR(q, C) j For all elements c except for , "1 [ d q (f(q), f(c))<_d q (f(q), f(IR(q,C) j )))]" and calculates a value by adding 1. The rank counting unit 112 also calculates the left side of equation (10). That is, the rank counting unit 112 calculates the left side of equation (10) by adding 1 to the rank of the target image group C β to IR(q, C) j For all elements c except for , "1 [ d q (f(q), f(IR(q,C) j ))<_d q(f(q), f(c))]" and C β Number of elements in |C β A value is calculated by subtracting the counted value from | (step S106).
[0070] Next, the ranking verification unit 108 checks whether the calculated value of the right side of the formula (9) is "Rank(q, IR(q, C) j , C β )-α" or more, and the calculated value of the left side of the equation (10) is "Rank(q,IR(q,C) j , C β ) + α". If the condition is met, the ranking verification unit 108 outputs "(α, β) - robustness verification has been performed (Verified)", and if the condition is not met, the ranking verification unit 108 outputs "(α, β) - robustness verification has not been performed (Not Verified)" (step S107).
[0071] After step S107, the robustness verification device 100 ends the process in Fig. 5. Note that the robustness verification device 100 may perform (α, β)-robustness verification for multiple j or for all j in the range 1≦j≦N, rather than performing (α, β)-robustness verification for only a specific rank j.
[0072] As described above, the similar image identification unit 102 identifies the similar image IR(q, C) j The comparison image calculation unit 104 identifies a group of target images C β The upper and lower limit calculation unit 106 calculates the upper and lower limits of d(f(q+δ), f(c)) for each target image. The ranking verification unit 108 verifies the conditions of equations (9) and (10). As a result, the robustness verification device 100 performs (α, β)-robustness verification, that is, the robustness verification of the target image group C β In this case, the image IR(q, C) when the input image is q+δ is j The ranking of the image IR(q, C) when the input image is q jIn other words, the robustness verification device 100 can verify whether search results are not affected in content-based image retrieval even when an adversarial sample is given to an input image that is a query, where the adversarial perturbation is applied.
[0073] The upper and lower limit calculation unit 106 also calculates the target image group C β For each target image c, for any δ that satisfies equation (4), the upper limit d of d(f(q+δ), f(c)) is q (f(q), f(c)) and lower bound d q (f(q), f(c)). Then, the robustness verification device 100 performs (α, β)-robustness verification using these upper and lower bounds. This allows the robustness verification device 100 to perform (α, β)-robustness verification with a small amount of calculation (practical calculation time).
[0074] Furthermore, the comparison image calculation unit 104 calculates the target image group C β The robustness verification device 100 uses a parameter β to allow for fluctuations in the ranking when determining (α, β). This allows the robustness verification device 100 to adjust the accuracy of the verification, such as making the (α, β)-robustness verification easier as the parameter β increases.
[0075] The ranking verification unit 108 also uses a parameter α to determine the amount of ranking allowed, which allows the robustness verification device 100 to adjust the accuracy of the verification, such as increasing α to make the (α, β)-robustness verification easier.
[0076] Second Embodiment Next, a second embodiment of the present invention will be described. The second embodiment is a robustness verification device 200 that verifies the robustness of a content-based image retrieval device 900 against candidate image attacks.
[0077] Let q be an input image that is a query for the content-based image retrieval device 900. When the input image q is input, the content-based image retrieval device 900 generates a set of candidate images C={c i∈χ} (i=1 to N), the j-th image similar to q is searched for as IR(q,C) j In addition, noise δ with magnitude less than ε i A group of candidate images with (i = 1 to N) ~ C = {c i +δ i |c i ∈C} (i=1 to N). In this case, the robustness verification device 200 converts the candidate images of the content-based image retrieval device 900 into a candidate image set to which noise has been added. ~ Even if C is used, IR(q, C) j Specifically, the robustness verification device 200 verifies whether the ranking of candidate images in the content-based image retrieval device 900 changes significantly. ~ Even if C is used, IR(q, C) j It is verified whether the rank of j when the candidate image group is C varies by at most α.
[0078] <α-Robustness Verification Against Candidate Image Attacks> First, α-Robustness Verification, which is a basic concept when the robustness verification device 200 verifies robustness, will be described.
[0079] The α-robustness verification against query image attacks is defined as follows: α is a natural number equal to or greater than 0. In this case, For IR(q,C) j is verified as α-robust, where, is.
[0080] Equation (11) is the image c of the candidate image group C. i Noise δ added to (i = 1 to N) i χ represents the input space of the image. "δ∈χ" indicates that δ is also an element of the input space of the image. "||δ|| p " is the infinity norm of δ L ∞ represents ||δ|| p ≦ε” is the infinity norm L of δ ∞This shows that the magnitude of the δ is less than or equal to ε. This is shown in Figure 2. 1 , …, ∀δ N ∈{δ∈χ, ||δ|| p ≦ε}” is the noise i is any element of the set that satisfies this condition. Note that the p-norm is not limited to the ∞-norm, and may be any of the 1-, 2-, and p-norms.
[0081] Equation (13) is a set of images (hereinafter also referred to as a target image group) to be subjected to robustness verification in equation (12). ~ It represents C. ~ C = {c i +δ i |c i ∈C}(i=1 to N)" represents each candidate image c of the candidate image group C. i For any noise δ i Image c with i +δ i The target image set ~ Note that, unlike the case of query image attacks, the parameter β, which determines the candidate image taking into consideration the change in ranking, is not introduced. This is because, in the robustness verification in the candidate image attack, any candidate image c in the candidate image group C is i This is because it is assumed that noise may be present in (i = 1 to N). In other words, if the feature extractor is f, then f(c i ) (i = 1 to N) may be altered by noise, so similar images cannot be excluded based on the distance in feature space, as in the case of query image attacks.
[0082] Equation (12) expresses the specific conditions for α-robustness verification. "Rank(q, c, C)" expresses the ranking of image c in the candidate image group C in terms of similarity using feature extractor f when input image q is used. Note that this feature extractor f is the feature extractor of the content-based image retrieval device 900. Therefore, "Rank(q, IR(q, C)" j , ~ C) is the target image group calculated by equation (13). ~ Image IR(q, C) when q is the input image in Cj Also, "j" represents the rank of Rank(q,IR(q,C) j , C)), and when an input image in the candidate image group C is q, the image IR(q, C) j Therefore, the equation (12) is the rank j of the target image group obtained by the equation (13). ~ Image IR(q, C) when q is the input image in C j The ranking of the image IR(q, C) when the input image in the candidate image group C is q j This condition means that the rank j of the criterion j varies by at most α. α is a parameter that indicates the amount of fluctuation in the rank that is allowed, and indicates that a fluctuation in rank of at most α is allowed. The larger α is, the easier it is to verify α-robustness.
[0083] <Robustness Verification Device 200> The robustness verification device 200 performs the above-mentioned α-robustness verification, but accurate calculation of the α-robustness verification is difficult due to the amount of calculation involved. In other words, it is difficult for the robustness verification device 200 to verify equation (12) for any δ that satisfies (11). Therefore, the robustness verification device 200 performs the following: Here, q is the input image, and c is an element of the candidate image group C.
[0084] That is, the robustness verification device 200 Here, "d(f(q), f(c+δ))" represents the Euclidean distance d(f(q, f(c+δ)) between the feature value f(q) of the input image q and the feature value f(c+δ) of the image c with noise δ added. c (f(q), f(c)) is the upper bound of d(f(q), f(c+δ)) for any δ that satisfies equation (14). c "c" indicates that noise is added to "_d" c "_d(f(q), f(c))" is the lower bound of d(f(q), f(c+δ)) for any δ that satisfies equation (14). c "c" indicates that noise is included in c.
[0085] The robustness verification device 200 performs the calculation using, for example, the well-known technique Interval Bound Propagation (IBP) described in the aforementioned reference. IBP is a method for solving the problem of noise δ∈{δ|||δ|| ∞ ≦ε}, when an image x+δ with noise δ is input, the upper and lower limits of each element i of the feature f(x+δ) are calculated sequentially. Here, if the feature is an n-dimensional vector, i represents the i-th element (1≦i≦n).
[0086] The robustness verification device 200 uses IBP to calculate f(c+δ) i Upper limit of f(c) i and lower bound_f(c) i where i represents the i-th element of the n-dimensional vector. Then, the robustness verification device 200 calculates the upper limit d of d(f(q), f(c+δ)) using the upper and lower limits. c (f(q), f(c)) and lower bound d c (f(q), f(c)) are calculated using equations (16) and (17), respectively.
[0087]
[0088] In equation (16), "|  ̄ f(c) i -f(q) i | 1 " represents the absolute value of the difference between the upper limit of the i-th element of the feature amount of image c and the i-th element of the feature amount of input image q. i -_f(c) i | 1 " represents the absolute value of the difference between the i-th element of the feature of input image q and the lower limit of the i-th element of the feature of image c. The right-hand side of equation (16) represents the square of the larger of these values, summed over all n-dimensional elements i. This value is the upper limit d of d(f(q), f(c+δ)). c (f(q), f(c)).
[0089] In equation (17), "f(c) i -f(q)i " represents the difference between the upper limit of the i-th element of the feature amount of image c and the i-th element of the feature amount of input image q. "f(q) i -_f(c) i " represents the difference between the i-th element of the feature of input image q and the lower limit of the i-th element of the feature of image c. The right-hand side of equation (17) represents the square of the smaller of these values and 0, and the sum of the squares of the sum over all n-dimensional elements i. This value is the lower limit_d of d(f(q), f(c+δ)). c (f(q), f(c)).
[0090] When the robustness verification device 200 calculates the upper and lower bounds of Equation (15) using the above-mentioned IBP-based calculation method, the norm of Equation (14) is the infinity norm. The robustness verification device 200 may also calculate the upper and lower bounds of d(f(q), f(c+δ)) using a calculation method other than IBP, in which case the norm of Equation (14) is not limited to the infinity norm.
[0091] The robustness verification device 200 calculates the upper limit d(f(q), f(c+δ)) c (f(q), f(c)) and lower bound d c α-robustness verification is performed using (f(q), f(c)).
[0092] 6 is a schematic block diagram showing an example of the functional configuration of a robustness verification device 200 according to the second embodiment. The robustness verification device 200 includes a similar image identification unit 202, an upper / lower limit calculation unit 206, and a rank verification unit 208. The rank verification unit 208 includes a rank counting unit 212. The robustness verification device 200 performs the robustness verification using an input image q∈χ, which is a query, and a set of candidate images C={c i ∈χ} (i=1 to N), a feature extractor f, a perturbation size ε, a parameter α, and a rank j are input. Note that the robustness verification device 200 may not receive the candidate image group C as input, but may instead include a storage unit that stores the candidate image group C.
[0093] The similar image identification unit 202 receives an input image q, a group of candidate images C, a feature extractor f, and a rank j, and identifies an image IR(q, C) that is j-th similar to the input image q in the group of candidate images C. jSpecifically, the similar image identification unit 202 uses a feature extractor f to extract the input image q and each candidate image c i ∈C features f(q), f(c i ) (i=1 to N). Then, the similar image identification unit 202 calculates the feature amount f(q) and each f(c i ) and the Euclidean distance dist(f(q), f(c i Then, the similar image identification unit 202 calculates the image that is the jth most similar (having the jth smallest distance) to the input image q as IR(q, C). j The similar image identification unit 202 corresponds to the search performed by the content-based image search device 900.
[0094] The upper and lower limit calculation unit 206 receives the candidate image group C, the input image q, the feature extractor f, and the perturbation size ε, and calculates the upper limit d of d(f(q), f(c+δ)) that satisfies the above-mentioned equation (5) for any δ that satisfies the above-mentioned equation (14) for each target image cεC. c (f(q), f(c)) and lower bound d c Calculate (f(q), f(c)).
[0095] Specifically, the upper and lower limit calculation unit 206 uses the above-mentioned IBP (Interval Bound Propagation) to calculate the upper limit d of d(f(q), f(c+δ)) shown in equation (16) for each target image c∈C. c (f(q), f(c)), and the lower limit d(f(q), f(c+δ)) shown in equation (17) c (f(q), f(c)). Note that the method by which the upper and lower limit calculation unit 206 calculates the upper and lower limits of d(f(q), f(c+δ)) is not limited to IBP, and other methods may be used.
[0096] The ranking verification unit 208 compares the input image q with the image IR(q, C) j and the upper bound d of d(f(q), f(c+δ)) for the candidate image group C and each target image c∈C. c (f(q), f(c) and lower bound_d c(f(q), f(c)) and a parameter α are received as input. Then, the ranking verification unit 208 performs α-robustness verification, that is, performs a target image group obtained by adding noise to the candidate image group C. ~ When C is set, the target image group ~ Image IR(q, C) when q is the input image in C j The ranking of the image IR(q, C) when the input image in the candidate image group C is q j It is verified that the rank j of f(q), f(c+δ) fluctuates by at most α. The condition for the α-robustness verification performed by the rank verification unit 208 is not based on the definition of equation (12), but is a condition using the upper and lower limits of d(f(q), f(c+δ)). Specifically, the rank verification unit 208 verifies whether the following equations (18) and (19) are satisfied.
[0097]
[0098] The rank counting unit 212 of the rank verification unit 208 calculates the right side of equation (18) and the left side of equation (19). The rank counting unit 212 first calculates the right side of equation (18). c (f(q), f(c)) is the upper bound of d(f(q), f(c+δ)), and "_d c (f(q), f(IR(q,C) j ))' is d(f(q), f(IR(q,C) j +δ)). The rank counting unit 212 calculates the c (f(q), f(c))<_d c (f(q), f(IR(q,C) j )))]" is counted as 1 when the upper limit is less than the lower limit. j For all elements c except for , "1 [ d c (f(q), f(c))<_d c (f(q), f(IR(q,C) j )))] and add 1 to calculate the value.
[0099] The rank counting unit 212 then calculates the left side of equation (19). c(f(q), f(IR(q,C) j ))' is d(f(q), f(IR(q,C) j +δ)) and "_d c (f(q), f(c)) is the lower limit of d(f(q), f(c+δ)). c (f(q), f(IR(q,C) j ))<_d c When the upper limit is less than the lower limit, the rank counting unit 212 counts IR(q,C) from the candidate image group C. j For all elements c except for , "1 [ d c (f(q), f(IR(q,C) j ))<_d c (f(q), f(c))] is counted, and the value obtained by subtracting the counted value from the number N of elements of C is calculated.
[0100] The ranking verification unit 208 verifies whether the calculated value of the right side of the formula (18) is equal to or greater than "j-α" and whether the calculated value of the left side of the formula (19) is equal to or less than "j+α". Here, "j" is the sum of the values of Rank(q,IR(q,C) j , C), and the image IR(q, C) in the similarity with the input image q when the candidate image group C is not added with noise. j The rank is the rank of the item. α is a parameter indicating the amount of fluctuation in rank that is allowed, and indicates that a rank fluctuation of at most α is allowed. The larger α is, the easier it is to verify the α-robustness. If the condition is met, the rank verification unit 208 outputs that the α-robustness verification has been performed (Verified), and if the condition is not met, the rank verification unit 208 outputs that the α-robustness verification has not been performed (Not Verified).
[0101] If the conditions of Equation (18) and Equation (19) verified by the ranking verification unit 208 are satisfied, it is known that the condition of Equation (12) for α-robustness verification is satisfied (sufficient condition). Therefore, the conditions of Equation (18) and Equation (19) are satisfied when the target image group obtained by adding noise to the candidate image group C is ~ When C is set, the target image group ~ Image IR(q, C) when q is the input image in Cj The ranking of the image IR(q, C) when the input image in the candidate image group C is q j This means that the value f(q) fluctuates by at most α with respect to the rank j of f(q), f(c+δ)). Note that in the robustness verification device 200, the upper and lower limit calculation unit 206 uses the upper and lower limits of d(f(q), f(c+δ)), so even if an input is originally verified (Verified) in the definition of α-robustness verification, it may be deemed not verified (Not Verified) by the robustness verification device 200.
[0102] In the robustness verification device 200, the rank verification unit 208 may include a rank calculation unit 210, similar to the robustness verification device 100. In this case, the rank calculation unit 210 outputs "j" input to the robustness verification device 200 as is. "j" is calculated by Rank(q,IR(q,C) j , C), and the image IR(q, C) in the similarity with the input image q when the candidate image group C is not added with noise. j This is the ranking.
[0103] Next, the operation of the robustness verification device 200 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the processing procedure for the robustness verification device 200 to perform α-robustness verification.
[0104] First, the robustness verification device 200 calculates an input image qεχ, which is a query, and a set of candidate images C={c i ∈χ} (i=1 to N), a feature extractor f, a perturbation size ε, a parameter α, and a rank j are received as input (step S201).
[0105] Next, the similar image identification unit 202 identifies the jth image IR(q, C) from the candidate image group C that is similar to the input image q. j Specifically, the similar image identification unit 202 uses a feature extractor f to identify the input image q and each candidate image c i ∈C features f(q), f(c i ) (i = 1 to N), and calculate the feature value f(q) and each f(c i ) and the Euclidean distance dist(f(q), f(c iThen, the similar image identification unit 202 calculates the image having the j-th smallest distance from the input image q as IR(q, C). j (step S202).
[0106] Next, for each target image c in the candidate image group C, the upper and lower limit calculation unit 206 calculates the upper limit d of d(f(q), f(c+δ)) that satisfies the formula (15) for any δ that satisfies the formula (14). c (f(q), f(c)) and lower bound d c (f(q), f(c)) is calculated (step S203).
[0107] Next, the rank counting unit 212 of the rank verification unit 208 calculates the right side of the equation (18). That is, the rank counting unit 212 calculates the ranks IR(q, C) from the candidate image group C in the equation (18). j For all elements c except for , "1 [ d c (f(q), f(c))<_d c (f(q), f(IR(q,C) j )))]" and calculates a value by adding 1. The rank counting unit 212 also calculates the left side of equation (19). That is, the rank counting unit 212 calculates the ranks of the candidate image group C to the ranks of the candidate image group IR(q,C) in equation (19). j For all elements c except for , "1 [ d c (f(q), f(IR(q,C) j ))<_d c (f(q), f(c))]" and subtract the counted value from the number N of elements of C (step S204).
[0108] Next, the ranking verification unit 208 verifies whether the calculated value of the right side of equation (18) is equal to or greater than "j-α" and whether the calculated value of the left side of equation (19) is equal to or less than "j+α". If the condition is met, the ranking verification unit 208 outputs that α-robustness verification has been performed (Verified), and if the condition is not met, the ranking verification unit 208 outputs that α-robustness verification has not been performed (Not Verified) (step S205).
[0109] After step S205, the robustness verification device 200 ends the process in Fig. 7. Note that the robustness verification device 200 may perform α-robustness verification for multiple j or all j in the range 1≦j≦N, rather than performing α-robustness verification for only a specific rank j.
[0110] As described above, the similar image identification unit 202 identifies the similar image IR(q, C) j The upper and lower limit calculation unit 206 calculates the upper and lower limits of d(f(q), f(c+δ)) for each target image. The ranking verification unit 208 verifies the conditions of equations (18) and (19). This allows the robustness verification device 200 to perform α-robustness verification, that is, to verify the target images obtained by adding noise to the candidate image group C. ~ When C is set, the target image group ~ Image IR(q, C) when q is the input image in C j The ranking of the image IR(q, C) when the input image in the candidate image group C is q j In other words, the robustness verification device 200 can verify the degree of influence on search results when adversarial samples in which adversarial perturbations are applied to a group of candidate images are given in content-based image retrieval.
[0111] Furthermore, for each target image c in the candidate image group C, the upper and lower limit calculation unit 206 calculates the upper limit d of d(f(q), f(c+δ)) for any δ that satisfies the formula (14). c (f(q), f(c)) and lower bound d c (f(q), f(c)). Then, the robustness verification device 200 performs (α, β)-robustness verification using these upper and lower bounds. This allows the robustness verification device 200 to perform α-robustness verification with a small amount of calculation (practical calculation time).
[0112] The ranking verification unit 208 also uses a parameter α to determine the amount of fluctuation in ranking that is allowed. This allows the robustness verification device 200 to adjust the accuracy of verification, such as making the α larger to facilitate α-robustness verification.
[0113] Third Embodiment In a content-based image search using a feature extractor, the j-th image in a group of candidate images C that is similar to an input image q is designated as a similar image IR(q,C). j As described above, in the (α, β)-robustness verification and the α-robustness verification, when an input image q or a candidate image c is given a noise δ whose magnitude is equal to or smaller than a predetermined value ε, the similar image IR(q, C) to the input image is calculated. j It is difficult to calculate the change in rank among a set of candidate images.
[0114] For this reason, the robustness verification devices 100 and 200 of the first and second embodiments utilize the fact that, when the feature extraction model is defined as f, the distance is defined as d, and the magnitude of noise is defined as a predetermined value ε, for any noise δ of magnitude ε or less, the upper and lower limits of the distances d(f(q+δ), f(c)) and d(f(q), f(c+q)) in the feature space when noise is added to an image can be calculated with a small amount of calculation. The upper and lower limits here refer to the values calculated by equations (6), (7), (16), and (17). Then, using these upper and lower limits, the similar image IR(q,C) to the input image when noise δ is added to the input image q or the candidate image c is calculated. j The change in the ranking of the candidate images is calculated.
[0115] However, the success rate of robustness verification against query image attacks and candidate image attacks depends on the upper and lower limits of the distances d(f(q+δ), f(c)) and d(f(q), f(c+q)) in the feature space when noise δ is added to the image, and robustness verification can be performed more successfully when these upper and lower limits are closer to the distance d(f(q), f(c)) in the feature space when no noise is added. In other words, there is a problem in that robustness verification cannot be performed successfully if the distances d(f(q+δ), f(c)) and d(f(q), f(c+q)) are significantly different from the distance d(f(q), f(c)).
[0116] Therefore, in the following third to sixth embodiments, the image x1 , x 2 For feature extractor f, the upper limit of the distance in the feature space is d(f(x 1 ), f(x 2 )), and lower bound_d(f(x 1 ), f(x 2 )) is the distance d(f(x 1 ), f(x 2 We deal with a learning device that trains the feature extractor f so that it approaches the
[0117] <Triplet Loss> In the present invention, triplet loss is used for training the feature extractor f. Triplet loss is a learning model used in metric learning. Triplet loss is a training data set that is derived from the following equation: D = {(x a , x p , x n ) i} (i = 1 to N) is given. a is called an anchor, and x p is called a positive sample, and x n is called a negative sample. Anchor x a and the positive sample x p are data belonging to the same class, and the anchor x a and negative sample x n are data belonging to different classes. (x a , x p , x n ) is called a triplet.
[0118] Triplet loss is anchor x a Positive sample x belonging to the same class as p The distance between the pair of anchors x a and negative samples x belonging to different classes. n Specifically, the feature extractor f is trained using the training data D so as to minimize a loss function called Triplet (or Triplet Loss) expressed by Equation (20).
[0119]
[0120] In equation (20), "f(x a "), "f(x p "), "f(x n ) are x a , x p , x n represents the feature quantity obtained by the feature extractor f. a ), f(x p )) is f(x a ), f(x p ) distance, and "d(f(x a ), f(x n )) is f(x a ), f(x n ) distance. "m" is a positive real constant representing a hyperparameter for the margin, and the distance between the two distances d(f(x a ), f(x p )) and d(f(x a ), f(x n )) is separated by m. The value of the loss function is calculated by the max function: a ), f(x p ))-d(f(x a ), f(x n If the value of "))+m" is positive, it will be that value, and if it is negative, it will be 0.
[0121] Fig. 8 is a diagram showing minimization of the loss function of TripletLoss. Minimizing the loss function of TripletLoss expressed by equation (20) means training the feature extractor f to change from the left state of Fig. 8 to the right state. This is done by using the anchor x a Positive sample x belonging to the same class as p Close the distance and anchor x a and negative samples x belonging to different classes. n In the case of the present invention, the data is an image.
[0122] 9 is a schematic block diagram showing an example of the functional configuration of a learning device 300 according to the third embodiment. The learning device 300 includes a training data storage unit 302, a triplet acquisition unit 304, an upper / lower limit calculation unit 306, and a learning unit 308. The learning device 300 according to the third embodiment generates training data D={(x, x + , x - ) i} (i=1 to N) to train the feature extractor f. The training unit 308 is an example of a training means.
[0123] The training data storage unit 302 stores training data D={(x, x + , x - ) i} (i=1 to N) are stored. + , x - is an image. (x, x + , x - ) is the triplet mentioned above. For an anchor x, x + is a positive sample, which is an image that belongs to the same class as x. For an anchor x, x - are negative samples, images that belong to a different class than x.
[0124] The triplet acquisition unit 304 acquires each triplet (x, x + , x - ) and outputs it to the upper and lower limit calculation unit 306 and the learning unit 308.
[0125] The upper and lower limit calculation unit 306 receives the triplet (x, x + , x - The upper and lower limit calculation unit 306 first receives the anchor x and the positive sample x + For d(f(x), f(x) satisfying equation (21), + The upper and lower limits of (22) and (23) are calculated using equations (22) and (23), respectively.
[0126]
[0127] The upper and lower limit calculation unit 306 performs the calculation using the well-known technique Interval Bound Propagation (IBP) described in the reference cited above. The upper and lower limit calculation unit 306 uses IBP to calculate f(x). i Upper limit of f(x) i and lower bound f(x) i where i represents the i-th element of the n-dimensional vector. Using these upper and lower bounds, d(f(x), f(x + )) upper bound d(f(x), f(x + )) and lower bound _d(f(x), f(x + ) are calculated using equations (22) and (23), respectively.
[0128] In equation (22), "|  ̄ f(x) i -f(x + ) i | 1 " is the upper limit of the i-th element of the feature of image x and the + represents the absolute value of the difference between the i-th element of the feature value of + ) i -_f(x) i | 1 " is image x + The right side of equation (22) represents the square of the larger of these values, and the sum of the squares of all n-dimensional elements i. This value is expressed as d(f(x), f(x + )) upper bound d(f(x), f(x + )).
[0129] In equation (23), "f(x) i -f(x + ) i " is the upper limit of the i-th element of the feature of image x and the + represents the difference between the i-th element of the feature of "f(x + ) i -_f(x) i " is image x +represents the difference between the i-th element of the feature of image x and the lower limit of the i-th element of the feature of image x. The right-hand side of equation (23) represents the square of the smaller of these values and 0, and the sum of the squares of the values obtained over all n-dimensional elements i. This value is expressed as d(f(x),f(x + )) lower bound _d(f(x), f(x + )).
[0130] Next, the upper and lower limit calculation unit 306 calculates the anchor x and the negative sample x - For d(f(x), f(x) satisfying equation (24), - The upper and lower limits of (1) and (2) are calculated using IBP in the same way as in (22) and (23), respectively, using equations (25) and (26).
[0131]
[0132] The upper and lower limit calculation unit 306 outputs the calculated upper and lower limits to the learning unit 308 .
[0133] The training unit 308 trains the feature extractor f using triplet loss. Specifically, the training unit 308 trains the feature extractor f using triplet loss. + , x - ) and the calculated upper and lower limits, and performs training of the feature extractor f so that the loss function shown in equation (27) is minimized.
[0134]
[0135] In equation (27), "Triple (x, x + , x - )) is a term expressed by the following equation (28).
[0136] This term is the same as the function described above in equation (20), and is a term that is generally used in learning models using triplet loss. This term is used for the positive sample x that belongs to the same class as the anchor x. + and the negative sample x that belongs to a different class from the anchor x. -This term is introduced to train the feature extractor f so as to increase the distance between the normal data and the triplet (x, x). + , x - ") and can be anything.
[0137] In equation (27), "|d(f(x), f(x + ))− ̄d(f(x), f(x + )) | 1 " is f(x) and f(x + ) and the absolute value of the difference between the upper limit of that distance. Minimizing the term containing this means minimizing f(x) and f(x + This means that the feature extractor f is trained so that the distance between |d(f(x),f(x)) and the upper limit of that distance are as close as possible. + )) -_d(f(x), f(x + )) | 1 " is f(x) and f(x + ) and the absolute value of the difference between the lower limit of that distance. Minimizing the terms that include this means minimizing f(x) and f(x + This means that the feature extractor f is trained so that the distance between f and the lower limit of the distance is as close as possible. "max( , )" means that the larger of these terms is taken.
[0138] Also, "|d(f(x), f(x - ))− ̄d(f(x), f(x - )) | 1 " is f(x) and f(x - ) and the absolute value of the difference between the upper limit of that distance. Minimizing the term containing this means minimizing f(x) and f(x - This means that the feature extractor f is trained so that the distance between |d(f(x),f(x)) and the upper limit of that distance are as close as possible. - )) -_d(f(x), f(x - )) | 1 " is f(x) and f(x -) and the absolute value of the difference between the lower limit of that distance. Minimizing the terms that include this means minimizing f(x) and f(x - This means that the feature extractor f is trained so that the distance between f and the lower limit of the distance is as close as possible. "max( , )" means that the larger of these terms is taken.
[0139] "κ 2 The term {} is the sum of two "max( , )". 2 The term "{}" is the distance d(f(x), f(x + ) upper limit d(f(x), f(x + ) and lower bound _d(f(x), f(x + ) as close as possible to the distance d(f(x), f(x + ) and the distance d(f(x), f(x - ) upper limit d(f(x), f(x - ) and lower bound _d(f(x), f(x - ) as close as possible to the distance d(f(x), f(x - ) is obtained. 1 and κ 2 is a parameter for adjusting the size of the first and second items.
[0140] The learning unit 308 learns all triplets (x, x + , x - ), or some triplet (x, x + , x - ) is used to train the feature extractor f.
[0141] Next, the operation of the learning device 300 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of a processing procedure in which the learning device 300 learns a feature extractor.
[0142] The learning device 300 stores training data D={(x, x + , x - ) i} (i=1 to N) are stored. (x, x + , x- ) is the image x, x + , x - For an anchor x, x + is a positive sample, which is an image that belongs to the same class as x. For an anchor x, x - are negative samples, images that belong to a different class than x.
[0143] First, the triplet acquisition unit 304 of the learning device 300 acquires one triplet (x, x + , x - ) is acquired (step S301).
[0144] Next, the upper and lower limit calculation unit 306 calculates the anchor x and the positive sample x + For d(f(x), f(x) satisfying equation (21), + The upper and lower limits of the anchor x and negative sample x are calculated by the formulas (22) and (23), respectively. - For d(f(x), f(x) satisfying equation (24), - ) are calculated using equations (25) and (26), respectively (step S302).
[0145] Next, the learning unit 308 learns the triplet (x, x + , x - ) and the calculated upper and lower limits, and performs training of the feature extractor f so that the loss function shown in equation (27) is minimized (step S303).
[0146] Next, the learning device 300 determines whether a predetermined termination condition is met (step S304). The termination condition here is not limited to a specific one. For example, the termination condition may be that the decrease in the loss function in equation (27) is smaller than a predetermined threshold. Alternatively, the termination condition may be that the loop from S301 to S303 has been executed a predetermined number of times. Alternatively, the termination condition may be that learning has been completed for triplets that satisfy the predetermined condition, among the triplets stored in the training data storage unit 302. If the termination condition is not met, the learning device 300 transfers control to step S301. If the termination condition is met, the learning device 300 terminates the processing of FIG. 10.
[0147] As described above, the training data storage unit 302 stores training data that is triplets. The triplet acquisition unit 304 acquires triplets. The upper and lower limit calculation unit 306 calculates the upper and lower limit of the anchor x and the positive sample x. + For the distance d(f(x), f(x + ) and also calculate the upper and lower bounds of the anchor x and negative sample x - For the distance d(f(x), f(x - The learning unit 308 calculates the upper and lower bounds of the anchor x and the positive sample x. + Also, the upper and lower bounds of the distance between the anchor x and the negative sample x - The feature extractor f is trained so as to minimize a loss function that includes the upper and lower limits of the distance.
[0148] This allows the learning device 300 to train the feature extractor f so that the upper and lower limits of the distance in the feature space between images when the feature extractor f is used are as close as possible to the distance. In particular, the learning device 300 uses the anchor x and the positive sample x included in the triplet of training data. + The upper and lower limits of the distance in the feature space between the images are as close as possible to the distance, and the anchor x and negative sample x - The feature extractor f can be trained so that the upper and lower limits of the distance in the feature space between images, where f is the distance between the images, become as close as possible to the distance.
[0149] This also allows the feature extractor f to be trained so that the distances d(f(q+δ), f(c)) and d(f(q), f(c+q)) in the feature space when noise δ is added to an image are close to the distance d(f(q), f(c)) in the feature space when no noise is added. This allows for accurate (α, β)-robustness verification and α-robustness verification for content-based image retrieval.
[0150] 11 is a schematic block diagram showing an example of the functional configuration of a learning device 400 according to the fourth embodiment. The learning device 400 includes a training data storage unit 302, a triplet acquisition unit 304, an upper / lower limit calculation unit 406, and a learning unit 408. Note that the training data storage unit 302 and the triplet acquisition unit 304 of the learning device 400 have the same functions as the training data storage unit 302 and the triplet acquisition unit 304 of the learning device 300, and therefore are denoted by the same reference numerals and description thereof will be omitted.
[0151] The upper and lower limit calculation unit 406 receives the triplet (x, x + , x - The upper and lower limit calculation unit 406 first receives the anchor x and the positive sample x + For d(f(x), f(x) satisfying equation (21), + )) is calculated using equation (22). - For d(f(x), f(x) satisfying equation (24), - )) is calculated using equation (26). The upper and lower limit calculation unit 406 outputs the calculated upper and lower limits to the learning unit 408.
[0152] The training unit 408 trains the feature extractor f using triplet loss. Specifically, the training unit 408 trains the feature extractor f using triplet loss. + , x - ) and the calculated upper and lower limits, and performs training of the feature extractor f so that the loss function shown in equation (29) is minimized.
[0153]
[0154] In the formula (29), "Triple (x, x + , x - ) is the same as the term expressed in the above-mentioned equation (28). If this term is intended to improve accuracy for normal data, it can be expressed as "Triple (x, x + , x - ") and can be anything.
[0155] In the formula (29), "CertTriple (x, x + , x - ) is expressed by the following equation (30).
[0156] This term is the positive sample x that belongs to the same class as the anchor x. + The upper limit of the distance between the anchor x and the negative sample x belonging to a different class is - The feature extractor f is trained so as to move the lower limit of the distance between f(x), f(x). "m" is a positive real constant representing a hyperparameter for margin, and the upper limit d(f(x), f(x)) is + )) and lower bound _d(f(x), f(x - This means that the feature extractor f is trained so that the distances between the κ 1 and κ 2 is a parameter for adjusting the size of the first and second items.
[0157] The learning unit 408 learns all triplets (x, x + , x - ), or some triplet (x, x + , x - ) is used to train the feature extractor f.
[0158] Next, the operation of the learning device 400 will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of a processing procedure in which the learning device 400 learns a feature extractor.
[0159] The learning device 400 stores training data D={(x, x + , x - ) i} (i=1 to N) are stored. (x, x + , x - ) is the image x, x + , x - For an anchor x, x + is a positive sample, which is an image that belongs to the same class as x. For an anchor x, x - are negative samples, images that belong to a different class than x.
[0160] First, the triplet acquisition unit 304 of the learning device 400 acquires one triplet (x, x + , x - ) is acquired (step S401).
[0161] Next, the upper and lower limit calculation unit 406 calculates the anchor x and the positive sample x + For d(f(x), f(x) satisfying equation (21), + )) is calculated using equation (22). - For d(f(x), f(x) satisfying equation (24), - )) is calculated using equation (26) (step S402).
[0162] Next, the learning unit 408 learns the triplet (x, x + , x - ) and the calculated upper and lower limits, and performs training of the feature extractor f so that the loss function shown in equation (29) is minimized (step S403).
[0163] Next, the learning device 400 determines whether a predetermined termination condition is met (step S404). The termination condition here is not limited to a specific one. It may be the same condition as the termination condition described for the learning device 400. If the termination condition is not met, the learning device 400 transfers control to step S401, and if the termination condition is met, the processing of FIG. 12 ends.
[0164] As described above, the training data storage unit 302 stores training data that is triplets. The triplet acquisition unit 304 acquires triplets. The upper and lower limit calculation unit 406 calculates the upper and lower limit of the anchor x and the positive sample x. + For the distance d(f(x), f(x + )) and also calculate the upper bound of the anchor x and negative sample x - For the distance d(f(x), f(x - The learning unit 408 calculates only the lower bound of the anchor x and the positive sample x + Also, the upper bound of the distance between the anchor x and the negative sample x - The feature extractor f is trained so as to minimize the loss function that includes the lower limit of the distance.
[0165] As a result, the learning device 400 can obtain "CertTriple(x, x + , x - ) term, the positive sample x that belongs to the same class as the anchor x + The upper limit of the distance between the anchor x and the negative sample x belonging to a different class is - The feature extractor f can be trained so as to move the lower limit of the distance between the triplet (x, x) and the + , x - ) term, the positive sample x that belongs to the same class as the anchor x + and the negative sample x that belongs to a different class from the anchor x. - Considering these, the feature extractor f can be trained to increase the distance between the anchor x and the positive sample x by equation (29). + The upper limit of the distance in the feature space between the images is as close as possible to the distance, and the anchor x and negative sample x - It is considered that the feature extractor f can be trained so that the lower limit of the distance in the feature space between images, where f is the distance between the images, is as close as possible to the distance.
[0166] This also allows the feature extractor f to be trained so that the distances d(f(q+δ), f(c)) and d(f(q), f(c+q)) in the feature space when noise δ is added to an image are close to the distance d(f(q), f(c)) in the feature space when no noise is added. This allows for accurate (α, β)-robustness verification and α-robustness verification for content-based image retrieval.
[0167] 13 is a schematic block diagram showing an example of the functional configuration of a learning device 500 according to the fifth embodiment. The learning device 500 includes a training data storage unit 302, a triplet acquisition unit 304, an upper / lower limit calculation unit 506, and a learning unit 508. Note that the training data storage unit 302 and the triplet acquisition unit 304 of the learning device 500 have the same functions as the training data storage unit 302 and the triplet acquisition unit 304 of the learning device 300, and therefore are denoted by the same reference numerals and description thereof will be omitted.
[0168] The upper and lower limit calculation unit 506 receives the triplet (x, x + , x - The upper and lower limit calculation unit 506 calculates the upper limit of d(f(x), f(x)) that satisfies equation (31) for the anchor x using equation (32).
[0169]
[0170] The upper and lower limit calculation unit 506 performs the calculation using the well-known technique Interval Bound Propagation (IBP) described in the reference cited above. The upper and lower limit calculation unit 506 uses IBP to calculate f(x). i Upper limit of f(x) i and lower bound f(x) i where i represents the i-th element of the n-dimensional vector. Using these upper and lower limits, the upper limit d(f(x), f(x)) of d(f(x), f(x)) is calculated using equation (32).
[0171] In equation (32), "|  ̄ f(x) i -f(x) i | 1" represents the absolute value of the difference between the upper limit of the i-th element of the feature amount of image x and the i-th element of the feature amount of image x. i -_f(x) i | 1 " represents the absolute value of the difference between the i-th element of the feature of image x and the lower limit of the i-th element of the feature of image x. The right-hand side of equation (32) represents the square of the larger of these values, summed over all n-dimensional elements i. This value is the upper limit d(f(x), f(x)) of d(f(x), f(x)). The upper and lower limit calculation unit 506 outputs the calculated upper limit to the learning unit 508.
[0172] The training unit 508 trains the feature extractor f using triplet loss. Specifically, the training unit 508 trains the feature extractor f using triplet loss. + , x - ) and the calculated upper limit, and performs training of the feature extractor f so that the loss function shown in equation (33) is minimized.
[0173]
[0174] In the formula (33), "Triple (x, x + , x - ) is the same as the term expressed in the above-mentioned equation (28). If this term is intended to improve accuracy for normal data, it can be expressed as "Triple (x, x + , x - ") and can be anything.
[0175] In equation (33), "d(f(x), f(x))" is the term expressed by the above equation (32). This term is used to train the feature extractor f so that the upper limit of the distance of the anchor image x itself is as small as possible. 1 and κ 2 is a parameter for adjusting the size of the first and second items.
[0176] The learning unit 508 learns all triplets (x, x + , x - ), or some triplet (x, x + , x -) is used to train the feature extractor f.
[0177] Next, the operation of the learning device 500 will be described with reference to Fig. 14. Fig. 14 is a flowchart showing an example of a processing procedure in which the learning device 500 learns a feature extractor.
[0178] The learning device 500 stores training data D={(x, x + , x - ) i} (i=1 to N) are stored. (x, x + , x - ) is the image x, x + , x - For an anchor x, x + is a positive sample, which is an image that belongs to the same class as x. For an anchor x, x - are negative samples, images that belong to a different class than x.
[0179] First, the triplet acquisition unit 304 of the learning device 500 acquires one triplet (x, x + , x - ) is acquired (step S502).
[0180] Next, the upper and lower limit calculation unit 506 calculates the upper limit d(f(x), f(x)) of d(f(x), f(x)) that satisfies equation (31) for the anchor x using equation (32) (step S402).
[0181] Next, the learning unit 508 learns the triplet (x, x + , x - ) and the calculated upper limit, and performs training of the feature extractor f so that the loss function shown in equation (33) is minimized (step S503).
[0182] Next, the learning device 500 determines whether a predetermined termination condition is met (step S504). The termination condition here is not limited to a specific one. It may be the same condition as the termination condition described for the learning device 300. If the termination condition is not met, the learning device 500 transfers control to step S501, and if the termination condition is met, the processing of FIG. 14 ends.
[0183] As described above, the training data storage unit 302 stores training data that is triplets. The triplet acquisition unit 304 acquires triplets. The upper and lower limit calculation unit 506 calculates the upper limit of the distance d(f(x), f(x)) for image x. The learning unit 508 trains the feature extractor f so as to minimize a loss function that includes the upper limit of the distance of image x itself.
[0184] This allows the learning device 500 to train the feature extractor f so as to minimize the upper limit of the distance in the feature space of the image x itself. Therefore, the learning device 500 can train the feature extractor f so as to minimize the upper limit of the distance in the feature space between different images.
[0185] This also allows the feature extractor f to be trained so that the distances d(f(q+δ), f(c)) and d(f(q), f(c+q)) in the feature space when noise δ is added to an image are close to the distance d(f(q), f(c)) in the feature space when no noise is added. This allows for accurate (α, β)-robustness verification and α-robustness verification for content-based image retrieval.
[0186] Sixth Embodiment In the learning of the third to fifth embodiments, in a situation where an input image q or a group of candidate images C for content-based image retrieval is not available, training data D={(x, x + , x - ) i} (i = 1 to N) to train the feature extractor f. This was intended to improve the accuracy of verification of robustness against query image attacks and candidate image attacks. In contrast, the training of the sixth embodiment is intended to improve the accuracy of verification of robustness when the candidate image group C is used for an input image q, which is a query that will likely come in the future, in a state where a database that stores the candidate image group C is given, after the feature extractor has been trained by the training of the third to fifth embodiments.
[0187] 15 is a schematic block diagram showing an example of the functional configuration of a learning device 600 according to the sixth embodiment. The learning device 600 includes an image storage unit 602, an image acquisition unit 604, an upper and lower limit calculation unit 606, and a learning unit 608. The learning device 600 according to the sixth embodiment stores a candidate image group C={c i ∈χ} (i=1 to N) to train the feature extractor f.
[0188] The image storage unit 602 stores a candidate image group C={c i ∈χ} (i=1 to N). The candidate image group C is a group of images to be searched by the content-based image search device 900.
[0189] The image acquisition unit 604 acquires each candidate image c from the image storage unit 602 and outputs it to the upper / lower limit calculation unit 606 and the learning unit 608 .
[0190] The upper and lower limit calculation unit 606 receives the candidate image c from the image acquisition unit 604. The upper and lower limit calculation unit 606 calculates the difference between the two candidate images c. 1 , c 2 For this, the distance d(f(c 1 ), f(c 2 The upper and lower limits of (35) and (36) are calculated, respectively.
[0191]
[0192] The upper and lower bound calculation unit 606 performs the calculation using the well-known technique Interval Bound Propagation (IBP) described in the reference cited above. The upper and lower bound calculation unit 606 uses IBP to calculate f(c) i Upper limit of f(c) i and lower bound_f(c) i where i represents the i-th element of the n-dimensional vector. Using these upper and lower bounds, d(f(c 1 ), f(c 2 )) upper limit d(f(c 1 ), f(c 2 )) and lower bound d(f(c 1 ), f(c 2 ) are calculated using equations (35) and (36), respectively.
[0193] In equation (35), "|  ̄f(c 1 ) i -f(c 2 ) i | 1 " is image c 1 and the upper limit of the i-th element of the feature of image c 2 represents the absolute value of the difference between the i-th element of the feature 2 ) i -_f(c 1 ) i | 1 " is image x + The right side of equation (35) represents the square of the larger of these values, and the sum of the squares of all n-dimensional elements i. This value is d(f(c 1 ), f(c 2 )) upper limit d(f(c 1 ), f(c 2 )).
[0194] In equation (36), "f(c 1 ) i -f(c 2 ) i " is image c 1 and the upper limit of the i-th element of the feature of image c 2 represents the difference between the i-th element of the feature of 2 ) i -_f(c 1 ) i " is image c 2 and the i-th element of the feature of image c 1 The right side of equation (36) represents the square of the smaller of these values and 0, and the sum of the squares of the values obtained by summing over all n-dimensional elements i. This value is d(f(c 1 ), f(c 2 )) lower limit _d(f(c 1 ), f(c 2 )).
[0195] The upper and lower limit calculation unit 606 outputs the calculated upper and lower limits to the learning unit 608 .
[0196] The learning unit 608 uses a loss function to train the feature extractor f. Specifically, the learning unit 608 1 , c 2 , the calculated upper and lower bounds, and the previously trained feature extractor f 0 and trains the feature extractor f so that the loss function shown in equation (37) is minimized.
[0197]
[0198] In equation (37), "|d(f(c 1 ), f(c 2 ))- ̄d(f(c 1 ), f(c 2 )) | 1 " is f(c 1 ) and f(c 2 ) and the absolute value of the difference between the upper limit of that distance. 1 ) and f(c 2 ) and the upper limit of that distance are made as close as possible. 1 ), f(c 2 ))-_d(f(c 1 ), f(c 2 )) | 1 " is f(c 1 ) and f(c 2 ) and the absolute value of the difference between the lower limit of that distance. 1 ) and f(c 2 This means that the feature extractor f is trained so that the distance between f and the lower limit of the distance is as close as possible. "max( , )" means that the larger of these terms is taken.
[0199] On the other hand, "d(f 0 (c 1 ), f(c 1 )) in "f 0 ( ) represents the feature extractor before updating (the one trained immediately before). "f ( )" represents the feature extractor to be trained this time. "d(f 0 (c1 ), f(c 1 ))) is included in the pre-update f 0 After updating, image c is 1 This is because the accuracy of f for normal data will be reduced if only the second term is constrained, so this term is added to maintain accuracy for normal data. 0 Learning f() using ( ) is called fine-tuning. 1 and κ 2 is a parameter for adjusting the size of the first and second items.
[0200] The learning unit 608 uses all or some of the candidate images stored in the image storage unit 602 to train the feature extractor f.
[0201] Next, the operation of the learning device 600 will be described with reference to Fig. 16. Fig. 16 is a flowchart showing an example of a processing procedure in which the learning device 600 learns a feature extractor. The learning device 600 stores a candidate image group C={c i ∈χ} (i=1 to N).
[0202] First, the image acquisition unit 604 acquires each candidate image c from the image storage unit 602 (step S601).
[0203] Next, the upper and lower limit calculation unit 606 calculates the upper and lower limit of the two candidate images c 1 , c 2 For d(f(c 1 ), f(c 2 )) upper limit d(f(c 1 ), f(c 2 )) and lower bound d(f(c 1 ), f(c 2 ) are calculated using equations (35) and (36), respectively (step S602).
[0204] Next, the learning unit 608 selects the candidate image c 1 , c 2 , the calculated upper and lower bounds, and the previously trained feature extractor f 0and trains the feature extractor f so that the loss function shown in equation (37) is minimized (step S603).
[0205] Next, the learning device 600 determines whether a predetermined termination condition is met (step S604). The termination condition here is not limited to a specific one. For example, the termination condition may be that the decrease in the loss function in equation (37) is smaller than a predetermined threshold. Alternatively, the termination condition may be that the loop from S601 to S603 has been executed a predetermined number of times. Alternatively, the termination condition may be that learning has been completed for candidate images c stored in the image storage unit 602 that satisfy the predetermined condition. If the termination condition is not met, the learning device 600 transfers control to step S601. If the termination condition is met, the learning device 600 terminates the processing of FIG. 15.
[0206] As described above, the image storage unit 602 stores the candidate image group C. The image acquisition unit 604 acquires candidate images. The upper and lower limit calculation unit 606 calculates the upper and lower limit of the two candidate images C. 1 , c 2 For d(f(c 1 ), f(c 2 )) upper limit d(f(c 1 ), f(c 2 )) and lower bound d(f(c 1 ), f(c 2 The learning unit 608 calculates the two candidate images c 1 , c 2 and the upper and lower bounds of the distance of the candidate image c 1 The feature extractor f is trained so as to minimize a loss function including the distance between the feature obtained by the feature extractor f before the update and the feature obtained by the feature extractor f after the update.
[0207] This allows the learning device 600 to train the feature extractor f so that the upper and lower limits of the distance in the feature space between candidate images when using the feature extractor f are as close as possible to the distance in question. Furthermore, the learning device 600 can maintain the accuracy of the features for the candidate images before and after updating the feature extractor.
[0208] This also allows the feature extractor f to be trained so that the distances d(f(q+δ), f(c)) and d(f(q), f(c+q)) in the feature space when noise δ is added to an image are close to the distance d(f(q), f(c)) in the feature space when no noise is added. This allows for accurate (α, β)-robustness verification and α-robustness verification for content-based image retrieval.
[0209] 17 is a diagram showing an example of the configuration of a learning device according to Embodiment 7. In the configuration shown in FIG.
[0210] With this configuration, the learning unit 811 learns the feature extractor f so that the upper and lower limits of the distance in the feature space between images when the feature extractor f is used become close to the distance. The learning unit 811 is an example of a learning means.
[0211] The learning device 810 can train the feature extractor so that the upper and lower limits of the distance in feature space between images when the feature extractor is used are as close as possible to the distance. That is, the learning device 810 can train the feature extractor f so that the distances d(f(q+δ), f(c)) and d(f(q), f(c+q)) in feature space when noise δ is added to an image are close to the distance d(f(q), f(c)) in feature space when no noise is added. This allows for accurate verification of robustness in content-based image retrieval.
[0212] Eighth Embodiment Fig. 18 is a diagram showing an example of a processing procedure in a learning method according to an eighth embodiment. The learning method shown in Fig. 18 includes training a feature extractor (step S811).
[0213] In training the feature extractor (step S811), the feature extractor is trained so that the upper and lower limits of the distance in the feature space between images when using the feature extractor f become close to the distance itself.
[0214] 18 , the feature extractor can be trained so that the upper and lower limits of the distance in the feature space between images when the feature extractor is used are as close as possible to the distance. That is, the training method shown in FIG. 18 allows the feature extractor f to be trained so that the distances d(f(q+δ), f(c)) and d(f(q), f(c+q)) in the feature space when noise δ is added to an image are close to the distance d(f(q), f(c)) in the feature space when no noise is added. This allows for accurate verification of robustness in content-based image retrieval.
[0215] FIG. 19 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. In the configuration illustrated in FIG. 19 , the computer 700 includes a CPU (Central Processing Unit) 710, a main memory device 720, an auxiliary memory device 730, and an interface 740. One or more of the robustness verification devices 100, 200, and the learning devices 300, 400, 500, and 600 may be implemented in the computer 700. In this case, the operation of each of the above-described processing units is stored in the auxiliary memory device 730 in the form of a program. The CPU 710 reads the program from the auxiliary memory device 730, loads it into the main memory device 720, and executes the above-described processing in accordance with the program. The CPU 710 also allocates storage areas in the main memory device 720 corresponding to each of the above-described storage units in accordance with the program. Communication between each device and other devices is achieved by the interface 740, which has a communication function, and communicating under the control of the CPU 710.
[0216] When the robustness verification device 100 is implemented in a computer 700, the operations of the similar image identification unit 102, the comparison image calculation unit 104, the upper / lower limit calculation unit 106, and the ranking verification unit 108 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing in accordance with the program. The output of the (α, β)-robustness verification of the robustness verification device 100 is executed by an interface 740 having an output function such as a communication function or a display function, which performs output processing under the control of the CPU 710.
[0217] When the robustness verification device 200 is implemented in a computer 700, the operations of the similar image identification unit 202, the upper / lower limit calculation unit 206, and the ranking verification unit 208 are stored in the form of a program in an auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing in accordance with the program. The output of the α-robustness verification of the robustness verification device 200 is executed by an interface 740 having an output function such as a communication function or a display function, which performs output processing under the control of the CPU 710.
[0218] When the learning device 300 is implemented in a computer 700, the operations of the triplet acquisition unit 304, the upper / lower limit calculation unit 306, and the learning unit 308 are stored in the form of a program in the auxiliary storage device 730. The training data in the training data storage unit 302 is also stored in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing in accordance with the program. The learning unit 308 of the learning device 300 may output the feature extractor f or its parameters, and the interface 740 has an output function such as a communication function or a display function, and performs output processing under the control of the CPU 710.
[0219] When the learning device 400 is implemented in a computer 700, the operations of the triplet acquisition unit 304, the upper / lower limit calculation unit 406, and the learning unit 408 are stored in the form of a program in the auxiliary storage device 730. The training data in the training data storage unit 302 is also stored in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing in accordance with the program. The learning unit 408 of the learning device 400 may output the feature extractor f or its parameters, and the interface 740 has an output function such as a communication function or a display function, and performs output processing under the control of the CPU 710.
[0220] When the learning device 500 is implemented in a computer 700, the operations of the triplet acquisition unit 304, the upper / lower limit calculation unit 506, and the learning unit 508 are stored in the form of a program in the auxiliary storage device 730. The training data in the training data storage unit 302 is also stored in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing in accordance with the program. The learning unit 508 of the learning device 500 may output the feature extractor f or its parameters, and the interface 740 has an output function such as a communication function or a display function, and performs output processing under the control of the CPU 710.
[0221] When the learning device 600 is implemented in a computer 700, the operations of the image acquisition unit 604, upper / lower limit calculation unit 606, and learning unit 608 are stored in the form of a program in the auxiliary storage device 730. Furthermore, the candidate images in the image storage unit 302 are stored in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program. The learning unit 608 of the learning device 600 may output the feature extractor f or its parameters, and the interface 740 has an output function such as a communication function or a display function, and performs output processing under the control of the CPU 710.
[0222] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like are also included within the scope that does not deviate from the gist of the present invention.
[0223] The embodiments of the present invention may be applied to a robustness verification device, a robustness verification method, a learning device, a learning method, a program, and a recording medium.
[0224] 100, 200 Robustness verification device 102, 202 Similar image identification unit 104 Comparison image calculation unit 106, 206 Upper / lower limit calculation unit 108, 208 Rank verification unit 110 (210) Rank calculation unit 112, 212 Rank counting unit 300, 400, 500, 600 Learning device 302 Training data storage unit 304 Triplet acquisition unit 306, 406, 506, 606 Upper / lower limit calculation unit 308, 408, 508, 608 Learning unit 602 Image storage unit 604 Image acquisition unit 900 Content-based image retrieval device 902 Image storage unit 904 Feature extraction unit 906 Rank calculation unit
Claims
1. A learning means for learning the feature extractor so that the upper and lower limits of the distance in the feature space between images when the feature extractor is used become close to the distance. A learning device comprising:
2. The learning means trains the feature extractor by minimizing a loss function. The learning device according to claim 1 .
3. The learning means performs learning of the feature extractor by minimizing the loss function including a term representing a difference between the distance in a feature space between the images and an upper limit of the distance, and a term representing a difference between the distance and a lower limit of the distance. The learning device according to claim 2 .
4. The learning means trains the feature extractor by minimizing the loss function including a term that represents an upper limit of the distance in a feature space between identical images. The learning device according to claim 2 .
5. The learning means uses triplet loss to minimize the loss function between the anchor image and the positive sample image, and between the anchor image and the negative sample image, the loss function including a term representing a difference between the distance in a feature space between the images and an upper limit of the distance, and a term representing a difference between the distance and a lower limit of the distance, thereby learning the feature extractor. The learning device according to claim 2 .
6. The learning means performs learning of the feature extractor by minimizing the loss function using triplet loss so as to bring the upper limit of the distance between the anchor image and the positive sample image closer and to move the lower limit of the distance between the anchor image and the negative sample image further away. The learning device according to claim 2 .
7. The image is an image included in a set of candidate images used in content-based image retrieval. The learning device according to claim 3 .
8. The learning means performs learning of the feature extractor by minimizing the loss function including a term representing a distance between a feature of an image obtained by using a feature extractor before updating and a feature of the image obtained by using a feature extractor to be updated. The learning device according to claim 7.
9. A step of training the feature extractor so that the upper and lower limits of the distance in the feature space between images when the feature extractor is used are close to the distance. Learning methods including:
10. On the computer, A step of training the feature extractor so that the upper and lower limits of the distance in the feature space between images when the feature extractor is used are close to the distance. A program for executing.
