Data selection method and device in noisy image retrieval scene
By introducing a data selection network into image retrieval and automatically adjusting sample weights, the problem of noise label influence is solved, improving the accuracy and robustness of the model and achieving efficient image retrieval in noisy environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XI AN JIAOTONG UNIV
- Filing Date
- 2024-07-31
- Publication Date
- 2026-05-29
AI Technical Summary
In the presence of noisy labels, existing image retrieval models cannot effectively learn the correct features, resulting in decreased performance. Furthermore, they rely on pre-defined weight functions and lack prior knowledge.
By establishing a data selection network, the sample categories are automatically determined and assigned different weights. By utilizing the feature extraction loss function and the iterative training process of the data selection network, the weight of noisy samples is reduced, the weight of clean samples is increased, and the impact of noisy labels on model training is reduced.
It improves the accuracy and robustness of image retrieval, reduces reliance on prior knowledge, increases retrieval efficiency, and makes the quality of retrieval results quantifiable.
Smart Images
Figure CN118861348B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image retrieval technology, and in particular to a data selection method and apparatus for noisy image retrieval scenarios. Background Technology
[0002] Content-Based Image Retrieval (CBIR) technology, as an important branch of information retrieval, aims to search for images that match user needs by analyzing the visual content in image libraries. CBIR technology has a wide range of applications, including pedestrian re-identification, remote sensing, medical image search, and online marketplace shopping recommendations, highlighting its crucial importance.
[0003] In category-level image retrieval, semantic labels are typically obtained through web searches or manual annotation. However, due to the subjectivity and uncertainty inherent in the annotation process, some incorrect labels, i.e., noise labels, are inevitably generated. These noise labels can mislead the model during training, causing it to fail to learn the correct feature extraction network, thereby affecting the performance of the deep learning model.
[0004] Existing technologies typically require manually pre-defined weighting functions, which may require additional information, such as distribution assumptions about the training samples or class confidence levels. However, in real-world scenarios, this prior knowledge or additional information is often lacking.
[0005] Therefore, how to filter sample categories without relying on a preset weighting function in the presence of noisy labels and effectively improve the accuracy of image retrieval has become a problem to be solved. Summary of the Invention
[0006] In this embodiment of the application, a data selection method for noisy image retrieval scenarios is provided, which solves the problem of how to filter sample categories without relying on a preset weight function form when there are noisy labels, and effectively improves the accuracy of image retrieval.
[0007] In a first aspect, embodiments of this application provide a data selection method for noisy image retrieval scenarios, characterized by comprising: establishing a dataset; wherein the dataset includes a training dataset, a meta-dataset, and a retrieval database, the training dataset including noise labels; establishing a feature extraction network to extract key features of input image samples, initially training the feature extraction network using the training dataset, and calculating the loss value of each sample during the training process using a feature extraction loss function; wherein, calculating the loss value of each sample during the training process using the feature extraction loss function includes: according to the formula... Calculate the feature extraction loss function for each sample during training; where, The feature extraction loss function is... The feature vector of the input sample is extracted from the input sample by the feature extraction network. This is a proxy vector that is closely related to the true class label y of the sample currently being processed. For other categories whose proxy vectors do not belong to the true category label y of the currently being processed sample, Here, is a temperature coefficient used to control the smoothness of the output probability distribution; C is the set of all possible categories; and c is an index variable. The calculated loss value is used as the input to a pre-established data selection network. The data selection network outputs a weight value for each sample based on the loss value, and determines the sample category based on the weight value. The sample category is either a noisy sample or a clean sample. The weight values output by the data selection network are used to weight the training dataset during subsequent training of the feature extraction network. This includes iteratively executing iterative training steps until a stopping condition is met. The stopping condition is reaching a preset maximum number of iterations or the loss value decreasing to less than a preset threshold. The iterative training steps include randomly selecting a small batch of samples from the training dataset, calculating the loss value through forward propagation, and updating the feature extraction network and feature extraction loss function using a stochastic gradient descent algorithm. A small batch of samples is randomly selected from the dataset. Using these selected samples, a feature extraction network is used to extract key features of the samples, and these features are used as input to a data selection network. The loss value of the data selection network is calculated, and the parameters of the data selection network are updated using the stochastic gradient descent algorithm. Using the updated data selection network parameters, the weight value of each sample in the training dataset is recalculated. The feature extraction network is then trained again using the recalculated weight values. The weighting process reduces the weight of noisy samples and increases the weight of clean samples. The trained feature extraction network is used to extract features from the query image and each image in the retrieval database, and the similarity between each query image and all images in the dataset is calculated. The images are sorted in descending order of similarity, and a predetermined number of images are output as retrieval results. The recall rate is used to evaluate the accuracy of the retrieval results.
[0008] In one possible implementation, before establishing the feature extraction network to extract key features of the input image samples, the following steps are included: pre-training the feature extraction network using a pre-trained deep convolutional neural network, and fine-tuning the parameters of the feature extraction network on the training dataset to adapt it to the feature representation of the noisy image.
[0009] In one possible implementation, the steps to build the data selection network are as follows: a data selection network with a multilayer perceptron structure is constructed on the metadata dataset, and the data selection network includes a hidden layer.
[0010] In one possible implementation, the step of using the weight values output by the data selection network to weight the training dataset during subsequent training of the feature extraction network further includes: when there are no noisy labels, minimizing the expected loss function on the training dataset to obtain the optimal network parameters of the feature extraction network; when there are noisy labels, constructing the data selection network to assign a weight to the loss of each training sample, minimizing the weighted loss function on the training dataset to obtain the optimal network parameters of the feature extraction network, and minimizing the meta-loss function on the meta-dataset to obtain the optimal network parameters of the data selection network.
[0011] In one possible implementation, the expression for minimizing the expected loss function is: ;in, To obtain the optimal network parameters for the feature extraction network, Let N be the operator whose parameters minimize the expected loss function, and N be the total number of samples in the training dataset. For loss function, Let w be the true label of the i-th sample, and w be the network parameters. For the feature extraction network on the i-th sample The actual output; the expression for minimizing the weighted loss function is: ;in, This represents the loss value of the feature extraction network on training sample i. Choose the weight function given by the network for the data, and accept the loss of the feature extraction network on sample i. And data selection network parameters The weighting function assigns a weight to the loss value of each sample to mitigate the influence of noise labels; the expression for the minimum loss function is: ;in, To select the optimal network parameters for the data, M is the total number of samples in the metadata set. , For loss function, For the clean label of the i-th sample in the metadata set, Use optimal parameters for the feature extraction network Input data for the i-th sample in the metadata set The predictions made Selecting network parameters based on data The optimal parameters of the calculated feature extraction network.
[0012] In one possible implementation, evaluating the accuracy of the retrieval results using recall includes: searching the test set for the k images most similar to the query image; if at least one of these k images belongs to the same category as the query image, a successful retrieval is recorded; and the accuracy of the retrieval results is evaluated using the average number of successful retrievals for all query images.
[0013] In one possible implementation, the recall rate is used to evaluate the accuracy of the search results, and the calculation formula is as follows: ;in, This represents the percentage of successful retrievals of the k most similar images to the query image in the test set, where n is the total number of queries. This means that for the *a*-th query, if at least one of the *k* returned results belongs to the same category as the query image, then... ,otherwise, .
[0014] Secondly, embodiments of this application provide a data selection device for noisy image retrieval scenarios. The device includes: a setup module for setting up a dataset; wherein the dataset includes a training dataset, a meta-dataset, and a retrieval database, and the training dataset includes noise labels; and an extraction module for setting up a feature extraction network to extract key features from input image samples, initially training the feature extraction network using the training dataset, and calculating the loss value of each sample during training using a feature extraction loss function; wherein calculating the loss value of each sample during training using the feature extraction loss function includes: according to the formula... Calculate the feature extraction loss function for each sample during training; where, The feature extraction loss function is... The feature vector of the input sample is extracted from the input sample by the feature extraction network. This is a proxy vector that is closely related to the true class label y of the sample currently being processed. For other categories whose proxy vectors do not belong to the true category label y of the currently being processed sample, The temperature coefficient is used to control the smoothness of the output probability distribution, C is the set of all possible categories, and c is the index variable; the judgment module is used to use the calculated loss value as the input of the pre-established data selection network. The data selection network outputs a weight value for each sample based on the loss value, and judges the sample category based on the weight value; wherein, the sample category is a noise sample or a clean sample; the processing module is used to use the weight value output by the data selection network to perform weighted processing on the training dataset during the subsequent training of the feature extraction network, including: iteratively executing iterative training steps until a stopping condition is met; the stopping condition is reaching a preset maximum number of iterations or the loss value decreasing to less than a preset threshold; the iterative training steps include: randomly selecting a small batch of samples from the training dataset, calculating the loss value through forward propagation, and updating the feature extraction network and feature extraction loss function using the stochastic gradient descent algorithm; from the meta-dataset A small batch of samples is randomly selected from the dataset. Using these selected samples, a feature extraction network extracts key features of the samples, and these features are used as input to a data selection network. The loss value of the data selection network is calculated, and its parameters are updated using a stochastic gradient descent algorithm. Using the updated data selection network parameters, the weight value of each sample in the training dataset is recalculated. The recalculated weight values are then used to continue training the feature extraction network. The weighting process reduces the weight of noisy samples and increases the weight of clean samples. A calculation module is used to extract features from the query image and each image in the retrieval database using the trained feature extraction network, and calculate the similarity between each query image and all images in the dataset. A sorting module is used to sort the images in descending order of similarity and output a preset number of images as retrieval results. An evaluation module is used to evaluate the accuracy of the retrieval results using recall.
[0015] One or more technical solutions provided in the embodiments of this application have at least the following technical effects:
[0016] This application provides a data selection method for noisy image retrieval scenarios. By introducing a data selection network, it can automatically determine whether a sample is a noisy or clean sample based on its loss value during training, and assign different weights to different samples. This adaptive weight adjustment mechanism can reduce the impact of noise labels on model training, allowing the model to focus more on learning the features of clean samples, thereby improving the accuracy of image retrieval. The data selection network automatically adjusts sample weights based on the loss value without relying on any prior knowledge or additional information, improving universality and practicality. Because this application can automatically identify and reduce the weight of noisy samples, the model focuses more on learning the features of clean samples during training, thus the model has higher tolerance for noise and stronger robustness. This allows the model to maintain good performance under different noise environments and datasets. By utilizing a trained feature extraction network to extract features and calculate similarity between the query image and images in the retrieval database, it can quickly find the image most similar to the query image and sort them in descending order of similarity, improving retrieval efficiency. Using recall as an indicator to evaluate the accuracy of search results allows for quantifiable assessment of the quality of search results, facilitating intuitive comparison and analysis by users. This application improves the accuracy and robustness of image retrieval by introducing a data selection network to adaptively adjust sample weights, reducing reliance on prior knowledge and additional information, increasing retrieval efficiency, and enabling quantifiable assessment of search result quality. These technical effects make this technology widely applicable and valuable in real-world scenarios. It solves the problem of how to effectively improve the accuracy of image retrieval without relying on a pre-defined weighting function when noisy labels are present. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 A flowchart illustrating a data selection method for noisy image retrieval scenarios provided in this application embodiment;
[0019] Figure 2 A detailed flowchart of the iterative training steps provided in the embodiments of this application;
[0020] Figure 3 The present application provides a method for calculating recall in a CUB dataset containing 20% symmetric noise. A diagram illustrating the comparison results;
[0021] Figure 4 The present application provides a method for calculating recall in a CUB dataset containing 20% symmetric noise. A diagram illustrating the comparison results;
[0022] Figure 5 The present application provides a method for calculating recall in a CUB dataset containing 20% symmetric noise. A diagram illustrating the comparison results;
[0023] Figure 6 A schematic diagram of sample weight distribution on a CUB dataset containing 20% symmetric noise, provided for an embodiment of this application;
[0024] Figure 7 The present application provides a method for calculating recall rates on the Stanford Dogs dataset containing 40% symmetric noise, as provided in the embodiments of this application. A diagram illustrating the comparison results;
[0025] Figure 8 The present application provides a method for calculating recall rates on the Stanford Dogs dataset containing 40% symmetric noise, as provided in the embodiments of this application. A diagram illustrating the comparison results;
[0026] Figure 9 The present application provides a method for calculating recall rates on the Stanford Dogs dataset containing 40% symmetric noise, as provided in the embodiments of this application. A diagram illustrating the comparison results;
[0027] Figure 10 A schematic diagram of sample weight distribution on the Standford Dogs dataset containing 40% symmetric noise, provided for embodiments of this application;
[0028] Figure 11 A schematic diagram of a data selection device for a noisy image retrieval scenario provided in the application embodiment;
[0029] Figure 12 This is a schematic diagram of a data selection server for a noisy image retrieval scenario provided in an embodiment of this application. Detailed Implementation
[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0031] The following description of some technologies involved in the embodiments of this application is provided to aid understanding and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, some descriptions of well-known functions and structures are omitted in the following description.
[0032] This application provides a data selection method for noisy image retrieval scenarios, such as... Figure 1 As shown, the method includes steps S101 to S107. Wherein, Figure 1 This is merely one execution order shown in the embodiments of this application, and does not represent the only execution order for a data selection method in a noisy image retrieval scenario. The execution order can be adjusted to achieve the desired final result. Figure 1 The steps shown can be performed in parallel or in reverse order.
[0033] S101: Create the dataset. The dataset includes a training dataset, a meta-dataset, and a retrieval database. The training dataset includes noise labels.
[0034] Specifically, the CUB (Caltech-UCSD Birds-200-2011) dataset can be downloaded from the official website or public datasets. This dataset is used as the constructed dataset. The first 100 classes are selected as the training dataset. Ten images are randomly selected from each class in the training dataset as the meta-dataset. This ensures that the meta-dataset is representative of all classes in the training dataset, but the sample size is much smaller than the training dataset. The last 100 classes (test set) are used as the query and retrieval database. These classes are not visible during training and are used to evaluate the performance of image retrieval. The training dataset is defined as follows: .in, The training dataset consists of N pairs of data points. The training dataset is defined as representing each image in the training dataset. Assign it the appropriate label .Label Typically, these are the semantic labels of the image. Symmetrical noise is used to introduce noisy labels into the training dataset, that is, the labels of a subset of samples are uniformly shuffled across all class labels. For example, for a 20% noise ratio, 20% of the samples in the training dataset are randomly selected, and their labels are replaced with randomly selected class labels. Similar operations are performed for 40% and 60% noise ratios, but the proportion of replaced samples increases accordingly. In image retrieval, these samples are typically images, while the labels describe the semantic content of the images. The meta-dataset consists of clean samples selected from the training dataset, and is defined as follows: ;in, The metadata consists of M pairs of data points. For the i-th sample in the metadata set The clean labels in the meta dataset, since the sample size is much smaller than that in the training dataset, can be used as a guide for the feature extraction network, helping the network learn the correct semantic information between sample pairs.
[0035] S102: Establish a feature extraction network to extract key features from input image samples, use the training dataset to initially train the feature extraction network, and use the feature extraction loss function to calculate the loss value of each sample during the training process.
[0036] Before building a feature extraction network to extract key features from input image samples, the following steps are taken: pre-training the feature extraction network using a pre-trained deep convolutional neural network, and fine-tuning the parameters of the feature extraction network on the training dataset to adapt it to the feature representation of noisy images.
[0037] Specifically, the feature extraction network is built on a ResNet (deep convolutional neural network) model pre-trained on ImageNet (an image database), and the expression for the feature extraction network is defined as: Where x is the training data input into the feature extraction network, For the parameter extraction network, the parameter w is set to the weights of the pre-trained ResNet, and then these parameters are fine-tuned on the training dataset to adapt to the feature representation of the noisy image.
[0038] Furthermore, the loss value for each sample during training is calculated using the feature extraction loss function, including: according to the formula Calculate the feature extraction loss function for each sample during training. Wherein, The feature extraction loss function is... The feature vector of the input sample is extracted from the input sample by the feature extraction network. This is a proxy vector that is closely related to the true class label y of the sample currently being processed. For other categories whose proxy vectors do not belong to the true category label y of the currently being processed sample, Here, is the temperature coefficient, used to control the smoothness of the output probability distribution. C is the set of all possible classes, and c is the index variable. When the temperature coefficient is close to 0, the probability distribution output by the loss function becomes sharper, meaning the model's prediction of the sample class is more certain. When the temperature coefficient is much greater than 1, the probability distribution output by the loss function becomes smoother, and the model's prediction of the sample class becomes more uncertain.
[0039] Specifically, the feature extraction loss function is designed to maximize the dot product between the input sample and the true class proxy vector (i.e., getting closer), while minimizing the dot product with proxy vectors of other classes (i.e., getting farther apart). When noisy labels exist—that is, some samples are mislabeled—the model receives incorrect supervision information during training. In this case, samples that should be considered positive proxies might be incorrectly treated as negative proxies, or vice versa. This erroneous supervision information misleads the model's training process, preventing it from learning the correct feature extraction network and thus affecting its performance. Therefore, ensuring the quality of the training data labels is crucial when using a feature extraction loss function.
[0040] S103: The calculated loss value is used as the input to a pre-established data selection network. The data selection network outputs a weight value for each sample based on the loss value, and determines the sample category based on the weight value. The sample category is either a noisy sample or a clean sample.
[0041] Specifically, the method for determining the sample category based on the weight value can be to sort the weight values and select samples with high weight values as clean samples and samples with low weight values as noise samples.
[0042] The steps to build a data selection network are as follows: Construct a data selection network with a multilayer perceptron structure on the metadata dataset. The data selection network includes a hidden layer.
[0043] Specifically, the expression for the feature extraction network is defined as: The data selection network is a multilayer perceptron (MLP) that receives the loss value from the feature extraction network as input and outputs the weight value for each sample. The specific structure of the feature extraction network is as follows: The input layer of the feature extraction network receives... As input, where , For loss function, Let w be the true label of the i-th sample, and w be the network parameters. For the feature extraction network on the i-th sample The actual output of the feature extraction network. The hidden layer contains 100 nodes, each using ReLU (Rectified Linear Unit) as the activation function. The ReLU function accelerates the training process and prevents the gradient vanishing problem. The output layer of the feature extraction network uses the Sigmoid activation function, restricting the output values to the [0,1] interval. The weights output by the data selection network are equivalent to selecting samples, choosing clean positive and negative samples, while ensuring that no information from any sample is discarded. Since the multilayer perceptron network can theoretically be regarded as a universal approximator for almost any continuous function, it has extremely high flexibility. Before training begins, the parameters of the data selection network need to be set. Perform initialization. Common initialization methods include random initialization (such as normal or uniform distribution) and pre-training initialization (if available).
[0044] S104: Use data to select the weight values of the network output, and perform weighting on the training dataset during subsequent training of the feature extraction network. Specifically, the weighting process reduces the weight of noisy samples and increases the weight of clean samples.
[0045] Specifically, the weight values of the network output are selected using data, and the training dataset is weighted during subsequent training of the feature extraction network, including: performing iterative training steps. The iterative training steps are performed until a stopping condition is met. The stopping condition is reaching a preset maximum number of iterations or the loss value decreasing to less than a preset threshold.
[0046] Specifically, a pre-training process of 3 epochs is required before performing the iterative training steps to stabilize the initial state of the model. Then, the parameters of the feature extraction network, each loss function, and the data selection network are jointly updated. The feature extraction network uses stochastic gradient descent (SGD), with the SGD momentum set to 0.9, which helps accelerate the training process and reduce oscillations near the optimal solution. The weight decay coefficient is set to... The initial learning rate is set to 0.001. The choice of the initial learning rate is crucial for model training. If the learning rate is too large, it may lead to training instability or divergence; if the learning rate is too small, it may lead to slow training or failure to converge to a good solution. Training is conducted for a total of 50 epochs, using a stepped decay strategy, decreasing the learning rate every 5 epochs with a decay coefficient of 0.5. This helps the model finely adjust parameters in the later stages of training. The data selection network will use the Adam (Adaptive Moment Estimation) optimization algorithm. The weight decay coefficient is used to prevent overfitting and is usually set to a small value, such as 5e-4. The initial learning rate is usually set to a small value, such as 1e-5, to ensure the stability of the training process. Specifically, the above parameter settings can be other than those provided here; they are merely examples and this application is not limited to the values mentioned above.
[0047] Figure 2 A detailed flowchart of the iterative training steps provided in the embodiments of this application is shown below. Figure 2 As shown, it includes steps S201 to S205.
[0048] S201: Randomly select a small batch of samples from the training dataset, calculate the loss value through forward propagation, and use the stochastic gradient descent algorithm to update the feature extraction network and the feature extraction loss function.
[0049] Specifically, in deep learning, since the entire training dataset is usually very large, directly calculating the gradient of the entire dataset would be extremely time-consuming and computationally intensive. Therefore, a mini-batch gradient descent method is commonly used, where only a small batch of samples (e.g., 32, 64, 128, etc.) is randomly selected in each iteration to calculate the gradient and update the model parameters. This accelerates the training process, and the introduction of randomness helps the model escape local optima. After selecting the mini-batch of samples, these samples are input into the feature extraction network, and the output of each layer of the network is calculated through forward propagation until the final prediction result is obtained. Then, the loss value is calculated based on the prediction result and the true label. Based on the calculated loss value, the parameters of the feature extraction network are updated using stochastic gradient descent (SGD) or its variants (such as momentum-based SGD, Adam, etc.). The SGD algorithm calculates the gradient of the loss function with respect to the model parameters and updates the parameters in the opposite direction of the gradient, hoping to reduce the loss value. At the same time, the parameters of the feature extraction loss function associated with the feature extraction network also need to be updated.
[0050] S202: Randomly select a small batch of samples from the metadata set, use the selected small batch of samples to extract the key features of the samples through the feature extraction network, and use these features as the input of the data selection network.
[0051] Clean samples in the metadata set are very useful for selecting high-quality training samples or performing data augmentation. The feature extraction network (MEN) is a pre-trained network that extracts meaningful feature representations from the input data. A small batch of samples selected from the metadata set is input into the MEN, and the network's forward propagation process calculates the key features of these samples. The feature selection network (FSN) is used to determine which samples are more valuable for the training process. It judges the importance or quality of samples based on the input features and selects appropriate samples for training accordingly. The key features extracted from the MEN are used as input to the FSN. The FSN processes and analyzes these features, selecting a subset of samples for subsequent model training.
[0052] S203: Calculate the loss value of the data selection network and update the parameters of the data selection network using the stochastic gradient descent algorithm.
[0053] Specifically, to optimize the performance of the data selection network, the stochastic gradient descent (SGD) algorithm is used to update its parameters in order to minimize its loss value.
[0054] S204: Select network parameters using the updated data and recalculate the weight value of each sample in the training dataset.
[0055] Specifically, these new weight values reflect the data selection network's new assessment of the importance of each sample under the updated parameters.
[0056] S205: Continue training the feature extraction network using the recalculated weight values.
[0057] Specifically, these new weight values reflect the data selection network's new assessment of the importance of each sample under the updated parameters. These weight values will be used to weight the training samples in the next iteration of training.
[0058] Specifically, the weight values output by the data selection network are used to weight the training dataset during subsequent training of the feature extraction network. This includes: minimizing the expected loss function on the training dataset when no noisy labels are present to obtain the optimal network parameters for the feature extraction network; and, when noisy labels are present, constructing the data selection network to assign a weight to the loss of each training sample, minimizing the weighted loss function on the training dataset to obtain the optimal network parameters for the feature extraction network, and minimizing the meta-loss function on the meta-dataset to obtain the optimal network parameters for the data selection network.
[0059] The expression for minimizing the expected loss function is: .in, To obtain the optimal network parameters for the feature extraction network, Let N be the operator whose parameters minimize the expected loss function, and N be the total number of samples in the training dataset. For loss function, Let w be the true label of the i-th sample, and w be the network parameters. For the feature extraction network on the i-th sample The actual output.
[0060] The expression for minimizing the weighted loss function is: .in, This represents the loss value of the feature extraction network on training sample i. Choose the weight function given by the network for the data, and accept the loss of the feature extraction network on sample i. And data selection network parameters The weighting function assigns a weight to the loss value of each sample to mitigate the impact of noisy labels.
[0061] The expression for the minimum loss function is: .in, To select the optimal network parameters for the data, M is the total number of samples in the metadata set. , For loss function, Let i be the true label of the i-th sample in the metadata set. Use optimal parameters for the feature extraction network Input data for the i-th sample in the metadata set The predictions made Selecting network parameters based on data The optimal parameters of the calculated feature extraction network.
[0062] Specifically, the entire training process is a two-layer optimization framework. The internal optimization (updating the parameters of the feature extraction network and each loss function) is performed based on the current parameters of the data selection network, while the external optimization (updating the parameters of the data selection network) is based on the results of the internal optimization (i.e., the optimal parameters of the feature extraction network). This meta-learning training strategy helps to guide the learning process of the feature extraction network through the data selection network in the presence of noisy labels, thereby improving the robustness and performance of the model.
[0063] S105: Use the trained feature extraction network to extract features from the query image and each image in the retrieved database, and calculate the similarity between each query image and all images in the dataset.
[0064] Specifically, the similarity can be calculated using the cosine similarity formula.
[0065] S106: Sort the images in descending order of similarity and output a preset number of images as search results.
[0066] Specifically, the preset quantity can be set according to actual needs.
[0067] S107: Use recall to evaluate the accuracy of search results.
[0068] The accuracy of search results is evaluated using recall, which includes searching the test set for the k images most similar to the query image. A successful retrieval is recorded if at least one of these k images belongs to the same category as the query image. The accuracy of the search results is then evaluated using the average number of successful retrievals for all query images.
[0069] Recall is used to evaluate the accuracy of search results. The formula is as follows: .in, This represents the percentage of successful retrievals of the k most similar images to the query image in the test set, where n is the total number of queries. This means that for the *a*-th query, if at least one of the *k* returned results belongs to the same category as the query image, then... ,otherwise, .
[0070] Specifically, this formula calculates the percentage of all queries that find at least one image of the same category as the query image among the k returned results. A higher percentage indicates better accuracy of the search results.
[0071] The above content is Embodiment 1 of this application. Embodiment 2 of this application is provided below. The only difference between Embodiment 2 of this application and Embodiment 1 of this application is that the dataset selected in S101 for dataset calculation is different.
[0072] S101: Create the dataset. The dataset includes a training dataset, a meta-dataset, and a retrieval database. The training dataset includes noise labels.
[0073] Specifically, the Stanford Dogs dataset can be selected as the dataset to be built. The first 60 classes are selected as the training dataset, and 10 images are randomly selected from each class in the training dataset as the meta dataset. This ensures that the meta dataset is representative of all classes in the training dataset, but the sample size is much smaller than that of the training dataset. The last 60 classes (test set) are used as the query and retrieval database. These classes are not visible during training and are used to evaluate the performance of image retrieval.
[0074] Figure 3The recall rate is calculated on the CUB dataset containing 20% symmetric noise (i.e., Example 1) as provided in the embodiments of this application. A diagram illustrating the comparison results. It measures whether, for each query, the first result returned (i.e., the most similar image) belongs to the same category as the query image.
[0075] Figure 4 The recall rate is calculated on the CUB dataset containing 20% symmetric noise (i.e., Example 1) as provided in the embodiments of this application. A diagram illustrating the comparison results. It measures whether, for each query, at least one of the first four results belongs to the same category as the query image.
[0076] Figure 5 The recall rate is calculated on the CUB dataset containing 20% symmetric noise (i.e., Example 1) as provided in the embodiments of this application. A diagram illustrating the comparison results. It measures whether, for each query, at least one of the first eight results returned by the system belongs to the same category as the query image.
[0077] like Figure 3 , Figure 4 and Figure 5 Both models are shown to have recall rates at different time points on the CUB dataset containing 20% symmetric noise. The first model used weighted nsoftmax loss, and the second model used standard softmax loss. Both models show trends from time point 0 to 50. The figures show that the data selection network in Implementation 1 is robust to noise, improving model performance even in noisy image retrieval scenarios. Furthermore, the data selection network performs stably on the test set without overfitting. This can be explained by the fact that the data selection network assigns higher weights to clean samples, emphasizing clean samples, and lower weights to noisy samples, thus effectively suppressing overfitting and improving model performance.
[0078] Figure 6 This is a schematic diagram of the sample weight distribution on the CUB dataset containing 20% symmetric noise, provided as an embodiment of this application. Figure 6As shown, clean weight is the weight of clean samples, and noisy weight is the weight of noisy samples. Guided by clean metadata, the sample selection network guides the feature extraction network to learn in the right direction and learn a more realistic sample distribution. Therefore, the sample selection network can apply smaller weights, or even 0, to noisy samples with large loss values to suppress the effect of noisy samples, and apply larger weights, close to 1, to clean samples with small loss values to emphasize the role of clean samples.
[0079] Figure 7 The recall rate was calculated on the Stanford Dogs dataset (i.e., Example 2) containing 40% symmetric noise, as provided in the embodiments of this application. A diagram illustrating the comparison results. It measures whether, for each query, the first result returned (i.e., the most similar image) belongs to the same category as the query image.
[0080] Figure 8 The recall rate was calculated on the Stanford Dogs dataset (i.e., Example 2) containing 40% symmetric noise, as provided in the embodiments of this application. A diagram illustrating the comparison results. It measures whether, for each query, at least one of the first four results belongs to the same category as the query image.
[0081] Figure 9 The recall rate was calculated on the Stanford Dogs dataset (i.e., Example 2) containing 40% symmetric noise, as provided in the embodiments of this application. A diagram illustrating the comparison results. It measures whether, for each query, at least one of the first eight results returned by the system belongs to the same category as the query image.
[0082] like Figure 7 , Figure 8 and Figure 9Both models are shown to have recall rates at different time points on the Stanford Dogs dataset containing 40% symmetrical noise. The first model used weighted nsoftmax loss, and the second model used standard softmax loss. Both models show trends from time 0 to 50. The figures show that the data selection network in Implementation 1 is robust to noise, improving model performance even in noisy image retrieval scenarios. Furthermore, the data selection network performs stably on the test set without overfitting. This can be explained by the fact that the data selection network assigns higher weights to clean samples, emphasizing clean samples, and lower weights to noisy samples, thus effectively suppressing overfitting and improving model performance.
[0083] Figure 10 This is a schematic diagram illustrating the sample weight distribution on the Standford Dogs dataset containing 40% symmetrical noise, provided as an embodiment of this application. Figure 10 As shown, clean weight is the weight of clean samples, and noisy weight is the weight of noisy samples. Guided by clean metadata, the sample selection network guides the feature extraction network to learn in the right direction and learn a more realistic sample distribution. Therefore, the sample selection network can apply smaller weights, or even 0, to noisy samples with large loss values to suppress the effect of noisy samples, and apply larger weights, close to 1, to clean samples with small loss values to emphasize the role of clean samples.
[0084] This application embodiment also provides a data selection device 1100 for noisy image retrieval scenarios, such as... Figure 11 As shown, the device includes: a creation module 1101, an extraction module 1102, a judgment module 1103, a processing module 1104, a calculation module 1105, a sorting module 1106, and an evaluation module 1107.
[0085] Module 1101 is used to create a dataset. The dataset includes a training dataset, a meta-dataset, and a retrieval database. The training dataset includes noise labels.
[0086] The extraction module 1102 is used to establish a feature extraction network to extract key features of the input image samples, to initially train the feature extraction network using a training dataset, and to calculate the loss value of each sample during the training process using a feature extraction loss function.
[0087] The judgment module 1103 is used to take the calculated loss value as input to a pre-established data selection network. The data selection network outputs a weight value for each sample based on the loss value, and determines the sample category based on the weight value. The sample category is either a noisy sample or a clean sample.
[0088] The processing module 1104 is used to select the weight values of the network output using data, and to perform weighting processing on the training dataset during the subsequent training process of the feature extraction network. Specifically, the weighting processing reduces the weight of noisy samples and increases the weight of clean samples.
[0089] The computation module 1105 is used to extract features from the query image and each image in the retrieved database using a trained feature extraction network, and to calculate the similarity between each query image and all images in the dataset.
[0090] The sorting module 1106 is used to sort images in descending order of similarity and output a preset number of images as search results.
[0091] Evaluation module 1107 is used to evaluate the accuracy of search results using recall.
[0092] Some modules in the apparatus described in this application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0093] The apparatus or module described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. For ease of description, the above apparatus is described by dividing it into various modules according to their functions. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.
[0094] The methods, apparatus, or modules described in this application can be implemented in a computer-readable program code manner. The controller can be implemented in any suitable manner, for example, as a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of a memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code manner, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included within it for implementing various functions can also be considered as structures within the hardware component. Alternatively, the device used to implement various functions can be viewed as either a software module that implements the method or a structure within a hardware component.
[0095] like Figure 12 As shown in the figure, this application embodiment also provides a data selection server for noisy image retrieval scenarios, including a memory 1201 and a processor 1202; the memory 1201 is used to store computer-executable instructions; the processor 1202 is used to execute computer-executable instructions to implement the data selection method for noisy image retrieval scenarios described above in this application embodiment.
[0096] This application also provides a computer-readable storage medium storing executable instructions. When a computer executes the executable instructions, it can implement the data selection method for noisy image retrieval scenarios described above in this application.
[0097] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, or it can be embodied in the process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in the embodiments of this application.
[0098] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations.
[0099] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of this application.
Claims
1. A data selection method for noisy image retrieval scenarios, characterized in that, include: Establish a dataset; wherein the dataset includes a training dataset, a meta-dataset, and a retrieval database, and the training dataset includes noise labels; A feature extraction network is established to extract key features from input image samples. The network is initially trained using a training dataset. A feature extraction loss function is used to calculate the loss value for each sample during training. The calculation of the loss value for each sample during training using the feature extraction loss function includes: according to the formula... Calculate the feature extraction loss function for each sample during training; where, The feature extraction loss function is... The feature vector of the input sample is extracted from the input sample by the feature extraction network. This is a proxy vector that is closely related to the true class label y of the sample currently being processed. For other categories whose proxy vectors do not belong to the true category label y of the currently being processed sample, Here, is a temperature coefficient used to control the smoothness of the output probability distribution; C is the set of all possible categories; and c is an index variable. The calculated loss value is used as the input to a pre-built data selection network. The data selection network outputs a weight value for each sample based on the loss value, and determines the sample category based on the weight value; wherein, the sample category is a noisy sample or a clean sample. The weights output by the data selection network are used to weight the training dataset during subsequent training of the feature extraction network. This includes iteratively executing training steps until a stopping condition is met. The stopping condition is reaching a preset maximum number of iterations or the loss value decreasing to less than a preset threshold. The iterative training steps include: randomly selecting a small batch of samples from the training dataset, calculating the loss value through forward propagation, and updating the feature extraction network and feature extraction loss function using stochastic gradient descent. Randomly selecting a small batch of samples from the metadata set, using the selected small batch of samples, extracting key features of the samples through the feature extraction network, and using these features as input to the data selection network. Calculating the loss value of the data selection network and updating its parameters using stochastic gradient descent. Using the updated data selection network parameters, recalculating the weight value of each sample in the training dataset. Continuing to train the feature extraction network using the recalculated weight values. The weighting process reduces the weight of noisy samples and increases the weight of clean samples. The trained feature extraction network is used to extract features from the query image and each image in the retrieved database, and the similarity between each query image and all images in the dataset is calculated. Images are sorted in descending order of similarity, and a preset number of images are output as search results. Use recall to assess the accuracy of search results.
2. The data selection method for noisy image retrieval scenarios according to claim 1, characterized in that, Before establishing the feature extraction network to extract key features from the input image samples, the following steps are included: The feature extraction network is pre-trained using a pre-trained deep convolutional neural network, and its parameters are fine-tuned on the training dataset to adapt it to the feature representation of noisy images.
3. The data selection method for noisy image retrieval scenarios according to claim 1, characterized in that, The steps to build a data selection network are as follows: Construct a data selection network with a multilayer perceptron structure on the metadata dataset. The data selection network includes a hidden layer.
4. The data selection method for noisy image retrieval scenarios according to claim 1, characterized in that, The step of using data to select the weight values output by the network, and weighting the training dataset during subsequent training of the feature extraction network, also includes: When there are no noisy labels, minimize the expected loss function on the training dataset to obtain the optimal network parameters for the feature extraction network; When noisy labels exist, a data selection network is constructed to assign a weight to the loss of each training sample. The weighted loss function is minimized on the training dataset to obtain the optimal network parameters for the feature extraction network, and the meta-loss function is minimized on the meta-dataset to obtain the optimal network parameters for the data selection network.
5. The data selection method for noisy image retrieval scenarios according to claim 4, characterized in that, The expression for minimizing the expected loss function is: ;in, To obtain the optimal network parameters for the feature extraction network, Let N be the operator whose parameters minimize the expected loss function, and N be the total number of samples in the training dataset. For loss function, Let w be the true label of the i-th sample, and w be the network parameters. For the feature extraction network on the i-th sample The actual output; The expression for minimizing the weighted loss function is: ;in, This represents the loss value of the feature extraction network on training sample i. Choose the weight function given by the network for the data, and accept the loss of the feature extraction network on sample i. And data selection network parameters The weighting function assigns a weight to the loss value of each sample to mitigate the impact of noisy labels; The expression for the minimum loss function is: ;in, To select the optimal network parameters for the data, M is the total number of samples in the metadata set. , For loss function, The clean label for the i-th sample in the metadata set. Use optimal parameters for the feature extraction network Input data for the i-th sample in the metadata set The predictions made Selecting network parameters based on data The optimal parameters of the calculated feature extraction network.
6. The data selection method for noisy image retrieval scenarios according to claim 1, characterized in that, The method of using recall to evaluate the accuracy of search results includes: Search the test set for the k images that are most similar to the query image. If at least one of these k images belongs to the same category as the query image, a successful retrieval is recorded. The accuracy of the search results is evaluated using the average number of successful retrievals for all query images.
7. The data selection method for noisy image retrieval scenarios according to claim 6, characterized in that, The recall rate is used to evaluate the accuracy of search results, and the calculation formula is as follows: ;in, This represents the percentage of successful retrievals of the k most similar images to the query image in the test set, where n is the total number of queries. This means that for the *a*-th query, if at least one of the *k* returned results belongs to the same category as the query image, then... ,otherwise, .
8. A data selection device for noisy image retrieval scenarios, characterized in that, include: A module is provided for building a dataset; wherein the dataset includes a training dataset, a meta-dataset, and a retrieval database, and the training dataset includes noise labels; The extraction module is used to establish a feature extraction network to extract key features from input image samples. It initially trains the feature extraction network using a training dataset and calculates the loss value for each sample during training using a feature extraction loss function. The calculation of the loss value for each sample during training using the feature extraction loss function includes: according to the formula... Calculate the feature extraction loss function for each sample during training; where, The feature extraction loss function is... The feature vector of the input sample is extracted from the input sample by the feature extraction network. This is a proxy vector that is closely related to the true class label y of the sample currently being processed. For other categories whose proxy vectors do not belong to the true category label y of the currently being processed sample, Here, is a temperature coefficient used to control the smoothness of the output probability distribution; C is the set of all possible categories; and c is an index variable. The judgment module is used to take the calculated loss value as the input of a pre-established data selection network. The data selection network outputs a weight value for each sample based on the loss value, and determines the sample category based on the weight value; wherein, the sample category is a noisy sample or a clean sample. The processing module is used to perform weighted processing on the training dataset during subsequent training of the feature extraction network using the weight values output by the data selection network. This includes: iteratively executing iterative training steps until a stopping condition is met; the stopping condition is reaching a preset maximum number of iterations or the loss value decreasing to less than a preset threshold. The iterative training steps include: randomly selecting a small batch of samples from the training dataset, calculating the loss value through forward propagation, and updating the feature extraction network and feature extraction loss function using stochastic gradient descent; randomly selecting a small batch of samples from the metadata set, using the selected small batch of samples to extract key features of the samples through the feature extraction network, and using these features as input to the data selection network; calculating the loss value of the data selection network and updating the parameters of the data selection network using stochastic gradient descent; recalculating the weight value of each sample in the training dataset using the updated data selection network parameters; and continuing to train the feature extraction network using the recalculated weight values. The weighted processing reduces the weight of noisy samples and increases the weight of clean samples. The computation module is used to extract features from the query image and each image in the retrieved database using a trained feature extraction network, and to calculate the similarity between each query image and all images in the dataset. The sorting module is used to sort images in descending order of similarity and output a preset number of images as search results. The evaluation module is used to assess the accuracy of search results using recall.