Feature extraction based on reordering, training method and device of feature extraction model

Abstract

The application discloses a feature extraction method and device based on reordering, and a training method and device of a feature extraction model. The first feature vector of a query image, the first feature vector of each candidate image and a first adjacency matrix are input into a graph convolutional neural network in the feature extraction model for feature transformation, so that the second feature vector of the query image and the second feature vector of each candidate image are obtained. According to the similarity between the second feature vector of each candidate image and the second feature vector of the query image, a target image that is semantically matched with the query image is determined from a plurality of candidate images. In the application, the similarity between the first feature vectors of the plurality of candidate images and the first feature vector of the query image is considered, and the similarity between the first feature vectors of the plurality of candidate images is also considered, so that the accuracy of image feature extraction is further improved, and the reliability is increased.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to reordering-based feature extraction, training methods and apparatus for feature extraction models. Background Technology

[0002] Image feature extraction is the process of extracting useful data or information from an image to obtain an image represented by feature vectors such as numerical values, vectors, and symbols.

[0003] Currently, the main method for feature extraction of a query image is to aggregate the feature vectors of multiple candidate images that are highly similar to the feature vector of the query image based on the similarity between the feature vectors of multiple candidate images and the feature vector of the query image, thereby obtaining an image that matches the query image.

[0004] However, in actual database searches, there are often candidate images that have high similarity to the image feature vectors of the query image but different semantic content. Using existing methods for image feature vector extraction can easily lead to these candidate images with different semantic content from the query image being mistakenly regarded as matching images, resulting in inaccurate image feature extraction and low reliability. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a method and apparatus for feature extraction based on reordering and for training a feature extraction model.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] This application provides a feature extraction method based on reordering, including:

[0008] The query image and multiple candidate images are input into the feature extraction network of the feature extraction model to extract features, thereby obtaining the first feature vector of the query image and the first feature vector of each candidate image.

[0009] Based on the similarity between the first feature vector of the query image and the first feature vector of each candidate image, a first adjacency matrix corresponding to the first feature vector of the query image and the first feature vector of each candidate image is determined;

[0010] The first feature vector of the query image, the first feature vector of each candidate image, and the first adjacency matrix are input into the graph convolutional neural network in the feature extraction model for feature transformation to obtain the second feature vector of the query image and the second feature vector of each candidate image;

[0011] Based on the similarity between the second feature vector of each candidate image and the second feature vector of the query image, a target image that semantically matches the query image is determined from multiple candidate images.

[0012] Further, determining the target image semantically matching the query image from multiple candidate images based on the similarity between the second feature vector of each candidate image and the second feature vector of the query image includes:

[0013] Based on the similarity between the second feature vector of each candidate image and the second feature vector of the query image, the candidate images are sorted from high to low to obtain a sorting result;

[0014] The top k candidate images from the sorting results are selected as the target images.

[0015] This application provides a method for training a feature extraction model, including:

[0016] The reference image, positive sample image, and negative sample image are input into a feature extraction network to extract features, thereby obtaining the third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image.

[0017] The first loss is obtained based on the third feature vector of the reference image, the third feature vector of the positive sample image, the third feature vector of the negative sample image, and the first loss function;

[0018] Based on the similarity between the third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image, a second adjacency matrix corresponding to the third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image is determined.

[0019] The third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image are combined with the second adjacency matrix and input into the graph convolutional neural network to perform feature transformation, thereby obtaining the fourth feature vector of the reference image, the fourth feature vector of the positive sample image, and the fourth feature vector of the negative sample image.

[0020] The second loss is obtained based on the fourth feature vector of the reference image, the fourth feature vector of the positive sample image, the fourth feature vector of the negative sample image, and the second loss function.

[0021] Based on the first loss and the second loss, determine the target training loss;

[0022] The network parameters of the feature extraction network are trained based on the target training loss.

[0023] The trained feature extraction network is defined as the feature extraction model.

[0024] Further, obtaining the second loss based on the fourth feature vector of the reference image, the fourth feature vector of the positive sample image, the fourth feature vector of the negative sample image, and the second loss function includes:

[0025] Based on the similarity between the fourth feature vector of the positive sample image and the fourth feature vector of the reference image, and the similarity between the fourth feature vector of the negative sample image and the fourth feature vector of the reference image, the fourth feature vectors of the positive sample image and the fourth feature vector of the negative sample image are reordered to obtain the reordering result.

[0026] The second loss is obtained based on the reordering result and the second loss function.

[0027] Further, obtaining the second loss based on the reordering result and the second loss function includes:

[0028] Calculate the average precision value based on the reordering result, the number of positive sample images, and the number of negative sample images;

[0029] The second loss is obtained by inputting the average accuracy value into the second loss function.

[0030] Further, obtaining the first loss based on the third feature vector of the reference image, the third feature vector of the positive sample image, the third feature vector of the negative sample image, and the first loss function includes:

[0031] The difference between the positive similarity and the negative similarity is obtained based on the positive similarity between the third feature vector of the positive sample image and the third feature vector of the reference image and the negative similarity between the third feature vector of the positive sample image and the third feature vector of the reference image.

[0032] The difference, the number of positive sample images, and the number of negative sample images are input into the first loss function to obtain the first loss.

[0033] Further, determining the target training loss based on the first loss and the second loss includes:

[0034] The adjusted second loss is obtained by multiplying the adjustment coefficient by the second loss.

[0035] The target training loss is determined based on the first loss and the adjusted second loss.

[0036] This application also provides a feature extraction device based on reordering, including:

[0037] First obtaining module, first determining module, second obtaining module, and second determining module;

[0038] The first obtaining module is used to input the query image and multiple candidate images into the feature extraction network in the feature extraction model for feature extraction, and obtain the first feature vector of the query image and the first feature vector of each candidate image;

[0039] The first determining module is configured to determine a first adjacency matrix corresponding to the first feature vector of the query image and the first feature vector of each candidate image based on the similarity between the first feature vector of the query image and the first feature vector of each candidate image;

[0040] The second obtaining module is used to input the first feature vector of the query image, the first feature vector of each candidate image and the first adjacency matrix into the graph convolutional neural network in the feature extraction model for feature transformation, so as to obtain the second feature vector of the query image and the second feature vector of each candidate image;

[0041] The second determining module is used to determine a target image that semantically matches the query image from multiple candidate images based on the similarity between the second feature vector of each candidate image and the second feature vector of the query image.

[0042] Further, the second determining module includes:

[0043] Sorting unit and selection unit;

[0044] The sorting unit is used to sort the multiple candidate images from high to low according to the similarity between the second feature vector of each candidate image and the second feature vector of the query image to obtain a sorting result;

[0045] The selection unit is used to select the top k candidate images from the sorting results and determine them as the target images.

[0046] This application provides a training device for a feature extraction model, including: a third obtaining module, a fourth obtaining module, a third determining module, a fifth obtaining module, a sixth obtaining module, a fourth determining module, a training module, and a fifth determining module;

[0047] The third acquisition module is used to input the reference image, positive sample image and negative sample image into the feature extraction network for feature extraction, and obtain the third feature vector of the reference image, the third feature vector of the positive sample image and the third feature vector of the negative sample image;

[0048] The third obtaining module is used to obtain a first loss based on the third feature vector of the reference image, the third feature vector of the positive sample image, the third feature vector of the negative sample image, and a first loss function;

[0049] The third determining module is used to determine the second adjacency matrix corresponding to the third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image based on the similarity between the third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image.

[0050] The fifth obtaining module is used to perform feature transformation on the third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image and the second adjacency matrix input graph convolutional neural network to obtain the fourth feature vector of the reference image, the fourth feature vector of the positive sample image, and the fourth feature vector of the negative sample image.

[0051] The sixth obtaining module is used to obtain a second loss based on the fourth feature vector of the reference image, the fourth feature vector of the positive sample image, the fourth feature vector of the negative sample image, and the second loss function;

[0052] The fourth determining module is used to determine the target training loss based on the first loss and the second loss;

[0053] The training module is used to train the network parameters of the feature extraction network according to the target training loss;

[0054] The fifth determining module is used to determine the trained feature extraction network as the feature extraction model. Further, the sixth obtaining module includes:

[0055] First obtaining unit and second obtaining unit;

[0056] The first obtaining unit is configured to reorder the fourth feature vector of the positive sample image and the fourth feature vector of the negative sample image based on the similarity between the fourth feature vector of the positive sample image and the fourth feature vector of the reference image, and the similarity between the fourth feature vector of the negative sample image and the fourth feature vector of the reference image, to obtain a reordering result.

[0057] The second obtaining unit is used to obtain the second loss based on the reordering result and the second loss function.

[0058] Further, the second obtaining unit is used for:

[0059] Calculate the average precision value based on the reordering result, the number of positive sample images, and the number of negative sample images;

[0060] The second loss is obtained by inputting the average accuracy value into the second loss function.

[0061] Furthermore, the third obtaining module is used for:

[0062] The difference between the positive similarity and the negative similarity is obtained based on the positive similarity between the third feature vector of the positive sample image and the third feature vector of the reference image and the negative similarity between the third feature vector of the positive sample image and the third feature vector of the reference image.

[0063] The difference, the number of positive sample images, and the number of negative sample images are input into the first loss function to obtain the first loss.

[0064] Furthermore, the fourth determining module is used for:

[0065] The adjusted second loss is obtained by multiplying the adjustment coefficient by the second loss.

[0066] The target training loss is determined based on the first loss and the adjusted second loss.

[0067] This application embodiment also provides a computer device, the computer device including a processor and a memory:

[0068] The memory is used to store program code and transmit the program code to the processor;

[0069] The processor is configured to execute, according to the instructions in the program code, the steps of a reordering-based feature extraction method or the steps of a feature extraction model training method as described above.

[0070] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a reordering-based feature extraction method or a training method for a feature extraction model as described above.

[0071] Compared with the prior art, this application has the following beneficial effects:

[0072] This application extracts features from a query image and multiple candidate images by inputting them into a feature extraction network within a feature extraction model. This yields a first feature vector for the query image and a first feature vector for each candidate image. A first adjacency matrix, determined based on the similarity between the first feature vectors of the query image and each candidate image, along with the first feature vectors of the query image and each candidate image, are then input into a graph convolutional neural network for feature transformation. This transforms the query image into a second feature vector and a second feature vector for each candidate image. Finally, based on the similarity between the second feature vectors of each candidate image and the second feature vector of the query image, a target image semantically matching the query image is determined from the multiple candidate images. This application considers not only the similarity between the first feature vectors of multiple candidate images and the first feature vector of the query image, but also the similarity among the first feature vectors of multiple candidate images themselves, thereby further improving the accuracy and reliability of image feature extraction. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 A flowchart illustrating a reordering-based feature extraction method provided in this application embodiment;

[0075] Figure 2 A flowchart illustrating the training process of a feature extraction model provided in this application embodiment;

[0076] Figure 3 This is a schematic diagram of a reordering-based feature extraction device provided in an embodiment of this application. Detailed Implementation

[0077] As described above, improving the accuracy of image feature extraction has become a technical problem that urgently needs to be solved by those skilled in the art.

[0078] The inventors discovered through research that existing technologies aggregate the feature vectors of multiple candidate images that have high similarity to the feature vectors of the query image to obtain an image that matches the query image. However, although the feature vectors of some candidate images have high similarity to the vector features of the query image, their semantic content is different. Using existing technologies can easily lead to these images being treated as images that match the query image, thereby reducing the accuracy of image feature extraction.

[0079] This application performs feature transformation on a graph convolutional neural network by inputting a first adjacency matrix determined based on the similarity between the first feature vector of the query image and the first feature vector of each candidate image, the first feature vector of the query image, and the first feature vector of each candidate image into a graph convolutional neural network. This process yields a second feature vector of the query image and a second feature vector of each candidate image. Then, based on the similarity between the second feature vector of each candidate image and the second feature vector of the query image, a target image semantically matching the query image is determined from multiple candidate images. By considering both the similarity between the first feature vectors of multiple candidate images and the first feature vector of the query image, and the similarity among the first feature vectors of multiple candidate images, the accuracy of image feature extraction is further improved.

[0080] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0081] See Figure 1 This figure is a flowchart of a reordering-based feature extraction method provided in an embodiment of this application. The reordering-based feature extraction method provided in this embodiment can be implemented through steps S101 to S104.

[0082] S101: Input the query image and multiple candidate images into the feature extraction network in the feature extraction model to extract features, and obtain the first feature vector of the query image and the first feature vector of each candidate image.

[0083] The feature extraction network in the feature extraction model is used to extract feature vectors from the image. In step S101, the feature extraction network extracts features from the query image and multiple candidate images to obtain a first feature vector of the query image and a first feature vector of each candidate image, so as to subsequently determine the first adjacency matrix corresponding to the first feature vector of the query image and the first feature vector of each candidate image.

[0084] S102: Based on the similarity between the first feature vector of the query image and the first feature vector of each candidate image, determine the first adjacency matrix corresponding to the first feature vector of the query image and the first feature vector of each candidate image.

[0085] The first adjacency matrix is ​​obtained by calculating the similarity between the first feature vectors. For example, if there are N first feature vectors, the first adjacency matrix is ​​an N×N matrix, and the values ​​in the first adjacency matrix consist of the similarity between each pair of first feature vectors. The first adjacency matrix is ​​determined so that the first feature vector of the query image, the first feature vector of each candidate image, and the first adjacency matrix can be input into the graph convolutional neural network for feature transformation.

[0086] S103: Input the first feature vector of the query image, the first feature vector of each candidate image and the first adjacency matrix into the graph convolutional neural network in the feature extraction model to perform feature transformation, and obtain the second feature vector of the query image and the second feature vector of each candidate image.

[0087] The first feature vector of the query image, the first feature vector of each candidate image, and the first adjacency matrix representing the similarity between the first feature vectors are input into the graph convolutional neural network. If the input consists of N first feature vectors, the graph convolutional neural network will still output N second feature vectors after calculation, and the order of these N second feature vectors corresponds one-to-one with the order of the N first feature vectors.

[0088] S104: Based on the similarity between the second feature vector of each candidate image and the second feature vector of the query image, determine the target image that semantically matches the query image from multiple candidate images.

[0089] After calculating the second feature vector of each candidate image and the second feature vector of the query image through a graph convolutional neural network, the similarity between the second feature vector of each candidate image and the second feature vector of the query image is calculated. The higher the similarity, the more semantically matched the candidate image and the query image are. Then, the target image that semantically matches the query image is determined from multiple candidate images.

[0090] This application embodiment determines a target image semantically matching the query image from multiple candidate images based on the similarity between the second feature vector of each candidate image and the second feature vector of the query image. Since the first adjacency matrix represents the pairwise similarity between the first feature vectors, and the second feature vector is obtained by inputting the first feature vector and the first adjacency matrix into a graph convolutional neural network and performing feature transformation, the second feature vector can also represent the pairwise similarity between the first feature vectors. Therefore, by determining the target image semantically matching the query image from multiple candidate images based on the similarity between the second feature vector of each candidate image and the second feature vector of the query image, the accuracy of image feature extraction is further improved.

[0091] Furthermore, S104 can be implemented through S201 to S202.

[0092] S201: Based on the similarity between the second feature vector of each candidate image and the second feature vector of the query image, sort the multiple candidate images from high to low to obtain a sorting result.

[0093] Specifically, the second feature vectors of candidate images that have a high similarity to the second feature vector of the query image are ranked first, so that the candidate images ranked first can be determined as the target image in subsequent steps.

[0094] S202: Select the top k candidate images from the sorting results and determine them as the target images.

[0095] Specifically, based on the sorting results obtained in S101, the top k candidate images are determined as target images. Compared to directly determining the target image based on the similarity between the first feature vectors of multiple candidate images and the first feature vector of the query image, the method provided in this application embodiment can obtain k candidate images that are more similar to the query image.

[0096] This application provides a training method for a feature extraction model, including steps S301 to S308, see below. Figure 2 The figure is a flowchart of the training process of a feature extraction model provided in an embodiment of this application.

[0097] S301: Input the reference image, positive sample image and negative sample image into the feature extraction network to extract features and obtain the third feature vector of the reference image, the third feature vector of the positive sample image and the third feature vector of the negative sample image.

[0098] Specifically, positive sample images represent images with the same semantics as the reference image, while negative sample images represent images with different semantics from the reference image. By inputting a reference image, a certain number of positive sample images, and a certain number of negative sample images into a feature extraction network for feature extraction, the third feature vector of the reference image, the third feature vector of the positive sample images, and the third feature vector of the negative sample images can be obtained.

[0099] S302: Obtain the first loss based on the third feature vector of the reference image, the third feature vector of the positive sample image, the third feature vector of the negative sample image, and the first loss function.

[0100] Based on the third feature vector and the first loss function, the first loss is obtained so that the target training loss can be obtained subsequently.

[0101] Furthermore, S302 can be implemented through S401 to S402.

[0102] S401: Based on the positive similarity between the third feature vector of the positive sample image and the third feature vector of the reference image, and the negative similarity between the third feature vector of the positive sample image and the third feature vector of the reference image, the difference between the positive similarity and the negative similarity is obtained.

[0103] S402: Input the difference, the number of positive sample images, and the number of negative sample images into the first loss function to obtain the first loss.

[0104] Specifically, the first loss function can be represented by equation (1):

[0105]

[0106] Assuming the number of reference images is 1, and the sum of the number of positive and negative sample images is N-1, where d(a, pi) represents the similarity between the positive sample image and the reference image; d(a, nj) represents the similarity between the negative sample image and the reference image; N p N represents the number of positive sample images. n The number of negative sample images; margin is a hyperparameter, a scalar and a positive number, which can be adjusted according to specific circumstances.

[0107] S303: Based on the similarity between the third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image, determine the second adjacency matrix corresponding to the third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image.

[0108] The second adjacency matrix is ​​obtained by calculating the similarity between the third eigenvectors, and it can represent the pairwise similarity between the third eigenvectors.

[0109] S304: Input the third feature vector of the reference image, the third feature vector of the positive sample image, the third feature vector of the negative sample image, and the second adjacency matrix into the graph convolutional neural network for feature transformation to obtain the fourth feature vector of the reference image, the fourth feature vector of the positive sample image, and the fourth feature vector of the negative sample image.

[0110] After feature transformation, the positions of the fourth feature vector and the third feature vector are in one-to-one correspondence. In supervised training, the order of the third feature vectors of the benchmark image, the positive sample image, and the negative sample image is known. Therefore, the order of the fourth feature vectors of the benchmark image, the positive sample image, and the negative sample image is also known. Obtaining the fourth feature vector with a known order facilitates the subsequent acquisition of the second loss.

[0111] S305: Obtain the second loss based on the fourth feature vector of the reference image, the fourth feature vector of the positive sample image, the fourth feature vector of the negative sample image, and the second loss function.

[0112] Obtain the second loss to determine the target training loss later.

[0113] Furthermore, S305 can be implemented through S501 to S502.

[0114] S501: Based on the similarity between the fourth feature vector of the positive sample image and the fourth feature vector of the reference image, and the similarity between the fourth feature vector of the negative sample image and the fourth feature vector of the reference image, the fourth feature vector of the positive sample image and the fourth feature vector of the negative sample image are reordered to obtain the reordering result.

[0115] Specifically, based on the similarity between the fourth feature vector of the positive sample image and the fourth feature vector of the reference image, and the similarity between the fourth feature vector of the negative sample image and the fourth feature vector of the reference image, the fourth feature vectors with higher similarity are arranged first so that the second loss can be obtained subsequently.

[0116] S502: Obtain the second loss based on the reordering result and the second loss function.

[0117] Obtain the second loss to determine the target training loss later.

[0118] Furthermore, S502 can be implemented through S601 to S602.

[0119] S601: Calculate the average precision value based on the reordering result, the number of positive sample images, and the number of negative sample images.

[0120] Specifically, assuming the number of reference images is 1, and the sum of the number of positive sample images and negative sample images is N-1, the average precision value can be expressed by equations (2)-(3) as follows:

[0121]

[0122]

[0123] In the above formula, R represents the order of the reordered positive and negative sample images; Let R represent the sequence of positive and negative sample images after traversing and reordering them in order. When traversing to the k-th image, if there are i positive sample images in total, then... otherwise AP(R) represents the average precision value.

[0124] S602: Input the second loss function according to the average accuracy value to obtain the second loss.

[0125] Specifically, the second loss function can be expressed by equation (4):

[0126] L rerank = -log AP(R) (4)

[0127] In the formula, L rerank This represents the second loss. The network parameters of the feature extraction network are trained by optimizing the second loss in a way that reduces it.

[0128] S306: Determine the target training loss based on the first loss and the second loss.

[0129] Furthermore, S306 can be implemented through S701 to S702.

[0130] S701: Obtain the adjusted second loss based on the product of the adjustment coefficient and the second loss.

[0131] S702: Determine the target training loss based on the first loss and the adjusted second loss.

[0132] Specifically, the target training loss function in this application embodiment can be represented by equation (5):

[0133] L = L metric +λ×L rerank (5)

[0134] In equation (5), λ is the adjustment coefficient, which can be set to a relatively small value, such as a value between 0.01 and 0.3.

[0135] S307: Train the network parameters of the feature extraction network according to the target training loss.

[0136] The network parameters of the feature extraction network are trained according to the target training loss determined in S306 to optimize the feature extraction model.

[0137] S308: The trained feature extraction network is determined as the feature extraction model.

[0138] The feature extraction models obtained from S301 to S308 can effectively improve the accuracy of image feature extraction.

[0139] See Figure 3The figure is a schematic diagram of a feature extraction device based on reordering provided in an embodiment of this application. Its specific implementation method is consistent with the implementation method and the technical effect achieved in the embodiments of the above method, and some contents will not be repeated.

[0140] This application also provides a feature extraction device based on reordering, including:

[0141] First obtaining module 1101, first determining module 1102, second obtaining module 1103, and second determining module 1104;

[0142] The first obtaining module 1101 is used to input the query image and multiple candidate images into the feature extraction network in the feature extraction model to extract features, and obtain the first feature vector of the query image and the first feature vector of each candidate image;

[0143] The first determining module 1102 is used to determine the first adjacency matrix corresponding to the first feature vector of the query image and the first feature vector of each candidate image based on the similarity between the first feature vector of the query image and the first feature vector of each candidate image;

[0144] The second obtaining module 1103 is used to input the first feature vector of the query image, the first feature vector of each candidate image and the first adjacency matrix into the graph convolutional neural network in the feature extraction model for feature transformation, so as to obtain the second feature vector of the query image and the second feature vector of each candidate image;

[0145] The second determining module 1104 is used to determine a target image that semantically matches the query image from multiple candidate images based on the similarity between the second feature vector of each candidate image and the second feature vector of the query image.

[0146] Further, the second determining module 1104 includes:

[0147] Sorting unit and selection unit;

[0148] The sorting unit is used to sort the multiple candidate images from high to low according to the similarity between the second feature vector of each candidate image and the second feature vector of the query image to obtain a sorting result;

[0149] The selection unit is used to select the top k candidate images from the sorting results and determine them as the target images.

[0150] This application provides a training device for a feature extraction model, including: a third obtaining module, a fourth obtaining module, a third determining module, a fifth obtaining module, a sixth obtaining module, a fourth determining module, a training module, and a fifth determining module;

[0151] The third acquisition module is used to input the reference image, positive sample image and negative sample image into the feature extraction network for feature extraction, and obtain the third feature vector of the reference image, the third feature vector of the positive sample image and the third feature vector of the negative sample image;

[0152] The third obtaining module is used to obtain a first loss based on the third feature vector of the reference image, the third feature vector of the positive sample image, the third feature vector of the negative sample image, and a first loss function;

[0153] The third determining module is used to determine the second adjacency matrix corresponding to the third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image based on the similarity between the third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image.

[0154] The fifth obtaining module is used to perform feature transformation on the third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image and the second adjacency matrix input graph convolutional neural network to obtain the fourth feature vector of the reference image, the fourth feature vector of the positive sample image, and the fourth feature vector of the negative sample image.

[0155] The sixth obtaining module is used to obtain a second loss based on the fourth feature vector of the reference image, the fourth feature vector of the positive sample image, the fourth feature vector of the negative sample image, and the second loss function;

[0156] The fourth determining module is used to determine the target training loss based on the first loss and the second loss;

[0157] The training module is used to train the network parameters of the feature extraction network according to the target training loss;

[0158] The fifth determining module is used to determine the trained feature extraction network as the feature extraction model. Further, the sixth obtaining module includes:

[0159] First obtaining unit and second obtaining unit;

[0160] The first obtaining unit is configured to reorder the fourth feature vector of the positive sample image and the fourth feature vector of the negative sample image based on the similarity between the fourth feature vector of the positive sample image and the fourth feature vector of the reference image, and the similarity between the fourth feature vector of the negative sample image and the fourth feature vector of the reference image, to obtain a reordering result.

[0161] The second obtaining unit is used to obtain the second loss based on the reordering result and the second loss function.

[0162] Further, the second obtaining unit is used for:

[0163] Calculate the average precision value based on the reordering result, the number of positive sample images, and the number of negative sample images;

[0164] The second loss is obtained by inputting the average accuracy value into the second loss function.

[0165] Furthermore, the third obtaining module is used for:

[0166] The difference between the positive similarity and the negative similarity is obtained based on the positive similarity between the third feature vector of the positive sample image and the third feature vector of the reference image and the negative similarity between the third feature vector of the positive sample image and the third feature vector of the reference image.

[0167] The difference, the number of positive sample images, and the number of negative sample images are input into the first loss function to obtain the first loss.

[0168] Furthermore, the fourth determining module is used for:

[0169] The adjusted second loss is obtained by multiplying the adjustment coefficient by the second loss.

[0170] The target training loss is determined based on the first loss and the adjusted second loss.

[0171] This application embodiment also provides a computer device, the computer device including a processor and a memory:

[0172] The memory is used to store program code and transmit the program code to the processor;

[0173] The processor is configured to execute, according to the instructions in the program code, the steps of a reordering-based feature extraction method or the steps of a feature extraction model training method as described above.

[0174] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a reordering-based feature extraction method or a training method for a feature extraction model as described above.

[0175] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and equipment embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0176] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A feature extraction method based on reordering, characterized in that, include: The query image and multiple candidate images are input into the feature extraction network of the feature extraction model to extract features, thereby obtaining the first feature vector of the query image and the first feature vector of each candidate image. Based on the similarity between the first feature vector of the query image and the first feature vector of each candidate image, a first adjacency matrix corresponding to the first feature vector of the query image and the first feature vector of each candidate image is determined; The first feature vector of the query image, the first feature vector of each candidate image, and the first adjacency matrix are input into the graph convolutional neural network in the feature extraction model for feature transformation to obtain the second feature vector of the query image and the second feature vector of each candidate image; Based on the similarity between the second feature vector of each candidate image and the second feature vector of the query image, a target image that semantically matches the query image is determined from multiple candidate images.

2. The method according to claim 1, characterized in that, The step of determining a target image semantically matching the query image from multiple candidate images based on the similarity between the second feature vector of each candidate image and the second feature vector of the query image includes: Based on the similarity between the second feature vector of each candidate image and the second feature vector of the query image, the candidate images are sorted from high to low to obtain a sorting result; The top k candidate images from the sorting results are selected as the target images.

3. A training method for a feature extraction model, characterized in that, include: The reference image, positive sample image, and negative sample image are input into a feature extraction network for feature extraction to obtain the third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image. The first loss is obtained based on the third feature vector of the reference image, the third feature vector of the positive sample image, the third feature vector of the negative sample image, and the first loss function; Based on the similarity between the third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image, a second adjacency matrix corresponding to the third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image is determined. The third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image are combined with the second adjacency matrix and input into the graph convolutional neural network to perform feature transformation, thereby obtaining the fourth feature vector of the reference image, the fourth feature vector of the positive sample image, and the fourth feature vector of the negative sample image. The second loss is obtained based on the fourth feature vector of the reference image, the fourth feature vector of the positive sample image, the fourth feature vector of the negative sample image, and the second loss function. Based on the first loss and the second loss, determine the target training loss; The network parameters of the feature extraction network are trained based on the target training loss. The trained feature extraction network is defined as the feature extraction model.

4. The method according to claim 3, characterized in that, The step of obtaining the second loss based on the fourth feature vector of the reference image, the fourth feature vector of the positive sample image, the fourth feature vector of the negative sample image, and the second loss function includes: Based on the similarity between the fourth feature vector of the positive sample image and the fourth feature vector of the reference image, and the similarity between the fourth feature vector of the negative sample image and the fourth feature vector of the reference image, the fourth feature vectors of the positive sample image and the fourth feature vector of the negative sample image are reordered to obtain the reordering result. The second loss is obtained based on the reordering result and the second loss function.

5. The method according to claim 4, characterized in that, The step of obtaining the second loss based on the reordering result and the second loss function includes: Calculate the average precision value based on the reordering result, the number of positive sample images, and the number of negative sample images; The second loss is obtained by inputting the average accuracy value into the second loss function.

6. The method according to claim 3, characterized in that, The step of obtaining the first loss based on the third feature vector of the reference image, the third feature vector of the positive sample image, the third feature vector of the negative sample image, and the first loss function includes: The difference between the positive similarity and the negative similarity is obtained based on the positive similarity between the third feature vector of the positive sample image and the third feature vector of the reference image and the negative similarity between the third feature vector of the positive sample image and the third feature vector of the reference image. The difference, the number of positive sample images, and the number of negative sample images are input into the first loss function to obtain the first loss.

7. The method according to claim 3, characterized in that, The step of determining the target training loss based on the first loss and the second loss includes: The adjusted second loss is obtained by multiplying the adjustment coefficient by the second loss. The target training loss is determined based on the first loss and the adjusted second loss.

8. A feature extraction device based on reordering, characterized in that, include: First obtaining module, first determining module, second obtaining module, and second determining module; The first obtaining module is used to input the query image and multiple candidate images into the feature extraction network in the feature extraction model for feature extraction, and obtain the first feature vector of the query image and the first feature vector of each candidate image; The first determining module is configured to determine a first adjacency matrix corresponding to the first feature vector of the query image and the first feature vector of each candidate image based on the similarity between the first feature vector of the query image and the first feature vector of each candidate image; The second obtaining module is used to input the first feature vector of the query image, the first feature vector of each candidate image and the first adjacency matrix into the graph convolutional neural network in the feature extraction model for feature transformation, so as to obtain the second feature vector of the query image and the second feature vector of each candidate image; The second determining module is used to determine a target image that semantically matches the query image from multiple candidate images based on the similarity between the second feature vector of each candidate image and the second feature vector of the query image.

9. A training device for a feature extraction model, characterized in that, include: The third acquisition module, the fourth acquisition module, the third determination module, the fifth acquisition module, the sixth acquisition module, the fourth determination module, the training module, and the fifth determination module; The third acquisition module is used to input the reference image, positive sample image and negative sample image into the feature extraction network for feature extraction, and obtain the third feature vector of the reference image, the third feature vector of the positive sample image and the third feature vector of the negative sample image; The third obtaining module is used to obtain a first loss based on the third feature vector of the reference image, the third feature vector of the positive sample image, the third feature vector of the negative sample image, and a first loss function; The third determining module is used to determine the second adjacency matrix corresponding to the third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image based on the similarity between the third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image. The fifth obtaining module is used to perform feature transformation on the third feature vector of the reference image, the third feature vector of the positive sample image, and the third feature vector of the negative sample image and the second adjacency matrix input graph convolutional neural network to obtain the fourth feature vector of the reference image, the fourth feature vector of the positive sample image, and the fourth feature vector of the negative sample image. The sixth obtaining module is used to obtain a second loss based on the fourth feature vector of the reference image, the fourth feature vector of the positive sample image, the fourth feature vector of the negative sample image, and the second loss function; The fourth determining module is used to determine the target training loss based on the first loss and the second loss; The training module is used to train the network parameters of the feature extraction network according to the target training loss; The fifth determining module is used to determine the trained feature extraction network as the feature extraction model.

10. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute, according to the instructions in the program code, the steps of a reordering-based feature extraction method as described in any one of claims 1-2, or the steps of a feature extraction model training method as described in any one of claims 3-7.

Images