Image retrieval method, face recognition method and pedestrian re-recognition method based on reference set

Abstract

The invention discloses an image retrieval method based on a reference set, a face recognition method and a pedestrian re-recognition method, and the method comprises the steps: obtaining a feature vector of a query image and a candidate library through feature extraction, constructing a reference set in which a category label does not intersect with the candidate library, calculating the correlation degree of the feature vector and the reference set, and carrying out the recognition of a human face. And combining cosine similarity to construct a metric function fusing absolute and relative distances to realize feature comparison, and outputting a retrieval result after sorting. According to the method, the reference set is introduced, so that the model not only pays attention to the absolute distance between samples during retrieval, but also can utilize the reference set sample as an intermediary to construct the relative distance and capture other fine-grained differences, and the performance of a retrieval comparison system is improved; according to the method and the system, the reference set samples with diversity, representativeness and robustness are screened out by performing statistical analysis on the comparison scores of the images of the candidate reference set and the test set and setting reasonable statistical conditions, the diversity of actual test scene data is covered, category bias is avoided, data redundancy is reduced, and the performance of an image retrieval system is improved.

Description

Technical Field

[0001] This invention relates to the technical field of image retrieval, and in particular to an image retrieval method, retrieval system, and face recognition method based on a reference set. Background Technology

[0002] With the rapid development of deep learning technology, image retrieval technology has been widely applied in many fields, such as face recognition, person re-identification, and object detection. Currently, mainstream image retrieval systems typically use absolute distances between single samples, such as Euclidean distance and cosine distance, as metrics during the model retrieval stage.

[0003] However, relying solely on absolute distance for matching has several drawbacks. First, during image acquisition, factors such as changes in lighting, differences in shooting angles, occlusion of target objects, and the complexity of the background can all lead to instability in the metric space. This results in excessively large differences between samples of the same class (i.e., excessive intra-class differences) and excessively high similarities between samples of different classes (i.e., excessively high inter-class similarities). Second, in cross-domain retrieval (such as retrieval of visible light images and infrared images) or cross-modal retrieval (such as retrieval of images and text), absolute distance matching struggles to effectively align features from different domains or modalities, leading to a significant decrease in retrieval performance. Furthermore, absolute distance metrics are highly sensitive to noisy data and outliers, further impacting the accuracy of retrieval results. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies that rely solely on absolute distance for image retrieval, such as unstable metric space, difficulty in cross-domain and cross-modal alignment, and sensitivity to noise. This invention provides a reference set-based image retrieval method that can capture fine-grained differences and improve retrieval accuracy, as well as a face recognition method and a pedestrian re-identification method based on this method.

[0005] To achieve the above objectives, the solution of the present invention is: An image retrieval method based on a reference set includes the following steps: S1. Perform feature extraction on the query image to obtain the query feature vector. , where d represents the dimension of the feature vector; S2. Obtain the feature set of the candidate library. m is the number of samples in the candidate library. ; S3, Constructing the Reference Set The reference set The category label set of the reference set has no intersection with the category label set of the samples in the candidate library, where n is the number of samples in the reference set, and ; S4. Calculate the query feature vectors respectively. First correlation with reference set R and the feature vector of each sample in the candidate library. Second correlation with reference set R ; S5. Construct a distance metric function by combining the cosine similarity, first correlation, and second correlation between the query feature vector and the candidate feature vector. Feature matching is achieved through this distance metric function; S6. Distance values ​​for all candidate samples The samples are sorted, and the candidate samples corresponding to the maximum values ​​are selected as the final search results.

[0006] Furthermore, in step S4, the first degree of correlation Second degree of correlation ,in Represents the query feature vector Features of the reference set samples Cosine similarity between them To represent the scale factor, This indicates the first few input data items arranged in descending order. Item, of which .

[0007] Furthermore, in step S5, the distance metric function... ,in s is the scale factor. express Cosine similarity between two feature vectors Let be any two eigenvectors.

[0008] Furthermore, the scale factor s has a value range of 1≤s≤100, and the weight of cosine similarity is strengthened or weakened by adjusting s.

[0009] Furthermore, in step S3, a reference set is constructed. The specific steps are as follows: S31. Obtain a reference set of alternatives. and test set n is the number of samples in the candidate set, and m is the number of samples in the test set; S32. Calculate the cosine distance matrix between the candidate set and the test site. ; S33, Cosine Distance Matrix Execute each row of data The operation yields the processed matrix. , ,and ;in, Indicates input data Arranged from largest to smallest item; S34, Calculate the matrix Mean of each row The mean vector is obtained. ; S35, Regarding the mean vector Sort the samples in descending order and select the top t samples to form the final reference set. ,in This is the index of the sorted sample in the original candidate set.

[0010] Furthermore, the range of parameter k is 1≤k≤m, and the range of parameter t is 1≤t≤n. During the feature alignment process, the TOPk operation is used to filter the samples in the reference set that are most relevant to the feature vector, thereby suppressing the interference of meaningless samples on the feature alignment results.

[0011] Furthermore, the category labels are combined as follows: For reference sets Two samples , Category tags A set of category labels that do not belong to the reference set ,Right now .

[0012] A face recognition method employs the aforementioned image retrieval method based on a reference set, wherein: the query image is a face image, the candidate library is a preset face library, the reference set is constructed using any one of the methods described in claims 5-6, and the face categories in the reference set have no overlap with the face categories in the candidate library.

[0013] A pedestrian re-identification method employs the aforementioned image retrieval method based on a reference set, wherein: the query image is a pedestrian image, the candidate library is a preset pedestrian library, the reference set is constructed using any one of the methods described in claims 5-6, and the pedestrian identities in the reference set have no overlap with the pedestrian identities in the candidate library.

[0014] Compared with the prior art, the present invention has the following advantages after adopting the above solution: This invention introduces a reference set to construct a relative distance metric, which breaks through the limitations of traditional absolute distance metrics. It can effectively solve the problems of unstable metric space, excessive intra-class differences, and excessive inter-class similarity, while improving the alignment effect of cross-domain and cross-modal retrieval.

[0015] The reference set screening strategy proposed in this invention can efficiently screen samples with diversity, representativeness and robustness by statistically analyzing the comparison scores between the candidate reference set and the test set, avoiding class bias and data redundancy, and providing a guarantee for improving retrieval performance.

[0016] The distance metric function constructed in this invention integrates the direct similarity between samples and the correlation with the reference set, which can capture fine-grained differences and improve the accuracy and robustness of retrieval results.

[0017] Applying reference set-based image retrieval methods to the field of face recognition can effectively address issues such as lighting variations, angle differences, and occlusion in face recognition, significantly improving the accuracy and reliability of face recognition.

[0018] The reference set-based image retrieval method of the present invention has good versatility and practicality, and can be widely applied to various image retrieval scenarios and face recognition-related fields. Detailed Implementation

[0019] To further explain the technical solution of the present invention, the present invention will be described in detail below through specific embodiments.

[0020] This invention discloses an image retrieval method based on a reference set, which accurately retrieves the most similar target sample from a pre-defined candidate library given the feature vector of a query image. Specifically, it includes the following steps: S1. Feature Extraction Step: Perform feature extraction on the query image to be retrieved to obtain the query feature vector. , where d represents the dimension of the feature vector.

[0021] S2. Steps for obtaining the candidate library feature set: Obtain the feature vectors of all samples in the candidate library to form the candidate feature set. m is the number of samples in the candidate library. Feature extraction can be achieved using existing deep learning models (such as ResNet, MobileNet, etc.) to ensure that the output feature vectors have good discriminative power.

[0022] S3. Reference Set Construction Steps: Constructing the Reference Set The reference set The category label set of the reference set has no intersection with the category label set of the samples in the candidate library, where n is the number of samples in the reference set, and The reference set must satisfy the condition that its category label set has no intersection with the category label set of the candidate library samples, so as to avoid category interference.

[0023] Building a reference set The build process is as follows: Define reference set alternatives Define the test set Where n is the number of samples in the candidate set and m is the number of samples in the test set; Using the test set Samples and reference sets in By performing a full alignment of the samples in the dataset, the cosine distance matrix can be obtained as follows: .and For the cosine distance matrix Each row of data Operation, among which Indicates input data Arranged from largest to smallest Item, of which Then it can be represented as , Furthermore, the cosine distance matrix can be expressed as... .

[0024] For the cosine distance matrix obtained in step 1 Calculate each row The average value of the data is expressed as Then the cosine distance matrix The mean can be expressed as .

[0025] Regarding the mean Sort them to satisfy ,in Represents a sequence The sequence numbers of the first, second, and nth samples in the original sequence after sorting are then selected. The reference set consists of 10 samples, denoted as _____. .

[0026] S4. Relevance Calculation Steps: Calculate the query feature vectors respectively. First correlation with reference set R and the feature vector of each sample in the candidate library. Second correlation with reference set R The formula for calculating the degree of correlation is: ,in Represents the sample feature vector Features of the reference set samples Cosine similarity is used to measure the degree of similarity between two objects. The scale factor is used to adjust the weights of similarity. This means sorting the n input similarity values ​​from largest to smallest and selecting the k largest values, where... This operation can filter out the samples in the reference set that are most relevant to the target features and suppress the interference of meaningless samples on the feature comparison results.

[0027] Step S5, Feature Comparison Step: Constructing a Distance Metric Function This distance metric function considers both the direct similarity between the query feature and the candidate feature (absolute distance correlation) and the correlation between both and the reference set (relative distance correlation). Feature comparison is achieved through the distance metric function, and the specific formula is as follows:

[0028] in, This represents the direct cosine similarity between the query feature and the candidate features. The exponential function is used to enhance the discriminative power of the similarity. This is used to reflect the degree of correlation between features and the reference set. The higher the correlation, the smaller the value of this item, so that the distance function can comprehensively reflect absolute similarity and relative correlation, and achieve more accurate feature comparison.

[0029] S6. Retrieval and sorting steps: Calculate the distance values ​​corresponding to all candidate samples. The samples are sorted from largest to smallest. The larger the distance value, the higher the similarity between the query image and the candidate sample. The candidate sample corresponding to the maximum value is selected as the final search result.

[0030] This invention introduces a reference set, enabling the model to move beyond focusing solely on the absolute distance between the sample to be retrieved and the candidate samples during the retrieval process. It can also utilize the reference set samples as an intermediary to construct relative distances, capture other fine-grained differences, and improve the performance of the retrieval and comparison system.

[0031] In the reference set construction stage, this invention performs statistical analysis on the comparison scores between candidate reference set images and the test set, and then sets reasonable statistical conditions to select reference set samples that are diverse, representative and robust, covering the diversity of actual test scenario data, avoiding category bias, reducing data redundancy and improving the performance of the image retrieval system.

[0032] This invention also provides a face recognition method that applies the above-mentioned reference set-based image retrieval method to the field of face recognition. The core of this method is to achieve accurate face recognition by fusing feature comparisons from the reference set. The specific steps are as follows: A1. Facial Feature Extraction Steps: Extract features from the query face image to obtain a query face feature vector of dimension d. .

[0033] S2. Steps for obtaining the feature set of the face candidate database: Obtain the feature vectors of all samples in the face candidate database to form the candidate path feature set. m is the number of samples in the face candidate database. Feature extraction can be achieved using existing deep learning models (such as ResNet, SphereFace, etc.) to ensure that the output facial feature vectors have good discriminative power.

[0034] S3. Steps for constructing a face reference set: Constructing a face reference set This facial reference set The set of category labels has no overlap with the set of category labels for samples in the face candidate database, where n is the number of samples in the face reference set, and The reference set must satisfy the condition that its category label set has no intersection with the category label set of the candidate library samples, so as to avoid category interference.

[0035] Building a face reference set The build process is as follows: Define a candidate set of face references Define the face test set ; where n is the number of face candidate samples and m is the number of face test samples; Using the face test set Samples and face reference sets By performing a full alignment of the samples in the dataset, the cosine distance matrix can be obtained as follows: .

[0036] For the cosine distance matrix Each row of data Operation, among which Indicates input data Arranged from largest to smallest Item, of which Then it can be represented as , Furthermore, the cosine distance matrix can be expressed as... .

[0037] For the obtained cosine distance matrix Calculate each row The average value of the data is expressed as Then the cosine distance matrix The mean can be expressed as .

[0038] Regarding the mean Sort them to satisfy ,in Represents a sequence The sequence numbers of the first, second, and nth samples in the original sequence after sorting are then selected. The reference set consists of 10 samples, denoted as _____. .

[0039] S4. Steps for calculating face correlation: Calculate the feature vector of the queried face. With reference set The third degree of correlation and the feature vector of each sample in the candidate library. With reference set The fourth degree of correlation ; The formula for calculating the degree of correlation is: ,in Represents the sample feature vector Features of the reference set samples Cosine similarity is used to measure the degree of similarity between two objects. The scale factor is used to adjust the weights of similarity. This means sorting the n input similarity values ​​from largest to smallest and selecting the k largest values, where... .

[0040] Step S5, Feature Comparison Step: Constructing a Face Distance Measurement Function This face distance metric function considers both the direct similarity (absolute distance correlation) between the face query features and the face candidate features, as well as the correlation (relative distance correlation) between the two and the face reference set. Feature comparison is achieved through the face distance metric function, and the specific formula is as follows:

[0041] in, This represents the direct cosine similarity between the query feature and the candidate feature. Let s be any two facial feature vectors, and s be the scale factor. This function fuses the absolute similarity and relative correlation of facial features to achieve fine-grained facial feature comparison and effectively cope with interference factors such as lighting, angle, and occlusion.

[0042] S6. Face recognition sorting steps: For all candidate face samples, calculate the distance values... The images are sorted from largest to smallest. The larger the distance value, the higher the similarity between the query face image and the candidate face sample. The candidate sample corresponding to the maximum value is selected as the final search result.

[0043] The image retrieval method based on the reference set of this invention can also be applied to the field of pedestrian re-identification. Its method is similar to that of face recognition, and the core is to achieve accurate pedestrian re-identification by fusing feature comparison of the reference set.

[0044] To verify the effectiveness of the present invention, the image retrieval method based on the reference set of the present invention was applied to face recognition and pedestrian re-identification tasks for experimental verification.

[0045] A private dataset was used as the candidate reference set. This dataset consisted entirely of facial images of internet celebrities, containing 289,295 images of 8,419 individuals. This application provides two test datasets. First, 13,233 digital facial images of 5,749 individuals were obtained from the public dataset LabeledFaceintheWild (LFW), serving as test set 1. Second, a private dataset containing 136,603 digital facial images of 52,404 individuals was constructed, serving as test set 2. The feature comparison method and reference set selection strategy described above were tested using these two test datasets.

[0046] Table 1 shows the comparison results of the proposed method with five other existing face recognition models on test set 1, specifically the correct recognition rate (TPR@FPR=0.1%) with an error rate of 0.1% and the detection hit rate (DIR@FAR=1%) with an error rate of 1%. Table 2 shows the comparison results of the proposed method with five other existing face recognition models on test set 1, specifically the correct recognition rate (TPR@FPR=0.01%) with an error rate of 0.01% with an error rate of 0.001% with an error rate of 0.001% with an error rate of 0.001% with an initial hit rate (Rank-1). A comparison of Tables 1 and 2 shows that the proposed method outperforms the five selected comparison models in both datasets.

[0047] Table 1

[0048] Table 2

[0049] For the pedestrian re-identification task, this application uses a private dataset as a candidate reference set. This private dataset is entirely derived from publicly available pedestrian re-identification datasets on the Internet, containing 189,256 images of 3,476 people. This invention uses two publicly available test sets, Market-1501 and DukeMTMC-reID, to compare with four other existing pedestrian re-identification models in terms of Rank-1 accuracy and Mean Average Precision (mAP). Table 3 shows the comparison results on the Market-1501 test set. Table 4 shows the comparison results on the DukeMTMC-reID test set. The comparison in Tables 3 and 4 shows that the image retrieval method based on the reference set of this invention can significantly improve the accuracy of the identification model.

[0050] Table 3

[0051] Table 4

[0052] The technical solution of the present invention is not limited to the parameter settings and model selection mentioned in the above embodiments. In practical applications, the feature extraction model, parameters k, t, s, etc. can be adjusted according to the specific scenario. Any appropriate changes or modifications made by those skilled in the art are within the protection scope of the present invention.

[0053] For example, in the feature extraction stage, lightweight models such as MobileNet and EfficientNet can be used to adapt to the deployment requirements of mobile devices; in the process of selecting reference sets (including face reference sets), the values ​​of k and t can be adjusted according to the size of the candidate set and the features of the test set; in the distance metric function, the scale factor s can be adjusted according to the actual scenario to optimize retrieval or recognition performance.

[0054] Furthermore, the reference set-based image retrieval method of the present invention can also be applied to other image retrieval scenarios, such as vehicle retrieval and product image retrieval. Performance improvements can be achieved simply by adapting the query image, candidate library, and reference set to the data types of the corresponding scenarios.

[0055] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, information or data related to "face" involved in this disclosure were obtained with full authorization.

Claims

1. An image retrieval method based on a reference set, characterized in that, Includes the following steps: S1. Perform feature extraction on the query image to obtain the query feature vector. , where d represents the dimension of the feature vector; S2. Obtain the feature set of the candidate library. m is the number of samples in the candidate library. ; S3, Constructing the Reference Set The reference set The category label set of the reference set has no intersection with the category label set of the samples in the candidate library, where n is the number of samples in the reference set, and ; S4. Calculate the query feature vectors respectively. First correlation with reference set R and the feature vector of each sample in the candidate library. Second correlation with reference set R ; S5. Construct a distance metric function by combining the cosine similarity, first correlation, and second correlation between the query feature vector and the candidate feature vector. Feature matching is achieved through this distance metric function; S6. Distance values ​​for all candidate samples The samples are sorted, and the candidate samples corresponding to the maximum values ​​are selected as the final search results.

2. The image retrieval method based on a reference set as described in claim 1, characterized in that: In step S4, the first degree of correlation Second degree of correlation ,in Represents the query feature vector Features of the reference set samples Cosine similarity between them To represent the scale factor, This indicates the first few input data items arranged in descending order. Item, of which .

3. The image retrieval method based on a reference set as described in claim 1 or 2, characterized in that: In step S5, the distance metric function ,in s is the scale factor. express Cosine similarity between two feature vectors Let be any two eigenvectors.

4. The image retrieval method based on a reference set as described in claim 3, characterized in that: The scaling factor s has a value range of 1≤s≤100, and the weight of cosine similarity is strengthened or weakened by adjusting s.

5. The image retrieval method based on a reference set as described in claim 1, characterized in that: In step S3, a reference set is constructed. The specific steps are as follows: S31. Obtain a reference set of alternatives. and test set n is the number of samples in the candidate set, and m is the number of samples in the test set; S32. Calculate the cosine distance matrix between the candidate set and the test site. ； S33, Cosine Distance Matrix Execute each row of data The operation yields the processed matrix. , ,and ;in, Indicates input data Arranged from largest to smallest item; S34, Calculate the matrix Mean of each row The mean vector is obtained. ; S35, Regarding the mean vector Sort the samples in descending order and select the top t samples to form the final reference set. ,in This is the index of the sorted sample in the original candidate set.

6. The image retrieval method based on a reference set as described in claim 5, characterized in that: The parameter k takes values ​​in the range of 1 ≤ k ≤ m, and the parameter t takes values ​​in the range of 1 ≤ t ≤ n. During the feature alignment process, the TOPk operation is used to filter the samples in the reference set that are most relevant to the feature vector, thereby suppressing the interference of meaningless samples on the feature alignment results.

7. The image retrieval method based on a reference set as described in claim 1, characterized in that: The category tags are combined as For reference sets Two samples , Category tags A set of category labels that do not belong to the reference set ,Right now .

8. A face recognition method, characterized in that, The image retrieval method based on a reference set according to any one of claims 1-7 is adopted, wherein: the query image is a face image, the candidate library is a preset face library, the reference set is constructed by any one of claims 5-6, and the face categories of the reference set have no overlap with the face categories of the candidate library.

9. A pedestrian re-identification method, characterized in that, The image retrieval method based on a reference set according to any one of claims 1-7 is adopted, wherein: the query image is a pedestrian image, the candidate library is a preset pedestrian library, the reference set is constructed by any one of claims 5-6, and the pedestrian identities in the reference set have no overlap with the pedestrian identities in the candidate library.

Images