A sample mining method, device, equipment and storage medium

By clustering the differences between human body characteristics and face characteristics, samples that are difficult to identify in the ReID model are automatically mined, which solves the problems of low accuracy and recall of ReID model in the prior art and improves model performance.

CN113111689BActive Publication Date: 2024-12-03TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202010032851.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-01-13
Publication Date
2024-12-03
Estimated Expiration
2040-01-13

AI Technical Summary

Technical Problem

The existing ReID models are not accurate and recalled in pedestrian recognition, especially when different people dress similarly or have large differences in human shapes, resulting in limited model performance development and it is difficult to effectively explore and improve model performance. Difficult sample.

Method used

By obtaining the target image set and the seed image set, human body features and face features are extracted, and clustering is performed. The first and second difficult samples are determined based on the differences between human body features and face features, so as to automatically mine the foregoing. Samples that are helpful to improve the performance of ReID model.

Benefits of technology

It realizes the excavation of non-fan high-similar samples and fan low-similar samples without manual participation, improves the mining efficiency and accuracy of difficult samples, and improves the performance of the ReID model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113111689B_ABST
    Figure CN113111689B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses a sample mining method, device, equipment and storage medium, wherein the sample mining method comprises: obtaining a target image set and a seed image set, wherein the target image set comprises images showing a target object to be identified, and the seed image set comprises images showing the target object and images showing a non-target object; combining the target image set and the seed image set to obtain a first image set; extracting human features and facial features of the images in the first image set; clustering the human features and facial features of the images in the first image set respectively to obtain a first clustering result and a second clustering result; determining non-same-person high-similar sample pairs and same-person low-similar sample pairs according to the first clustering result and the second clustering result. The method can automatically mine difficult samples that are helpful for improving the performance of the ReID model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and specifically to a sample mining method, device, equipment and storage medium. Background Art

[0002] In the application scenario of pedestrian tracking and identification, cameras installed in public places are often difficult to capture images with good facial imaging quality due to factors such as the installation angle and resolution, which limits the application of facial recognition technology in this scenario. As an important supplement to facial recognition technology, pedestrian re-identification (ReID) technology has emerged. It can rely on the ReID model to achieve cross-camera tracking and identification of pedestrians whose clear faces cannot be captured.

[0003] At the current stage, the recognition accuracy and recall rate of the ReID model are not very high. In actual business, when different people are dressed similarly, the ReID model is likely to mistakenly identify them as the same person, and the same person is mistakenly identified as different people by the ReID model because of the large differences in the human shapes presented in images taken by different cameras, etc. The above defects of the ReID model seriously restrict the development of ReID technology.

[0004] At present, increasing training sample data is still one of the most effective means to improve model performance. For the ReID model that has been launched, the sample data that can be correctly identified is not very helpful in improving model performance. On the contrary, the sample data that is easy to identify incorrectly is of high help value in improving model performance. This type of sample data is also called difficult samples.

[0005] In summary, how to mine difficult samples that are helpful in improving the performance of ReID models has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The embodiments of the present application provide a sample mining method, apparatus, device and storage medium, which can automatically mine difficult samples that are helpful in improving the performance of the ReID model.

[0007] In view of this, the first aspect of the present application provides a sample mining method, the method comprising:

[0008] Acquire a target image set and a seed image set; the target image set includes images showing a target object to be identified, and the seed image set includes images showing the target object and images showing non-target objects;

[0009] Combining the target image set and the seed image set to obtain a first image set;

[0010] Extracting human body features and facial features of images in the first image set;

[0011] Performing clustering processing on human features of images in the first image set to obtain a first clustering result; performing clustering processing on facial features of images in the first image set to obtain a second clustering result;

[0012] A first difficult sample and a second difficult sample are determined according to the first clustering result and the second clustering result; the first difficult sample includes images whose body features belong to the same category but whose facial features belong to different categories, and the second difficult sample includes images whose body features belong to different categories but whose facial features belong to the same category.

[0013] A second aspect of the present application provides a sample mining device, the device comprising:

[0014] An acquisition module, used to acquire a target image set and a seed image set; the target image set includes images showing a target object to be identified, and the seed image set includes images showing the target object and images showing objects other than the target object;

[0015] A first image set acquisition module, used for combining the target image set and the seed image set to obtain a first image set;

[0016] A feature extraction module, used to extract human body features and facial features of images in the first image set;

[0017] A clustering module, configured to perform clustering processing on human features of images in the first image set to obtain a first clustering result; and perform clustering processing on facial features of images in the first image set to obtain a second clustering result;

[0018] A difficult sample determination module is used to determine a first difficult sample and a second difficult sample based on the first clustering result and the second clustering result; the first difficult sample includes images whose human body features belong to the same class but whose facial features belong to different classes, and the second difficult sample includes images whose human body features belong to different classes but whose facial features belong to the same class.

[0019] A third aspect of the present application provides an electronic device, the device comprising a processor and a memory:

[0020] The memory is used to store computer programs;

[0021] The processor is used to execute the steps of the sample mining method as described in the first aspect according to the computer program.

[0022] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the steps of the sample mining method described in the first aspect above.

[0023] A fifth aspect of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the steps of the sample mining method described in the first aspect.

[0024] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0025] The embodiment of the present application provides a sample mining method, which determines the image samples that are difficult for the ReID model to accurately identify based on the difference between the face feature clustering results and the body feature clustering results, that is, digs out difficult samples that are helpful for improving the performance of the ReID model. Specifically, in the technical solution provided in the embodiment of the present application, a target image set and a seed image set are first obtained, wherein the target image set includes images showing the target object to be identified, and the seed image set includes both images showing the above target object and images showing non-target objects; then, the target image set and the seed image set are combined to obtain a first image set; then, the body features and face features of the images in the first image set are extracted; the body features of the images in the first image set are clustered to obtain a first clustering result, and the face features of the images in the first image set are clustered to obtain a second clustering result; finally, the first difficult sample and the second difficult sample are determined according to the first clustering result and the second clustering result, wherein the first difficult sample includes images whose body features belong to the same class but whose face features belong to different classes, and the second difficult sample includes images whose body features belong to different classes but whose face features belong to the same class. In this way, based on the matching relationship between the clustering categories of human features and the clustering categories of facial features obtained by clustering, high-similarity sample pairs of non-same-person samples (i.e., the first difficult samples) and low-similarity sample pairs of the same person (the second difficult samples) are automatically screened out from the image collection, thereby realizing the automatic mining of difficult samples that are helpful for improving the ReID model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of an application scenario of the sample mining method provided in an embodiment of the present application;

[0027] Figure 2 A flow chart of a sample mining method provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of a query image and a gallery image provided in an embodiment of the present application;

[0029] Figure 4 A schematic diagram of the process of manually labeling difficult samples provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of a flow chart of another sample mining method provided in an embodiment of the present application;

[0031] Figure 6 A schematic diagram of the structure of a first sample mining device provided in an embodiment of the present application;

[0032] Figure 7 A schematic diagram of the structure of a second sample mining device provided in an embodiment of the present application;

[0033] Figure 8 A schematic diagram of the structure of a third sample mining device provided in an embodiment of the present application;

[0034] Fig. 9 A schematic diagram of the structure of a fourth sample mining device provided in an embodiment of the present application;

[0035] Fig.10 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0036] Fig.11 A schematic diagram of the structure of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0040] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0041] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify targets, trace and measure targets, and further perform image processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0042] The technical solution provided in the embodiments of the present application relates to computer vision technology in the field of artificial intelligence.

[0043] Related technologies face the problem of mining difficult samples to improve the performance of ReID models. Currently, there are two main solutions to this problem: The first solution is to manually collect samples that are misidentified in online business. This solution is inefficient and usually can only collect high-similarity sample pairs of non-identical people, and it is difficult to collect low-similarity sample pairs of the same person. The second solution is to mine low-confidence sample pairs during the training of the ReID model and increase the number of times such samples participate in training. However, low-confidence sample pairs may actually be samples with noise, not difficult samples. Increasing the number of times such samples participate in training is likely to cause the trained ReID model to overfit to the noise.

[0044] In order to overcome the problems existing in the above-mentioned related technologies and ensure that difficult samples that are helpful in improving the performance of the ReID model can be mined efficiently and accurately, an embodiment of the present application provides a sample mining method.

[0045] In the sample mining method provided in the embodiment of the present application, a target image set and a seed image set are first obtained, wherein the target image set includes images showing a target object to be identified, and the seed image set includes both images showing the above target object and images showing objects other than the above target object; then the target image set and the seed image set are combined to obtain a first image set; then, the human body features and facial features of the images in the first image set are extracted, and the human body features of the images in the first image set are clustered to obtain a first clustering result, and the facial features of the images in the first image set are clustered to obtain a second clustering result; finally, a first difficult sample and a second difficult sample are determined according to the first clustering result and the second clustering result, wherein the first difficult sample here includes images whose human body features belong to the same class but whose facial features belong to different classes, and the second difficult sample here includes images whose human body features belong to different classes but whose facial features belong to the same class.

[0046] The above sample mining method introduces facial feature information into the process of mining difficult samples, and automatically determines non-same-person high-similar samples (i.e., first difficult samples) and same-person low-similar samples (i.e., second difficult samples) based on the difference between the human body feature clustering results and the facial feature clustering results, thereby realizing automatic mining of difficult samples that are helpful for improving the performance of the ReID model. Compared with the implementation method of mining difficult samples in related technologies, the technical solution provided in the embodiment of the present application can mine both non-same-person high-similar samples and same-person low-similar samples, and does not require manual participation, which can effectively improve the mining efficiency of difficult samples.

[0047] It should be understood that the sample mining method provided in the embodiment of the present application can be specifically applied to various devices with data processing capabilities, such as terminal devices, servers, etc. Among them, the terminal device can specifically be a computer, a personal digital assistant (PDA), a tablet computer, a smart phone, etc.; the server can specifically be an application server or a Web server. When actually deployed, the server can be an independent server or a cluster server.

[0048] To facilitate understanding of the technical solution provided by the embodiment of the present application, the following uses the sample mining method provided by the embodiment of the present application applied to a server as an example to exemplarily introduce the application scenarios to which the sample mining method provided by the embodiment of the present application is applicable.

[0049] See also Figure 1 , Figure 1 Schematic diagram of application scenarios of the sample mining method provided in the embodiment of the present application. Figure 1 As shown, the application scenario includes a server 100, which is used to execute the sample mining method provided in the embodiment of the present application, and can obtain a target image set (hereinafter referred to as a query set) and a seed image set (hereinafter referred to as a gallery set) processed in the actual application of the ReID model, and then filter out difficult samples from the obtained query set and gallery set.

[0050] In the process of applying ReID technology, online images are divided into query sets and gallery sets. The query set includes images showing the target object to be traced and identified. Different images in the query set can show the same target object or different target objects. The gallery set includes a large number of seed images. Some of these seed images show the target object to be traced and identified, while the objects shown in some images are not the target objects that need to be traced and identified. The essence of ReID technology is to retrieve images containing the target object in the gallery set based on the images in the query set.

[0051] Based on the sample mining method provided in the embodiment of the present application, when mining difficult samples that are helpful to improve the performance of the ReID model, the server 100 needs to obtain the query set and gallery set processed online by the ReID model, and use the images included in the query set and the images included in the gallery set to form a first image set 101. Then, the human features 102 and facial features 103 of the images in the first image set are extracted, the human features 102 of the images in the first image set are clustered to obtain a first clustering result 104, and the facial features 103 of the images in the first image set are clustered to obtain a second clustering result 105. Finally, the first difficult sample 106 is determined according to the first clustering result 104 and the second clustering result 105, wherein the first difficult sample can also be called a non-same person high similarity sample, which includes images whose human features belong to the same class but whose facial features belong to different classes, and the second difficult sample can also be called a same person low similarity sample, which includes images whose human features belong to different classes but whose facial features belong to the same class.

[0052] After server 100 selects the first difficult sample and the second difficult sample from the query set and the gallery set, it can use the selected first difficult sample and the second difficult sample to train the ReID model to improve the performance of the ReID model, so that the ReID model can more accurately realize trace tracing and identification of the target object.

[0053] It should be understood that Figure 1 The scenario shown is only an example. In practical applications, the sample mining method provided in the embodiment of the present application can be applied not only to Figure 1 The scenario shown can also be applied to other scenarios. For example, the sample mining method provided in the embodiment of the present application can be executed by a terminal device to mine difficult samples. No limitation is made herein on the application scenarios to which the sample mining method provided in the embodiment of the present application is applicable.

[0054] The sample mining method provided by this application is introduced below through examples.

[0055] See also Figure 2 , Figure 2 The following is a flow chart of the sample mining method provided in the embodiment of the present application. For the convenience of description, the following embodiment takes the server as the execution subject as an example to describe the implementation process of the sample mining method. Figure 2 As shown, the sample mining method includes the following steps:

[0056] Step 201: Acquire a target image set and a seed image set; the target image set includes images showing a target object to be identified, and the seed image set includes images showing the target object and images showing non-target objects.

[0057] When the server needs to mine difficult samples for training the ReID model, the server can obtain the query set (i.e., the target image set) and gallery set (i.e., the seed image set) used by the ReID model to process related tasks online and / or offline. The query set includes images showing target objects that need to be traced and identified, and different images in the query set can show the same target object or different target objects; the gallery set includes a large number of seed images, some of which show target objects that need to be traced and identified, and some of which show objects that are not target objects that need to be traced and identified.

[0058] It should be noted that the images included in the query set and gallery set are usually images that only include human figures, such as Figure 3 As shown in the images 301 and 302 in the query set and the gallery set, the images included in the query set and the gallery set are usually images obtained by performing a person extraction process on the overall image captured by the image acquisition device.

[0059] In specific implementation, if the server itself runs a ReID model, the server can directly obtain the query set and gallery set used by the ReID model to process related tasks, and then mine difficult samples based on the obtained query set and gallery set. If the server itself does not run the ReID model, the server can communicate with other devices running the ReID model to obtain the query set and gallery set used by the ReID model to process related tasks from other devices running the ReID model, or the server can retrieve the query set and gallery set from a related database, which stores the query set and gallery set used by the ReID model to process tasks. Of course, in actual applications, the server can also obtain the query set and gallery set in other ways, and this application does not limit the method of obtaining the query set and gallery set.

[0060] Optionally, in order to minimize the amount of computing required in the subsequent sample mining process, after the server obtains the query set and the gallery set, it can filter the images contained in the query set based on the gallery set to filter out images in the query set with a correct recognition rate higher than a preset threshold, that is, to filter out images in the query set that are easy to be correctly identified. Such images that are easy to be correctly identified are usually less helpful in improving the performance of the ReID model.

[0061] In a possible implementation, the server can implement filtering processing for the query set in the following manner: first determine a first threshold and a second threshold, and when searching for the query set in the gallery set, the retrieval accuracy determined based on the first threshold according to the first preset retrieval condition can meet the preset accuracy threshold, and the retrieval recall determined based on the second threshold according to the second preset retrieval condition meets the preset recall threshold; then, for each image in the query set, retrieve the target image with the highest similarity to the image in the gallery set, and determine whether the similarity between the two is in a target interval, if so, retain the image in the query set, otherwise filter out the image in the query set; the target interval here is determined based on the first threshold and the second threshold.

[0062] Exemplarily, the first threshold T1 and the second threshold T2 may be determined by experiments. For ease of description, the images in the query set are referred to as query images, and the images in the gallery set are referred to as gallery images.

[0063] When specifically determining the first threshold T1, each query image in the query set can be retrieved in the gallery set to determine the similar gallery images corresponding to each query image. Similar gallery images with a similarity greater than the first threshold T1 are selected from the gallery images corresponding to each query image, and the correct gallery image among the selected similar gallery images is determined. The object contained in the correct gallery image here is consistent with the object contained in the corresponding query image. Then, the proportion of the correct gallery image in the selected similar gallery images is calculated, and it is determined whether the proportion meets the preset accuracy threshold. If so, the first threshold T1 used in this experiment is determined as the first threshold for forming the target interval.

[0064] It should be understood that when determining the first threshold T1, when retrieving the similar gallery images corresponding to each query image, the image in the gallery collection with the highest similarity to the query image can be determined as the similar gallery image corresponding to the query image, that is, the gallery top1 image is retrieved. The above-mentioned preset accuracy threshold can be set according to actual needs, for example, set to 97%, etc., and this application does not specifically limit the preset accuracy threshold.

[0065] When specifically determining the second threshold value T2, each query image in the query set can be retrieved in the gallery set to determine the similar gallery images corresponding to each query image. From the gallery images corresponding to each query image, similar gallery images with a similarity greater than the second threshold value T2 are selected, and the correctly recalled gallery images among the selected similar gallery images are determined. The correctly recalled gallery images here refer to images that are consistent with the sample annotation results. Then, the proportion of the correctly recalled gallery images in the selected similar gallery images is calculated, and it is determined whether the proportion meets the preset recall rate threshold. If so, the second threshold value T2 used in this experiment is determined as the second threshold value for forming the target interval.

[0066] It should be understood that when determining the second threshold value T2, when retrieving the corresponding similar gallery images for each query image, the top five images in the gallery collection with the similarity to the query image can be determined as the similar gallery images corresponding to the query image, that is, the gallery top5 images are retrieved. The above-mentioned preset recall rate threshold value can be set according to actual needs, for example, set to 97%, etc., and the present application does not make a specific limitation on the preset recall rate threshold value.

[0067] After determining the first threshold T1 and the second threshold T2, the target interval [T1, T2] or [T2, T1] can be further determined based on the first threshold T1 and the second threshold T2. For each query image in the query set, the target gallery image that is most similar to the query image is retrieved from the gallery set, and the similarity S1 between the query image and the target gallery image is determined, and it is determined whether the similarity S1 is in the target interval. If so, the query image is retained, otherwise the query image is filtered out.

[0068] It should be noted that, in actual applications, in addition to filtering the query set in the above manner, the server may also filter the query set in other ways according to actual needs. This application does not impose any limitation on the method of filtering the query set.

[0069] Step 202: Combine the target image set and the seed image set to obtain a first image set.

[0070] After the server obtains the query set and the gallery set, it can reconstruct the query set and the gallery set, that is, use the images in the query set and the images in the gallery set to form a first image set, which is hereinafter referred to as the Body set.

[0071] It should be understood that if the server filters the query set before executing step 202, then when executing step 202, the server can combine the images retained after filtering the query set and the images in the gallery set to obtain the Body set, that is, the Body set is composed of the images retained in the query set and the images in the gallery set.

[0072] Optionally, in order to minimize the amount of computation required for subsequent processing, after the server combines the query set and the gallery set to obtain the Body set, it can further use the images in the Body set whose face detection quality meets preset standards to form a second image set, that is, select images with better face detection quality from the Body set to form the second image set, which will be referred to as the Face set hereinafter.

[0073] In a possible implementation, the server can determine the face detection quality of each image in the Body collection in the following manner: perform face detection on each image in the Body collection, determine the face detection result corresponding to each image, and the face detection result can indicate whether the face can be successfully detected in the image; retain the images in the Body collection whose face detection results indicate that the face can be successfully detected in the image, and filter out the images in the Body collection whose face detection results indicate that the face cannot be successfully detected in the image. Then, for the images retained in the Body collection, determine the quality score corresponding to each of these images, and the quality score is used to indicate the face detection quality in the image.

[0074] It should be noted that the server determines the quality sharing for the images retained in the Body set, and can refer to factors such as the blurriness and orientation of the face in the image. This application does not specifically limit the factors used for determining the quality sharing.

[0075] Accordingly, when the server composes the Face set, it can select images with quality scores greater than a preset score threshold from the images retained in the Body set to compose the Face set.

[0076] It should be understood that in actual applications, in addition to determining the face detection quality of each image in the Body collection in the above-mentioned manner, the server can also determine the face detection quality of various images in the Body collection in other ways. This application does not impose any limitation on the method of determining the face detection quality.

[0077] Step 203: extracting human body features and facial features of the images in the first image set.

[0078] After the server combination obtains the Body set, it can use the human feature extraction model to extract the human features of each image in the Body set, and use the face feature extraction model to extract the face features of each image in the Body set.

[0079] It should be understood that the human feature extraction model used here is a model that can be used to extract human features in images in the relevant technology, such as a ReID model, a human recognition model, etc., and this application does not make any restrictions on the model used to extract human features. Similarly, the facial feature extraction model used here is also a model that can be used to extract facial features in images in the relevant technology, such as a facial recognition model, etc., and this application does not make any restrictions on the model used to extract facial features.

[0080] It should be noted that if the server has screened out images with better face detection quality from the Body set to form a Face set before executing step 203, the server can perform human feature extraction processing on each image in the Body set and perform facial feature extraction processing on each image in the Face set when executing step 203.

[0081] Step 204: clustering the human features of each image in the first image set to obtain a first clustering result; clustering the facial features of each image in the first image set to obtain a second clustering result.

[0082] After the server extracts the human features and facial features of each image in the Body collection, it needs to cluster the human features of each image in the Body collection to obtain a first clustering result. The first clustering result may include multiple categories, and the human features included in each category have a high degree of similarity. If the target object is identified only from the perspective of human features, the objects identified based on the human features in the same category should be the same object. In addition, the server also needs to cluster the facial features of each image in the Body collection to obtain a second clustering result. The second clustering result may include multiple categories, and the facial features included in each category have a high degree of similarity. The facial features in the same category should actually correspond to the same object.

[0083] It should be noted that if the server has screened out images with better face detection quality from the Body collection to form a Face collection before executing step 203, and performed facial feature extraction processing on each image in the Face collection when executing step 203, then when the server executes step 204 accordingly, it should cluster the facial features of each image in the Face collection to obtain the second clustering result.

[0084] It should be noted that in actual applications, the server can use the corresponding clustering algorithm to cluster the human body features and facial features of each image in the Body collection according to actual needs. For example, a rank-order clustering algorithm or a graph convolution algorithm can be used for clustering. This application does not impose any limitation on the clustering algorithm used in clustering.

[0085] Step 205: Determine a first difficult sample and a second difficult sample based on the first clustering result and the second clustering result; the first difficult sample includes images whose human body features belong to the same category but whose facial features belong to different categories, and the second difficult sample includes images whose human body features belong to different categories but whose facial features belong to the same category.

[0086] After obtaining the first clustering result and the second clustering result through clustering processing, the server needs to select non-same-person high-similarity sample pairs (i.e., first difficult samples) and same-person low-similarity sample pairs (i.e., second difficult samples) from the Body set based on the first clustering result and the second clustering result.

[0087] When specifically selecting non-identical high-similarity sample pairs, the server can determine the images corresponding to each human body feature in a certain category in the first clustering result, and then determine whether the facial features corresponding to these images belong to the same category in the second clustering result, and select any two images whose facial features belong to different categories in the second clustering result to form a non-identical high-similarity sample pair.

[0088] When specifically selecting pairs of samples with low similarity to the same person, the server can determine the images corresponding to each facial feature in a certain category in the second clustering result, and then determine whether the human features corresponding to these images belong to the same category in the first clustering result, and select any two images whose human features belong to different categories in the first clustering result to form a pair of samples with low similarity to the same person.

[0089] Optionally, in order to further increase the amount of difficult samples that can be mined, the server can retrieve images that may be difficult samples in the Body collection in the following manner, so as to provide these images to relevant experts for annotation.

[0090] The server can perform secondary clustering processing on the human features of each image in the Body collection to obtain a third clustering result, and the clustering accuracy of the third clustering result is higher than the clustering accuracy of the first clustering result. Then, the average features corresponding to each of the N (N is an integer greater than 1) categories included in the third clustering result are determined. Furthermore, for each category included in the third clustering result, the rerank distance between the average feature corresponding to the category and the average feature corresponding to each of the remaining N-1 categories is calculated as the rerank distance corresponding to each of the N-1 categories, and the rerank distances corresponding to each of the N-1 categories are sorted in descending order, and the images in the top M (M is an integer greater than or equal to 1 and less than N-1) categories are determined as sample images to be labeled.

[0091] Specifically, the server can perform more fine-grained clustering processing on the human body features corresponding to each image in the Body set to obtain a third clustering result including N categories, and each category in the third clustering result corresponds to a different object; in order to ensure that each category in the third clustering result accurately corresponds to a different object, and that each human body feature in the same category corresponds to the same object, relevant technical personnel can manually intervene in this clustering process.

[0092] After the third clustering result is obtained through clustering processing, the server can calculate the average feature corresponding to each category in the third clustering result using the human features included in the category; in this way, the average features corresponding to each of the N categories in the third clustering result are obtained. Furthermore, for each of the N categories included in the third clustering result, the rerank distance between the average features corresponding to each of the remaining N-1 categories and the average features of the category is calculated as the rerank distance corresponding to each of the N-1 categories. The rerank distances corresponding to each of the N-1 categories are sorted in descending order, and the top M categories are selected. The images corresponding to each of the human features in these M categories are used as sample images to be labeled and pushed to relevant experts for labeling.

[0093] It should be understood that the above-mentioned M value can be set according to actual needs, for example, it can be set to 4 or 5, and this application does not make any specific limitation on the M value.

[0094] After relevant experts obtain the sample images to be annotated, they can manually annotate the non-identical high-similar samples and the identical low-similar samples. Figure 4 The implementation process diagram shown in the figure introduces the process of manually labeling difficult samples. Figure 4 As shown in the figure, assuming that four corresponding sample images to be annotated 402, 403, 404 and 405 are retrieved for image 401, relevant experts can respectively annotate images with the same person orientation (such as Figure 4The upper branch in the middle) and images with different orientations (such as Figure 4 For images 401, 403 and 405 with the same person orientation, relevant experts can manually identify whether images 401, 403 and 405 include the same object, and then combine according to the identification results to obtain two pairs of same-person low-similarity sample pairs 406 and 407, where sample pair 406 includes image 401 and image 405, and sample pair 407 includes image 401 and image 403; for images 401 and 404 with different person orientations, relevant experts can manually identify whether images 401 and image 404 include the same object, and then combine according to the identification results to obtain a pair of non-same-person high-similarity sample pairs 408, where sample pair 408 includes image 401 and image 404.

[0095] It should be understood that Figure 4 The process of manually labeling samples shown is only an example. In actual applications, relevant experts can label the sample images to be labeled in other ways. This application does not impose any limitation on the labeling method.

[0096] The above sample mining method introduces facial feature information into the process of mining difficult samples, and automatically determines non-same-person high-similar samples (i.e., first difficult samples) and same-person low-similar samples (i.e., second difficult samples) based on the difference between the human body feature clustering results and the facial feature clustering results, thereby realizing automatic mining of difficult samples that are helpful for improving the performance of the ReID model. Compared with the implementation method of mining difficult samples in related technologies, the technical solution provided in the embodiment of the present application can mine both non-same-person high-similar samples and same-person low-similar samples, and does not require manual participation, which can effectively improve the mining efficiency of difficult samples.

[0097] In order to further understand the sample mining method provided in the embodiment of the present application, Figure 5 The flowchart shown in FIG. 1 is an overall exemplary introduction to the implementation process of the sample mining method provided in the embodiment of the present application. Figure 5 As shown, the method comprises the following steps:

[0098] Step 501: Get the query collection and gallery collection.

[0099] Step 502: Filter easy samples in the query set.

[0100] In specific implementation, the first threshold T1 can be found through experiments, which can ensure that the accuracy of retrieving gallery top1 based on the query set in the gallery set reaches 97%; the second threshold T2 can be found through experiments, which can ensure that the recall rate of retrieving gallery top5 based on the query set in the gallery set reaches 97%. Retrieve each image in the query set in the gallery set, and calculate the similarity S1 between the gallery top1 image retrieved for each query image and its query image, retain the query images whose S1 is in the interval [T1, T2], and filter out other query images.

[0101] Step 503: extracting human features of each image in the Body set, where the Body set includes images in the gallery set and images in the query set that are retained after filtering.

[0102] Step 504: clustering the human body features of each image in the Body set to obtain a first clustering result.

[0103] Step 505: Perform face detection on each image in the Body set to obtain a face detection result corresponding to each image.

[0104] Step 506: Determine whether the face detection result corresponding to the image indicates that the face in the image can be successfully detected. If not, execute step 507; if yes, execute step 508.

[0105] Step 507: Determine that the image does not participate in the auxiliary calculation, that is, the image is not considered when forming the Face set.

[0106] Step 508: Determine a quality score corresponding to the image according to the blurriness and orientation of the image, and the quality score can represent the quality of face detection in the image.

[0107] Step 509: Determine whether the quality score is greater than a preset score threshold. If not, execute step 510; if yes, execute step 511.

[0108] Step 510: Determine that the image does not participate in the auxiliary calculation, that is, the image is not considered when forming the Face set.

[0109] Step 511: Use images with quality scores greater than a preset score threshold to form a Face set.

[0110] Step 512: Extract facial features of each image in the Face set.

[0111] Step 513: clustering the facial features of each image in the Face set to obtain a second clustering result.

[0112] It should be understood that in actual applications, the branch consisting of step 503 and step 504 and the branch consisting of step 505 to step 513 are two parallel branches, and the server can execute these two branches simultaneously during specific execution.

[0113] Step 514: According to the first clustering result and the second clustering result, the non-identical high-similarity sample pairs and the identical low-similarity sample pairs are automatically screened out.

[0114] Step 515: manually assist in implementing secondary fine-grained clustering of the Body set to obtain a third clustering result.

[0115] Step 516: Calculate the average human features corresponding to each category in the third clustering result.

[0116] Step 517: For each category in the third clustering result, perform rerank search on the remaining N-1 categories, and take images in the top 5 categories as sample images to be labeled.

[0117] Step 518: Push the sample images to be labeled to relevant experts for manual labeling.

[0118] With respect to the sample mining method described above, the present application also provides a corresponding sample mining device so that the above sample mining method can be applied and implemented in practice.

[0119] See also Figure 6 , Figure 6 It is the same as above Figure 2 The sample mining method shown is a schematic diagram of a sample mining device 600 corresponding to the structure of the sample mining method, the sample mining device 600 includes:

[0120] An acquisition module 601 is used to acquire a target image set and a seed image set; the target image set includes images showing a target object to be identified, and the seed image set includes images showing the target object and images showing non-target objects;

[0121] A first image set acquisition module 602, configured to combine the target image set and the seed image set to obtain a first image set;

[0122] A feature extraction module 603, used to extract human body features and facial features of images in the first image set;

[0123] The clustering module 604 is used to perform clustering processing on the human features of the images in the first image set to obtain a first clustering result; and to perform clustering processing on the facial features of the images in the first image set to obtain a second clustering result;

[0124] The difficult sample determination module 605 is used to determine a first difficult sample and a second difficult sample based on the first clustering result and the second clustering result; the first difficult sample includes images whose human body features belong to the same category but whose facial features belong to different categories, and the second difficult sample includes images whose human body features belong to different categories but whose facial features belong to the same category.

[0125] Optional, in Figure 6 Based on the sample excavation device shown, see Figure 7 , Figure 7 FIG. 7 is a schematic diagram of another sample mining device 700 provided in an embodiment of the present application. Figure 7 As shown, the sample mining device 700 also includes:

[0126] A second image set acquisition module 701, configured to form a second image set using images in the first image set whose face detection quality meets a preset standard;

[0127] The feature extraction module 603 is specifically used for:

[0128] Extracting human features of each image in the first image set, and extracting facial features of each image in the second image set;

[0129] The clustering module 604 is specifically used for:

[0130] Clustering is performed on the facial features of each image in the second image set to obtain the second clustering result.

[0131] Optional, in Figure 6 Based on the sample excavation device shown, see Figure 8 , Figure 8 FIG. 8 is a schematic diagram of another sample mining device 800 provided in an embodiment of the present application. Figure 8 As shown, the sample mining device 800 also includes:

[0132] A filtering module 801 is used to filter the target image set to remove images whose correct recognition rate is higher than a preset threshold in the target image set;

[0133] Then the first image set acquisition module 602 is specifically used for:

[0134] The images retained after filtering the target image set are combined with the images in the seed image set to obtain the first image set.

[0135] Optional, in Figure 8 Based on the sample mining device shown, the filtering module 801 is specifically used for:

[0136] Determine a first threshold and a second threshold; when searching the target image set in the seed image set, a retrieval accuracy determined based on the first threshold according to a first preset retrieval condition satisfies a preset accuracy threshold, and a retrieval recall determined based on the second threshold according to a second preset retrieval condition satisfies a preset recall threshold;

[0137] For each image in the target image set, a target image with the highest similarity to the image is retrieved from the seed image set, and it is determined whether the similarity between the target image and the image is in a target interval. If so, the image is retained, otherwise, the image is filtered out; the target interval is determined based on the first threshold and the second threshold.

[0138] Optional, in Figure 6 Based on the sample excavation device shown, see Fig. 9 , Fig. 9 FIG. 9 is a schematic diagram of another sample mining device 900 provided in an embodiment of the present application. Fig. 9 As shown, the sample mining device 900 also includes:

[0139] A secondary clustering module 901 is used to perform secondary clustering processing on the human features of each image in the first image set to obtain a third clustering result; the clustering accuracy of the third clustering result is higher than the clustering accuracy of the first clustering result;

[0140] An average feature determination module 902 is used to determine the average features corresponding to each of the N categories included in the third clustering result, where N is an integer greater than 1;

[0141] The module 903 for determining samples to be labeled is used to calculate, for each category included in the third clustering result, the rerank distance between the average feature corresponding to each of the remaining N-1 categories and the average feature corresponding to the category, as the rerank distance corresponding to each of the N-1 categories; sort the rerank distances corresponding to each of the N-1 categories in descending order, and determine that the images in the top M categories are the sample images to be labeled; M is an integer less than N-1 and greater than or equal to 1.

[0142] Optional, in Figure 7 Based on the sample mining device shown, the second image set acquisition module 701 is specifically used for:

[0143] Performing face detection on each image in the first image set to determine a face detection result corresponding to each image; the face detection result is used to indicate whether a face can be successfully detected in the image;

[0144] The face detection results represent images in which faces can be successfully detected, and the face detection results represent images in which faces cannot be successfully detected;

[0145] For the images retained in the first image set, determining quality scores corresponding to the images;

[0146] From the images retained in the first image set, images with quality scores greater than a preset score threshold are selected to form the second image set.

[0147] Optional, in Figure 6 or Fig. 9 Based on the sample mining device shown, the clustering algorithm used in the clustering process includes any one of the following: a rank-order clustering algorithm and a graph convolution clustering algorithm.

[0148] In the process of mining difficult samples, the above-mentioned sample mining device introduces facial feature information into it, and automatically determines non-identical high-similar samples (i.e., first difficult samples) and same-person low-similar samples (i.e., second difficult samples) based on the difference between the human body feature clustering results and the facial feature clustering results, thereby realizing automatic mining of difficult samples that are helpful for improving the performance of the ReID model. Compared with the implementation method of mining difficult samples in related technologies, the technical solution provided in the embodiment of the present application can mine both non-identical high-similar samples and same-person low-similar samples, and does not require manual participation, which can effectively improve the mining efficiency of difficult samples.

[0149] The embodiment of the present application also provides a device for mining samples, which may specifically be a terminal or a server. The above-mentioned device provided in the embodiment of the present application will be introduced below from the perspective of hardware entity.

[0150] See also Fig.10 , Fig.10 This is a schematic diagram of the structure of the terminal device provided in the embodiment of the present application. Fig.10 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), a sales terminal (English full name: Point of Sales, English abbreviation: POS), a car computer, etc., taking the terminal as a computer as an example:

[0151] Fig.10 FIG. 1 is a block diagram showing a partial structure of a computer related to a terminal provided in an embodiment of the present application. Fig.10 The computer includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Fig.10 The computer structure shown in the figure does not constitute a limitation of the computer, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0152] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the computer by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the computer (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0153] The processor 1080 is the control center of the computer. It uses various interfaces and lines to connect various parts of the entire computer. It executes various functions of the computer and processes data by running or executing software programs and / or modules stored in the memory 1020 and calling data stored in the memory 1020. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0154] In the embodiment of the present application, the processor 1080 included in the terminal also has the following functions:

[0155] Acquire a target image set and a seed image set; the target image set includes images showing a target object to be identified, and the seed image set includes images showing the target object and images showing non-target objects;

[0156] Combining the target image set and the seed image set to obtain a first image set;

[0157] Extracting human body features and facial features of images in the first image set;

[0158] Performing clustering processing on human features of images in the first image set to obtain a first clustering result; performing clustering processing on facial features of images in the first image set to obtain a second clustering result;

[0159] A first difficult sample and a second difficult sample are determined according to the first clustering result and the second clustering result; the first difficult sample includes images whose body features belong to the same category but whose facial features belong to different categories, and the second difficult sample includes images whose body features belong to different categories but whose facial features belong to the same category.

[0160] Optionally, the processor 1080 is also used to execute the steps of any implementation method of the sample mining method provided in the embodiments of the present application.

[0161] See also Fig.11 , Fig.11 A schematic diagram of the structure of a server 1100 provided for an embodiment of the present application. The server 1100 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1122 (for example, one or more processors) and a memory 1132, and one or more storage media 1130 (for example, one or more mass storage devices) storing application programs 1142 or data 1144. Among them, the memory 1132 and the storage medium 1130 may be temporary storage or permanent storage. The program stored in the storage medium 1130 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1122 may be configured to communicate with the storage medium 1130 to execute a series of instruction operations in the storage medium 1130 on the server 1100.

[0162] The server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input and output interfaces 1158, and / or one or more operating systems 1141, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0163] The steps performed by the server in the above embodiment can be based on the Fig.11The server structure shown.

[0164] The CPU 1122 is used to execute the following steps:

[0165] Acquire a target image set and a seed image set; the target image set includes images showing a target object to be identified, and the seed image set includes images showing the target object and images showing non-target objects;

[0166] Combining the target image set and the seed image set to obtain a first image set;

[0167] Extracting human body features and facial features of images in the first image set;

[0168] Performing clustering processing on human features of images in the first image set to obtain a first clustering result; performing clustering processing on facial features of images in the first image set to obtain a second clustering result;

[0169] A first difficult sample and a second difficult sample are determined according to the first clustering result and the second clustering result; the first difficult sample includes images whose body features belong to the same category but whose facial features belong to different categories, and the second difficult sample includes images whose body features belong to different categories but whose facial features belong to the same category.

[0170] Optionally, the CPU 1122 may also be used to execute steps of any implementation of the sample mining method provided in the embodiments of the present application.

[0171] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program, wherein the computer program is used to execute any one of the implementations of a sample mining method described in the aforementioned embodiments.

[0172] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute any one of the implementations of the sample mining method described in the aforementioned embodiments.

[0173] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0174] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0175] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0176] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0177] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store computer programs.

[0178] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0179] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A sample mining method, characterized in that: The method comprises: Acquire a target image set and a seed image set; the target image set includes images showing a target object to be identified, and the seed image set includes images showing the target object and images showing non-target objects; Combining the target image set and the seed image set to obtain a first image set; Extracting human body features and facial features of images in the first image set; Performing clustering processing on human features of images in the first image set to obtain a first clustering result; performing clustering processing on facial features of images in the first image set to obtain a second clustering result; A first difficult sample and a second difficult sample are determined according to the first clustering result and the second clustering result; the first difficult sample includes images whose body features belong to the same category but whose facial features belong to different categories, and the second difficult sample includes images whose body features belong to different categories but whose facial features belong to the same category.

2. The method according to claim 1, characterized in that Before extracting the human body features and facial features of the images in the first image set, the method further includes: Using images in the first image set whose face detection quality meets a preset standard to form a second image set; Then the step of extracting the human body features and facial features of the images in the first image set includes: Extracting human features of each image in the first image set, and extracting facial features of each image in the second image set; Then clustering the facial features of the images in the first image set to obtain a second clustering result includes: Clustering is performed on the facial features of each image in the second image set to obtain the second clustering result.

3. The method according to claim 1, characterized in that Before combining the target image set and the seed image set to obtain the first image set, the method further includes: Performing filtering processing on the target image set to filter out images in the target image set whose correct recognition rate is higher than a preset threshold; Then the combining the target image set and the seed image set to obtain a first image set includes: The images retained after filtering the target image set are combined with the images in the seed image set to obtain the first image set.

4. The method according to claim 3, characterized in that The filtering process on the target image set to filter out images in the target image set whose correct recognition rate is higher than a preset threshold includes: Determine a first threshold and a second threshold; when searching the target image set in the seed image set, a retrieval accuracy determined based on the first threshold according to a first preset retrieval condition satisfies a preset accuracy threshold, and a retrieval recall determined based on the second threshold according to a second preset retrieval condition satisfies a preset recall threshold; For each image in the target image set, a target image with the highest similarity to the image is retrieved from the seed image set, and it is determined whether the similarity between the target image and the image is in a target interval. If so, the image is retained, otherwise, the image is filtered out; the target interval is determined based on the first threshold and the second threshold.

5. The method according to claim 1, characterized in that The method further comprises: Performing secondary clustering processing on the human features of each image in the first image set to obtain a third clustering result; the clustering accuracy of the third clustering result is higher than the clustering accuracy of the first clustering result; Determine the average feature corresponding to each of the N categories included in the third clustering result, where N is an integer greater than 1; For each category included in the third clustering result, calculate the rerank distance between the average feature corresponding to each of the remaining N-1 categories and the average feature corresponding to the category as the rerank distance corresponding to each of the N-1 categories; sort the rerank distances corresponding to each of the N-1 categories in descending order, and determine that the images in the top M categories are the sample images to be labeled; M is an integer less than N-1 and greater than or equal to 1; By adopting a manual labeling method, non-same-person high-similar samples and same-person low-similar samples are labeled from the sample images to be labeled, so as to increase the amount of difficult samples.

6. The method according to claim 2, characterized in that Determine the face detection quality of the images in the first image set by: Performing face detection on each image in the first image set to determine a face detection result corresponding to each image; the face detection result is used to indicate whether a face can be successfully detected in the image; The face detection results represent images in which faces can be successfully detected, and the face detection results represent images in which faces cannot be successfully detected; For the images retained in the first image set, determining quality scores corresponding to the images; Then, the second image set is formed by using the images whose face detection quality meets the preset standard in the first image set, including: From the images retained in the first image set, images with quality scores greater than a preset score threshold are selected to form the second image set.

7. The method according to claim 1 or 5, characterized in that: The clustering algorithm used in the clustering process includes any one of the following: a rank-order clustering algorithm and a graph convolution clustering algorithm.

8. A sample excavation device, characterized in that: The device comprises: An acquisition module, used to acquire a target image set and a seed image set; the target image set includes images showing a target object to be identified, and the seed image set includes images showing the target object and images showing non-target objects; A first image set acquisition module, used for combining the target image set and the seed image set to obtain a first image set; A feature extraction module, used to extract human body features and facial features of images in the first image set; A clustering module, configured to perform clustering processing on human features of images in the first image set to obtain a first clustering result; and perform clustering processing on facial features of images in the first image set to obtain a second clustering result; A difficult sample determination module is used to determine a first difficult sample and a second difficult sample based on the first clustering result and the second clustering result; the first difficult sample includes images whose human body features belong to the same class but whose facial features belong to different classes, and the second difficult sample includes images whose human body features belong to different classes but whose facial features belong to the same class.

9. The device according to claim 8, characterized in that The device also includes: A second image set acquisition module, configured to form a second image set using images in the first image set whose face detection quality meets a preset standard; The feature extraction module is specifically used for: Extracting human features of each image in the first image set, and extracting facial features of each image in the second image set; The clustering module is specifically used for: Clustering is performed on the facial features of each image in the second image set to obtain the second clustering result.

10. The device according to claim 8, characterized in that The device also includes: A filtering module, used to filter the target image set to remove images whose correct recognition rate is higher than a preset threshold in the target image set; Then the first image set acquisition module is specifically used to: The images retained after filtering the target image set are combined with the images in the seed image set to obtain the first image set.

11. The device according to claim 10, characterized in that The filtering module is specifically used for: Determine a first threshold and a second threshold; when searching the target image set in the seed image set, a retrieval accuracy determined based on the first threshold according to a first preset retrieval condition satisfies a preset accuracy threshold, and a retrieval recall determined based on the second threshold according to a second preset retrieval condition satisfies a preset recall threshold; For each image in the target image set, a target image with the highest similarity to the image is retrieved from the seed image set, and it is determined whether the similarity between the target image and the image is in a target interval. If so, the image is retained, otherwise, the image is filtered out; the target interval is determined based on the first threshold and the second threshold.

12. The device according to claim 8, characterized in that The device also includes: A secondary clustering module, used for performing secondary clustering processing on the human features of each image in the first image set to obtain a third clustering result; the clustering accuracy of the third clustering result is higher than the clustering accuracy of the first clustering result; An average feature determination module, used to determine the average features corresponding to each of the N categories included in the third clustering result, where N is an integer greater than 1; a sample-to-be-annotated determination module, configured to calculate, for each category included in the third clustering result, a rerank distance between the average feature corresponding to each of the remaining N-1 categories and the average feature corresponding to the category as the rerank distance corresponding to each of the N-1 categories; sort the rerank distances corresponding to each of the N-1 categories in descending order, and determine that images in the top M categories are sample images to be annotated; wherein M is an integer less than N-1 and greater than or equal to 1; The device is also used to: use manual labeling to label non-same-person high-similar samples and same-person low-similar samples from the sample images to be labeled, so as to increase the amount of difficult samples.

13. The device according to claim 9, characterized in that The second image set acquisition module is specifically used to: Performing face detection on each image in the first image set to determine a face detection result corresponding to each image; the face detection result is used to indicate whether a face can be successfully detected in the image; The face detection results represent images in which faces can be successfully detected, and the face detection results represent images in which faces cannot be successfully detected; For the images retained in the first image set, determining quality scores corresponding to the images; From the images retained in the first image set, images with quality scores greater than a preset score threshold are selected to form the second image set.

14. The device according to claim 8 or 12, characterized in that The clustering algorithm used in the clustering process includes any one of the following: a rank-order clustering algorithm and a graph convolution clustering algorithm.

15. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer programs; The processor is used to execute the sample mining method according to any one of claims 1 to 7 according to the computer program.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the sample mining method according to any one of claims 1 to 7.

17. A computer program product comprising instructions, characterized in that When it runs on a computer, the computer is enabled to execute the sample mining method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Image processing method and device, electronic equipment and storage medium

    CN110569777A

  • Sample classification method and related device

    CN110569918A