Image data cleaning method, device and storage medium

Abstract

This application discloses an image data cleaning method, apparatus, and storage medium. The method includes: performing human detection and face detection on each image to obtain multiple human image images and face images; generating multiple movement trajectories based on the location information corresponding to each human image, determining face images matching the movement trajectories, and obtaining a set of matched faces; extracting high-quality face images from each set of matched faces; clustering each high-quality face image, and merging face images matching the movement trajectories belonging to the same cluster to obtain cleaned face images. By matching discrete face images with continuous human movement trajectories, the possibility of incorrectly associating faces of different people is greatly reduced. Through high-quality face image clustering, the method automatically cleans, filters, and merges a set of face images of the same person from the original video / image stream, improving data processing efficiency and accuracy.

Description

Technical Field

[0001] This application relates to the field of image management technology, and in particular to an image data cleaning method, apparatus and storage medium. Background Technology

[0002] Data cleaning refers to the process of extracting desired data from raw data. It is a key data preprocessing step that ensures data quality and improves the performance of subsequent algorithms.

[0003] Traditional data cleaning methods often rely on manual screening or simple rule matching and data clustering. When faced with massive, high-dimensional complex data (such as images and videos), they often suffer from low processing efficiency, low accuracy, and high cost. This is especially true in the field of face recognition, where datasets often contain mislabeled, low-quality, or duplicate images, which can seriously affect the accuracy and robustness of data cleaning.

[0004] Therefore, improving the accuracy and efficiency of image cleaning is one of the problems that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides at least one image data cleaning method, apparatus, and storage medium.

[0006] The first aspect of this application provides an image data cleaning method, which includes: performing human detection and face detection on each image to obtain multiple human image images and face images; generating multiple motion trajectories based on the location information corresponding to each human image, determining face images that match the motion trajectories, and obtaining a set of matching faces corresponding to the motion trajectories; wherein, the higher the probability that the face image and the human image corresponding to the motion trajectory belong to the same person, the higher the matching degree between the face image and the motion trajectory; extracting face images whose quality scores meet preset scoring conditions from each set of matching faces to obtain high-quality face images corresponding to each motion trajectory; clustering the high-quality face images corresponding to each motion trajectory, and obtaining motion trajectories belonging to the same cluster based on the clustering results; merging the face images matched by motion trajectories belonging to the same cluster to obtain cleaned face images.

[0007] In one embodiment, the image is a video frame from a video taken within a preset time period; clustering the high-quality face images corresponding to each movement trajectory, and obtaining movement trajectories belonging to the same cluster based on the clustering results, includes: calculating the image similarity between the high-quality face images corresponding to each movement trajectory extracted from the video to obtain a first cluster similarity; classifying the high-quality face images with the first cluster similarity higher than a preset similarity threshold into the same cluster to obtain a video-level clustering result; and determining the movement trajectories belonging to the same cluster based on the video-level clustering result.

[0008] In one embodiment, the video is associated with a corresponding user account; based on the video-level clustering results, the movement trajectories belonging to the same cluster are determined, including: merging the movement trajectories corresponding to high-quality face images in the same cluster based on the video-level clustering results to obtain video-level movement trajectories; for the video-level movement trajectories corresponding to all videos associated with the same user account, reselecting face images whose quality scores meet preset scoring conditions to obtain reselected high-quality face images; calculating the image similarity between the reselected high-quality face images to obtain a second clustering similarity; classifying high-quality face images with a second clustering similarity higher than a preset similarity threshold into the same cluster to obtain account-level clustering results; and using the movement trajectories corresponding to high-quality face images belonging to the same cluster in the video-level clustering results as movement trajectories belonging to the same cluster.

[0009] In one embodiment, the method further includes: obtaining a set of failed face images by matching the movement trajectories of high-quality face images that failed clustering; and removing abnormal face images from the set of failed face images to obtain cleaned face images.

[0010] In one embodiment, high-quality face images with image similarity higher than a preset similarity threshold are grouped into the same cluster; a set of failed face images is obtained based on the matching face images corresponding to the movement trajectories of the high-quality face images that failed to cluster, including: obtaining high-quality face images whose image similarity with other high-quality face images is lower than a preset similarity threshold, thus obtaining failed high-quality face images; obtaining face images corresponding to the movement trajectories of the failed high-quality face images, thus obtaining a set of failed face images.

[0011] In one embodiment, the failed face image set includes a manually cleaned set and an automatically cleaned set. Obtaining the face image matching the movement trajectory of a failed high-quality face image to form the failed face image set includes: if the image similarity corresponding to the failed high-quality face image is within a fuzzy similarity threshold range, then adding the face image matching the movement trajectory of the failed high-quality face image to the manually cleaned set; if the image similarity corresponding to the failed high-quality face image is within a low similarity threshold range, then detecting whether the failed high-quality face image is similar to any face in the base database; if not similar, adding the face image matching the movement trajectory of the failed high-quality face image to the automatically cleaned set; if similar, adding the face image matching the movement trajectory of the failed high-quality face image to the manually cleaned set; wherein the maximum value of the fuzzy similarity threshold range is less than a preset similarity threshold, and the minimum value of the fuzzy similarity threshold range is greater than the maximum value of the low similarity threshold range; before removing abnormal face images from the failed face image set, the method further includes: sending the manually cleaned set to a manual processing end, so that the manual processing end can determine and merge face images belonging to the same person.

[0012] In one embodiment, before extracting face images whose quality scores meet preset scoring conditions from each matched face set to obtain high-quality face images corresponding to each movement trajectory, the method further includes: filtering face images in the matched face set whose quality scores are lower than a preset scoring threshold; and / or filtering face images in the matched face set whose face angles do not meet preset angle conditions; and / or filtering matched face sets whose number of face images is lower than a preset number threshold.

[0013] In one embodiment, the method further includes: assigning a unified and unique face identifier to face images with matching movement trajectories belonging to the same cluster; extracting image features from the cleaned face images to obtain face feature vectors; and obtaining business metadata associated with the cleaned face images; associating and storing the face feature vectors and unique face identifiers in a vector database; and associating and storing the business metadata and unique face identifiers in a relational database.

[0014] A second aspect of this application provides an image data cleaning apparatus, comprising: an image detection module for performing human figure detection and face detection on each image to obtain multiple human figure images and face images; a matching module for generating multiple movement trajectories based on the location information corresponding to each human figure image, determining face images that match the movement trajectories, and obtaining a set of matched faces corresponding to the movement trajectories; wherein, the higher the probability that the face image and the human figure image corresponding to the movement trajectory belong to the same person, the higher the matching degree between the face image and the movement trajectory; a selection module for extracting face images whose quality scores meet preset scoring conditions from each set of matched faces to obtain high-quality face images corresponding to each movement trajectory; a clustering module for clustering the high-quality face images corresponding to each movement trajectory, and obtaining movement trajectories belonging to the same cluster based on the clustering results; and a merging module for merging the face images matched by movement trajectories belonging to the same cluster to obtain cleaned face images.

[0015] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the image data cleaning method described above.

[0016] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described image data cleaning method.

[0017] The above scheme obtains multiple human and face images by performing human and face detection on each image separately. Based on the location information corresponding to each human image, multiple movement trajectories are generated. Face images matching the movement trajectories are determined, resulting in a set of matching faces for each movement trajectory. Matching discrete face images with continuous human movement trajectories greatly reduces the possibility of incorrectly associating faces of different people. Then, face images whose quality scores meet preset scoring conditions are extracted from each set of matching faces, resulting in high-quality face images corresponding to each movement trajectory. This yields more representative and discriminative clustering objects. The high-quality face images corresponding to each movement trajectory are clustered, and movement trajectories belonging to the same cluster are obtained based on the clustering results. Face images matching movement trajectories belonging to the same cluster are merged to obtain cleaned face images. Different movement trajectories belonging to the same person are grouped into the same cluster, completing the merging of similar faces. This achieves automatic cleaning, filtering, and merging of a set of face images of the same person from the original video / image stream, improving data processing efficiency and accuracy.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0020] Figure 1 This is a schematic diagram illustrating the implementation environment of the solution in an exemplary embodiment of this application; Figure 2 This is a flowchart illustrating an exemplary embodiment of the image data cleaning method of this application; Figure 3 This is a schematic diagram illustrating face-human matching as an exemplary embodiment of this application; Figure 4 This is a schematic diagram illustrating face image cleaning in an exemplary embodiment of this application; Figure 5 This is a block diagram illustrating an image data cleaning apparatus according to an exemplary embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application; Figure 7 This is a schematic diagram illustrating the structure of a computer-readable storage medium, as shown in an exemplary embodiment of this application. Detailed Implementation

[0021] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0022] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0023] In this document, the term "and / or" is merely a description of the association information of related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0024] The image data cleaning method provided in the embodiments of this application will be described below.

[0025] Please refer to Figure 1 , Figure 1This is a schematic diagram illustrating an implementation environment of the scheme according to an exemplary embodiment of this application. The implementation environment may include an image source 110 and a server 120, which are interconnected.

[0026] The number of image source devices 110 can be one or more. Image source devices 110 can be cameras, smartphones, tablets, laptops, desktop computers, etc., but are not limited to these.

[0027] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0028] In one example, server 120 can perform image data cleaning on images obtained from image source 110 to obtain cleaned face images. Server 120 can store the cleaned face images locally, send them back to image source 110, or transmit them to other terminals.

[0029] In one example, a client running a target application is installed on the image source 110. This target application may be an application that provides image data cleaning functions, and the image data is cleaned based on this target application. The server 120 may be a backend server for this target application, used to provide backend services to the client of the target application.

[0030] The image data cleaning method provided in this application can be executed by the image source 110, such as the client of the target application installed and running in the image source 110, or by the server 120, or by the image source 110 and the server 120 interacting and cooperating to execute each step, that is, some steps of the method are executed by the image source 110 and other steps are executed by the server 120, or by other types of electronic devices. This application does not limit the type of the executing entity.

[0031] It should be noted that in the specific embodiments of this application, data related to human figures, faces, and user accounts are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0032] Please see Figure 2 , Figure 2 This is a flowchart illustrating an exemplary embodiment of the image data cleaning method of this application, as shown below. Figure 2 As shown, the image data cleaning method includes at least steps S210 to S250, which are described in detail below: Step S210: Perform human detection and face detection on each image to obtain multiple human images and face images.

[0033] The image can be a video frame from a video or a captured image; this application does not limit the scope of the image.

[0034] The image may contain a face, a human figure, or both.

[0035] Human detection is performed on the image to obtain the image coordinates of human regions within the image. One or more human images are then extracted from the image based on these coordinates. Similarly, face detection is performed on the image to obtain the image coordinates of face regions within the image. One or more face images are then extracted from the image based on these coordinates. It should be noted that human images may or may not contain face images; for example, if the face region is occluded, it will not contain a face image.

[0036] Among them, pre-trained object detection models can be used to perform human detection and face detection respectively. The object detection models can be implemented based on YOLO, Faster R-CNN, or RetinaNet model network architectures.

[0037] Optionally, pedestrian re-identification (REID) features of human images can be further extracted to facilitate subsequent generation of movement trajectories for human images; the quality score of face images, and / or face angle, and / or face key points, and / or face feature vectors can also be calculated to facilitate subsequent face image screening and storage. Among these, the quality score of face images can be calculated based on image clarity, and / or image brightness, and / or face angle, etc.

[0038] Step S220: Generate multiple movement trajectories based on the location information corresponding to each human figure image, determine the face images that match the movement trajectories, and obtain the set of matching faces corresponding to the movement trajectories.

[0039] The higher the probability that the face image and the human figure image corresponding to the movement trajectory belong to the same person, the higher the matching degree between the face image and the movement trajectory.

[0040] Specifically, obtain the location information corresponding to each human figure image.

[0041] For example, the physical deployment location of the shooting device corresponding to the captured human figure image can be obtained, and this physical deployment location can be used as the location information corresponding to the human figure image.

[0042] For example, by using pre-calibrated camera intrinsic and extrinsic parameters, the coordinates in the pixel coordinate system of the image can be mapped to the coordinates in the real-world coordinate system to obtain the position information corresponding to the human figure image.

[0043] The method for obtaining the location information corresponding to the human figure image can be flexibly selected according to the actual application scenario, and this application does not limit it.

[0044] Then, the human figures of the same person in different images are linked together, and a movement trajectory is generated based on the location information corresponding to each human figure.

[0045] Next, calculate the probability that the face image and the human figure image corresponding to the movement trajectory belong to the same person.

[0046] For example, please see Figure 3 , Figure 3 This is a schematic diagram illustrating face and human figure matching as an exemplary embodiment of this application, such as... Figure 3 As shown, key points are extracted from the face image to obtain face key points, and key points are extracted from the human figure image in the movement trajectory to obtain human figure key points. The coordinates of both face key points and human figure key points are unified into the image coordinate system. Face-human key point pairs are constructed by combining face key points and human figure key points. The face-human key point pairs are input into a pre-trained matching classification model to obtain the matching result between the face image and the human figure image output by the matching classification model. The matching result can be a specific matching degree or a representation of whether a match is successful (e.g., 0 indicates a successful match, and 1 indicates a failed match). The matching classification model can be implemented based on a lightweight multilayer perceptron (MLP) classification model or based on other neural network structures; this application does not limit this.

[0047] For example, we can extract image features from a face image to obtain a face feature vector, extract image features from a human figure image in a moving trajectory to obtain a human figure feature vector, calculate the vector similarity between the face feature vector and the human figure feature vector, and obtain the probability that the face image and the human figure image belong to the same person. The vector similarity is positively correlated with the probability of belonging to the same person.

[0048] For example, the spatiotemporal context information of the face image and the human figure image in the movement trajectory can be obtained separately, and the probability that the face image and the human figure image in the movement trajectory belong to the same person can be calculated based on the spatiotemporal context information. For example, if the human body image and face image captured by the same camera or adjacent cameras in a very short period of time belong to the same person, the probability is relatively high.

[0049] Of course, the probability that the face image and the human figure image in the movement trajectory belong to the same person can also be calculated by combining the above-mentioned multiple embodiments. This application does not limit the specific calculation method.

[0050] Alternatively, the calculation can be performed on all human figures in the face image and the movement trajectory, and the lowest, middle, highest, or average value can be selected as the final calculated probability of belonging to the same person. Alternatively, the calculation can be performed on some human figures in the face image and the movement trajectory, such as calculating the face image and human figures in the movement trajectory in a preset pose, or calculating the face image and human figures in the movement trajectory whose occluded area is smaller than a preset area, or calculating the face image and human figures in the movement trajectory whose clarity is higher than a preset clarity threshold. This application does not limit this.

[0051] Based on the probability that the face image and the human figure image corresponding to the movement trajectory belong to the same person, it is determined whether the face image and the movement trajectory match. For example, if the probability that the face image and the human figure image in the movement trajectory belong to the same person is greater than a preset probability threshold, it is judged as a high match, and the face image matches the movement trajectory; or, for any face image, the movement trajectory with the highest probability of belonging to the same person is determined. If the probability corresponding to the movement trajectory is greater than a preset probability threshold, it is judged as a high match, and the face image matches the movement trajectory.

[0052] Add the face image that matches the movement trajectory to the matching face set corresponding to that movement trajectory.

[0053] Step S230: Extract the face images that meet the preset scoring conditions from each matching face set to obtain the high-quality face image corresponding to each movement trajectory.

[0054] After obtaining the set of matching faces corresponding to each movement trajectory, for each set of matching faces, select the face images whose quality scores meet the preset scoring conditions from the set of matching faces and use them as the high-quality face images corresponding to that movement trajectory.

[0055] For example, the higher the image clarity, and / or the smaller the absolute value of the difference between the image brightness and the preset brightness value, and / or the smaller the absolute value of the difference between the face angle and the preset angle, the higher the quality score of the face image. Conversely, the lower the image clarity, and / or the larger the absolute value of the difference between the image brightness and the preset brightness value, and / or the larger the absolute value of the difference between the face angle and the preset angle, the lower the quality score of the face image.

[0056] The face images for the preset scoring conditions can be the face images with the highest quality scores, or face images with quality scores greater than a preset threshold, or the top N face images after sorting the face images in descending order according to their quality scores. This application does not limit the preset scoring conditions.

[0057] Step S240: Cluster the high-quality face images corresponding to each movement trajectory, and obtain the movement trajectories belonging to the same cluster based on the clustering results.

[0058] The high-quality face images corresponding to each selected movement trajectory are clustered. The clustering algorithm used can be density-based spatial clustering of applications with noise (DBSCAN), hierarchical clustering, or k-means clustering algorithm, etc. This application does not limit the clustering algorithm used.

[0059] By clustering, similar high-quality face images are grouped into the same cluster. Based on the movement trajectories corresponding to each high-quality face image in the same cluster, the movement trajectories belonging to the same cluster can be obtained.

[0060] Step S250: Merge face images with matching movement trajectories belonging to the same cluster to obtain cleaned face images.

[0061] After obtaining the movement trajectories belonging to the same cluster, the face images matched by each movement trajectory in the same cluster are merged. By merging similar face images, the cleaned face image is obtained.

[0062] For example, the unique face identifiers corresponding to the face images matched by each movement trajectory in the same cluster can be unified to distinguish the face images of different pedestrians; or the face images matched by each movement trajectory in the same cluster can be added to a face image set, with different face image sets corresponding to different pedestrians. This application does not limit the specific implementation of the merging.

[0063] This application significantly reduces the possibility of misassociating faces of different people by matching discrete face images with continuous human movement trajectories. It then scores the quality of faces in each matched face set and filters out high-quality face images that meet certain criteria. These high-quality face images are more representative and discriminative. Furthermore, it clusters the high-quality face images corresponding to each movement trajectory, grouping different movement trajectories belonging to the same person into the same cluster, thus merging similar faces. This achieves automatic cleaning, filtering, and merging of face image sets of the same person from the original video / image stream, improving data processing efficiency and accuracy.

[0064] The following describes some embodiments of this application in detail.

[0065] In some implementations, after obtaining a face image, the face image and the face information of the face image are associated and stored.

[0066] For example, facial information includes one or more combinations of the following information items: 'account_name':account,# The associated user account 'video_name':video_name,# The name of the video to which this video belongs. 'frame_num':frame_num,# Video frame number 'quality': quality, # Face quality score 'p_pred_deg': p_pred_deg, # The angle of the face's tilt. 'y_pred_deg': y_pred_deg, # The angle of the face yaw. 'r_pred_deg': r_pred_deg, # The angle of the face roll angle 'bbox':bbox,# Face detection box 'feat': feat, # Facial feature vector 'warped_img_112': warped_img_112 # The image name of the face image.

[0067] By pre-storing facial information, subsequent image matching, filtering, clustering, and other operations are facilitated.

[0068] In some implementations, before performing step S230 to extract face images whose quality scores meet preset scoring conditions from each matched face set and obtain high-quality face images corresponding to each movement trajectory, the face images are also filtered to remove low-quality images.

[0069] For example, the filtering conditions include one or more combinations of the following conditions: Filter out face images in the matched face set whose quality score is lower than a preset score threshold; Filter and match face images in the face set whose face angles do not meet the preset angle conditions; The set of matched faces that is filtered out when the number of face images is less than a preset threshold.

[0070] For example, consecutive video frames are used as images to be processed. Face images and human figures are detected, and motion trajectories are generated. Face images matching the motion trajectories are obtained, resulting in a set of matching faces corresponding to the motion trajectories. Based on the facial information corresponding to the face images, the face images in each set of matching faces are filtered. Specifically: face images with a quality score lower than a preset score threshold are filtered; using the pitch, yaw, and roll angles of the face image in the current video frame as a benchmark, the changes in the pitch, yaw, and roll angles of the face images in the adjacent frames corresponding to the current frame are calculated. If the change in angles is higher than a preset angle threshold, the face image is selected; otherwise, it is filtered; the number of face images in the set of matching faces after the above filtering is counted. If the number is lower than a preset number threshold, the set of matching faces is filtered; otherwise, it is retained.

[0071] Then, extract the face images that meet the preset scoring conditions from each matching face set to obtain the high-quality face images corresponding to each movement trajectory, and cluster the high-quality face images corresponding to each movement trajectory.

[0072] Video-level clustering and / or account-level clustering can be performed. Video-level clustering refers to clustering facial images extracted from a single video, while account-level clustering refers to clustering facial images extracted from all videos and / or captured images associated with the same user account. Of course, facial images extracted from all existing videos and / or captured images can also be clustered directly. The specific clustering method can be flexibly selected according to the actual application scenario, and this application does not limit it in this regard.

[0073] In some implementations, the images are video frames from a video taken within a preset time period; step S240 involves clustering the high-quality face images corresponding to each movement trajectory, and obtaining movement trajectories belonging to the same cluster based on the clustering results, including the following steps: Step S241: For each high-quality face image corresponding to the movement trajectory extracted from the video, calculate the image similarity between each high-quality face image to obtain the first cluster similarity.

[0074] Step S242: Assign high-quality face images with a first cluster similarity higher than a preset similarity threshold to the same cluster to obtain video-level clustering results.

[0075] The preset similarity threshold can be flexibly set based on experience, and this application does not impose any restrictions on it.

[0076] Step S243: Based on the video-level clustering results, determine the movement trajectories belonging to the same cluster.

[0077] The movement trajectory belonging to the same cluster can be obtained directly from the movement trajectory corresponding to the high-quality face image belonging to the same cluster in the video-level clustering results; however, since the probability of the same face appearing in videos under the same user account is relatively high, it is also possible to perform video-level clustering on each video separately to obtain video-level clustering results, and then perform account-level clustering.

[0078] For example, the video is associated with a corresponding user account; step S243, based on the video-level clustering results, determines the movement trajectory belonging to the same cluster, including the following steps: Step S2431: Based on the video-level clustering results, merge the motion trajectories corresponding to high-quality face images in the same cluster to obtain the video-level motion trajectory.

[0079] Multiple movement trajectories belonging to the same person will be merged into a single movement trajectory.

[0080] For example, each movement trajectory is marked with a trajectory identifier track_id. The track_id of the movement trajectories corresponding to high-quality face images in the same cluster is unified, that is, the movement trajectories corresponding to high-quality face images in the same cluster use the same track_id.

[0081] Step S2432: For all video-level motion trajectories associated with the same user account, reselect face images whose quality scores meet the preset scoring conditions to obtain reselected high-quality face images.

[0082] After merging video-level motion trajectories, for each motion trajectory in the new motion trajectory set, face images that meet the preset scoring conditions are re-extracted.

[0083] Step S2433: Calculate the image similarity between the reselected high-quality face images to obtain the second cluster similarity.

[0084] Step S2434: Assign high-quality face images with a second cluster similarity higher than the preset similarity threshold to the same cluster to obtain account-level clustering results.

[0085] The preset similarity threshold in step S2434 and the preset similarity threshold in step S242 can be the same or different.

[0086] Step S2435: Take the movement trajectory corresponding to the high-quality face image belonging to the same cluster in the video-level clustering results as the movement trajectory belonging to the same cluster.

[0087] Of course, video-level clustering can be skipped and account-level clustering can be performed directly. That is, for all videos associated with the same user account, the extracted movement trajectories are used to select face images that meet the preset scoring conditions to obtain high-quality face images, and then the high-quality face images are clustered.

[0088] Then, face images with matching movement trajectories belonging to the same cluster are merged.

[0089] For example, hierarchical clustering using a similarity threshold yields a cluster label for each movement trajectory, indicating its cluster. The frequency of each cluster label is counted, and labels with a frequency greater than 1 are selected. The movement trajectories corresponding to these labels are the movement trajectories of similar faces to be merged. Each movement trajectory to be merged is iterated through, finding the unique face identifier for all face images corresponding to that trajectory. Using the first unique face identifier as a reference, all other unique face identifiers are modified to match it. After merging, the above steps are repeated until a label with a frequency greater than 1 is found to be 0, at which point the recursion terminates.

[0090] In some implementations, the method further includes: obtaining a set of failed face images by matching the movement trajectories of high-quality face images that failed clustering; and removing abnormal face images from the set of failed face images to obtain cleaned face images.

[0091] High-quality face images that fail to cluster are those that cannot be classified into any cluster.

[0092] High-quality face images that fail to cluster include face images whose content makes it difficult to extract effective features (such as excessive blurring, low contrast, a lot of noise, occlusion, etc.), and / or face images that differ too much from other images to form an independent category.

[0093] For example, high-quality face images with image similarity higher than a preset similarity threshold are grouped into the same cluster; high-quality face images with image similarity lower than the preset similarity threshold with other high-quality face images are obtained to obtain failed high-quality face images; and the face images corresponding to the movement trajectories of the failed high-quality face images are obtained to obtain a set of failed face images.

[0094] In some implementations, the set of failed face images includes a manually cleaned set and an automatically cleaned set. Obtaining the face image matching the movement trajectory of a failed high-quality face image to form the set of failed face images includes: if the image similarity corresponding to the failed high-quality face image is within a fuzzy similarity threshold range, then adding the face image matching the movement trajectory of the failed high-quality face image to the manually cleaned set; if the image similarity corresponding to the failed high-quality face image is within a low similarity threshold range, then detecting whether the failed high-quality face image is similar to any face in the base database; if not similar, adding the face image matching the movement trajectory of the failed high-quality face image to the automatically cleaned set; if similar, adding the face image matching the movement trajectory of the failed high-quality face image to the manually cleaned set; wherein the maximum value of the fuzzy similarity threshold range is less than a preset similarity threshold, and the minimum value of the fuzzy similarity threshold range is greater than the maximum value of the low similarity threshold range; before removing abnormal face images from the set of failed face images, the method further includes: sending the manually cleaned set to a manual processing end so that the manual processing end can determine and merge face images belonging to the same person.

[0095] Different processing strategies are adopted based on the image similarity between the failed high-quality face images to improve the accuracy of face image cleaning.

[0096] For example, the clustering uses a preset similarity threshold of A. Based on the image similarity calculated in the above embodiment, the motion trajectories corresponding to failed high-quality face images whose image similarity falls within the fuzzy similarity threshold range are statistically analyzed. The face images in these motion trajectories are added to the manual cleaning set for manual screening, while the corresponding motion trajectories in the face trajectory information are deleted. Here, A is the maximum value of the fuzzy similarity threshold range, and B is the minimum value of the fuzzy similarity threshold range, where B is less than A.

[0097] Then, the image similarity of the remaining movement trajectories is all less than B. The remaining movement trajectories can be directly regarded as movement trajectories in the low similarity threshold range. The failed high-quality face images corresponding to these movement trajectories are compared with the base database faces to determine whether there are similar base database faces. If they are not similar to any base database face, the face image under the movement trajectory is stored in the automatic cleaning set for further automatic cleaning and merging. If they are similar to any base database face, the face image under the movement trajectory is stored in the manual cleaning set for further manual cleaning and merging.

[0098] Specifically, for the data in the manually cleaned set, the manually cleaned set is sent to the manual processing end, so that the manual processing end can determine whether there are facial images of the same person in the manually cleaned set. If so, they are merged.

[0099] Then, a second cleaning process is performed on the manually cleaned set and the automatically cleaned set to remove abnormal face images from both sets.

[0100] Abnormal facial images include, but are not limited to, facial images with special effects, animations, or other abnormalities. Abnormal facial images can be defined according to the actual application scenario, and this application does not limit them.

[0101] For example, face images from both the manually cleaned set and the automatically cleaned set are input into a pre-trained abnormal image recognition model to obtain abnormal face images output by the model; abnormal face images are then removed from both sets to obtain cleaned face images.

[0102] The abnormal image recognition model can be implemented based on a large model and / or a convolutional neural network (CNN) model, etc., and this application does not limit it in this regard.

[0103] Multi-level cleaning can generate high-quality facial image datasets, improving image cleaning results.

[0104] For example, please see Figure 4 , Figure 4 This is a schematic diagram illustrating face image cleaning in an exemplary embodiment of this application, as shown below. Figure 4 As shown, a user account is associated with multiple videos. Each video undergoes face and human figure detection to generate a motion trajectory. Face and human figure matching yields a set of matching faces for each motion trajectory. The matching face sets are then filtered, and the high-quality face images from each set are clustered to achieve video-level clustering. Combining the video-level clustering results for each video, account-level clustering is performed. Based on the account-level clustering results, high-quality face images that failed to cluster are identified to obtain manually cleaned and automatically cleaned sets. Based on the cleaning results of the account-level clustering, the manually cleaned set, and the automatically cleaned set, cleaned face images are obtained. The information of the cleaned face images is stored to obtain a face information database.

[0105] In some implementations, after obtaining the cleaned face image, the cleaned face image is stored, specifically including: assigning a unified and unique face identifier to face images whose movement trajectories match and belong to the same cluster; extracting image features from the cleaned face image to obtain a face feature vector; and obtaining business metadata associated with the cleaned face image; associating and storing the face feature vector and the unique face identifier in a vector database; and associating and storing the business metadata and the unique face identifier in a relational database.

[0106] By adopting a dual-database collaborative architecture of vector database and relational database, we can meet the requirements of high efficiency in facial feature similarity matching in actual business, and also meet the requirements of standardized management of structured information.

[0107] The vector database, acting as the face feature retrieval engine, is responsible for storing face feature vectors and providing similarity matching. The relational database, serving as the central storage hub for structured information, is responsible for managing business metadata associated with faces (such as identifiers, quality scores, storage paths, account ownership, etc.). The vector and relational databases are strongly linked through a unique face identifier (FaceID). Specifically, the vector database uses FaceID as an index to bind face feature vectors, while the relational database uses FaceID as the primary key to store structured business metadata. Through the collaboration of the vector and relational databases, a complete process of face feature vector retrieval, metadata association, and query result output can be achieved, while simultaneously supporting CRUD operations on FaceID.

[0108] Figure 5 This is a block diagram illustrating an image data cleaning apparatus according to an exemplary embodiment of this application. Figure 5 As shown, the exemplary image data cleaning apparatus 500 includes: The image detection module 510 is used to perform human figure detection and face detection on each image separately, and obtain multiple human figure images and face images; The matching module 520 is used to generate multiple movement trajectories based on the location information corresponding to each human figure image, determine the face image that matches the movement trajectory, and obtain the set of matching faces corresponding to the movement trajectory; wherein, the higher the probability that the face image and the human figure image corresponding to the movement trajectory belong to the same person, the higher the matching degree between the face image and the movement trajectory. Module 530 is selected to extract face images whose quality scores meet the preset scoring conditions from each matched face set, so as to obtain high-quality face images corresponding to each movement trajectory. Clustering module 540 is used to cluster the high-quality face images corresponding to each movement trajectory, and obtain the movement trajectories belonging to the same cluster based on the clustering results; The merging module 550 is used to merge face images that match the movement trajectories of images belonging to the same cluster to obtain cleaned face images.

[0109] It should be noted that the image data cleaning apparatus and the image data cleaning method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs its operations have been described in detail in the method embodiments and will not be repeated here. In practical applications, the image data cleaning apparatus provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0110] Please see Figure 6 , Figure 6 This is a schematic diagram illustrating the structure of an electronic device in an exemplary embodiment of this application. The electronic device 600 includes a memory 610 and a processor 620. The processor 620 executes program instructions stored in the memory 610 to implement the steps in any of the above-described image data cleaning method embodiments. In a specific implementation scenario, the electronic device 600 may include, but is not limited to, a microcomputer or a server. Furthermore, the electronic device 600 may also include mobile devices such as laptops and tablets, without limitation.

[0111] Specifically, processor 620 controls itself and memory 610 to implement the steps in any of the above-described image data cleaning method embodiments. Processor 620 may also be referred to as a Central Processing Unit (CPU). Processor 620 may be an integrated circuit chip with signal processing capabilities. Processor 620 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 620 may be implemented using integrated circuit chips.

[0112] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating the structure of a computer-readable storage medium in an exemplary embodiment of this application. The computer-readable storage medium 700 stores program instructions 710 that can be executed by a processor. The program instructions 710 are used to implement the steps in any of the above-described image data cleaning method embodiments.

[0113] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0114] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0115] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0116] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An image data cleaning method characterized by comprising: The method includes: Perform human and face detection on each image separately to obtain multiple human and face images; Multiple movement trajectories are generated based on the location information corresponding to each human figure image, and a face image matching the movement trajectory is determined to obtain a set of matching faces corresponding to the movement trajectory; wherein, the higher the probability that the face image and the human figure image corresponding to the movement trajectory belong to the same person, the higher the matching degree between the face image and the movement trajectory; Extract the face images that meet the preset scoring conditions from each matched face set to obtain the high-quality face image corresponding to each movement trajectory; Cluster the high-quality face images corresponding to each movement trajectory, and obtain the movement trajectories belonging to the same cluster based on the clustering results; Merge face images whose movement trajectories match and belong to the same cluster to obtain cleaned face images.

2. The method of claim 1, wherein, The image is a video frame from a video taken within a preset time period; the step of clustering the high-quality face images corresponding to each movement trajectory, and obtaining the movement trajectories belonging to the same cluster based on the clustering results, includes: For each high-quality face image corresponding to the movement trajectory extracted from the video, the image similarity between each high-quality face image is calculated to obtain the first cluster similarity. High-quality face images with a first cluster similarity higher than a preset similarity threshold are grouped into the same cluster to obtain video-level clustering results; Based on the video-level clustering results, the movement trajectories belonging to the same cluster are determined.

3. The method of claim 2, wherein, The video is associated with a corresponding user account; determining the movement trajectory belonging to the same cluster based on the video-level clustering results includes: Based on the video-level clustering results, the motion trajectories corresponding to high-quality face images in the same cluster are merged to obtain the video-level motion trajectory. For all videos associated with the same user account, video-level motion trajectories are used to reselect face images that meet the preset scoring conditions to obtain reselected high-quality face images. Calculate the image similarity between the reselected high-quality face images to obtain the second cluster similarity; High-quality face images with a second cluster similarity higher than a preset similarity threshold are grouped into the same cluster to obtain account-level clustering results; The movement trajectory corresponding to the high-quality face image belonging to the same cluster in the video-level clustering results is taken as the movement trajectory belonging to the same cluster.

4. The method of claim 1, wherein, The method further includes: A set of failed face images is obtained by matching the movement trajectory of high-quality face images that have failed clustering. Abnormal face images are removed from the set of failed face images to obtain cleaned face images.

5. The method of claim 4, wherein, High-quality face images with image similarity higher than a preset similarity threshold are grouped into the same cluster; The movement trajectory of the high-quality face image based on clustering failure corresponds to the matched face image, resulting in a set of failed face images, including: High-quality face images whose image similarity to other high-quality face images is lower than the preset similarity threshold are obtained, resulting in failed high-quality face images. Obtain the matching face image corresponding to the movement trajectory of the failed high-quality face image to obtain the failed face image set.

6. The method of claim 5, wherein, The set of failed face images includes a manually cleaned set and an automatically cleaned set; the process of obtaining the face image corresponding to the movement trajectory of the failed high-quality face image to obtain the set of failed face images includes: If the image similarity corresponding to the failed high-quality face image is within the fuzzy similarity threshold range, then the face image matching the movement trajectory of the failed high-quality face image is added to the manual cleaning set. If the image similarity corresponding to the failed high-quality face image is in the low similarity threshold range, then it is detected whether the failed high-quality face image is similar to any face in the base database. If they are not similar, then the face image matching the movement trajectory of the failed high-quality face image is added to the automatic cleaning set. If they are similar, then the face image matching the movement trajectory of the failed high-quality face image is added to the manual cleaning set. Wherein, the maximum value of the fuzzy similarity threshold range is less than the preset similarity threshold, and the minimum value of the fuzzy similarity threshold range is greater than the maximum value of the low similarity threshold range. Before removing abnormal face images from the set of failed face images, the method further includes: The manually cleaned set is sent to the human processing terminal so that the human processing terminal can identify and merge facial images belonging to the same person.

7. The method of claim 1, wherein, Before extracting face images whose quality scores meet preset scoring conditions from each matched face set to obtain high-quality face images corresponding to each movement trajectory, the method further includes: Filter face images in the matched face set whose quality score is lower than a preset score threshold; And / or, filter out face images in the matching face set whose face angles do not meet the preset angle conditions; And / or, filter the set of matching faces whose number of face images is less than a preset threshold.

8. The method of claim 1, wherein, The method further includes: Assign a unified and unique face identifier to the face images whose movement trajectories match and belong to the same cluster, extract face feature vectors from the cleaned face images, and obtain the business metadata associated with the cleaned face images. The facial feature vector and the unique facial identifier are associated and stored in a vector database, and the business metadata and the unique facial identifier are associated and stored in a relational database.

9. An electronic device, comprising: The electronic device includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the steps of the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the steps of the method as described in any one of claims 1-8.

Images