Face image training set updating method and device and electronic equipment

Abstract

The invention is suitable for the technical field of data processing, and provides a face image training set updating method and device and electronic equipment, and the method comprises the steps: obtaining a plurality of face images; for any face image, determining a label of the face image; for any face image, determining the quality level of the face image; generating corresponding sample data according to the label of each face image and the quality grade; generating a training set according to the sample data; training a to-be-trained face model according to the training set to obtain a trained face model; under the condition that an evaluation result of the trained face model is obtained, whether the training set is updated or not is selected according to the evaluation result, and the evaluation result is obtained based on application feedback of a client to the trained face model; the application feedback comprises information used for indicating whether the trained face model needs to be retrained or not. Through the method, the accuracy of the updated training set can be improved.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to methods, apparatuses, electronic devices, computer-readable storage media, and computer program products for updating face image training sets. Background Technology

[0002] In existing technologies, open datasets are typically used as training sets to train face models. However, since open datasets usually contain faces from specific scenarios, such as security checks, which are relatively simple, face models trained using open datasets are generally only applicable to simple scenarios and are difficult to apply to complex scenarios.

[0003] Therefore, a new method is needed to determine a training set suitable for complex scenarios. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for updating a face image training set, which can solve the problem that existing methods are difficult to generate training sets suitable for complex scenarios.

[0005] In a first aspect, embodiments of this application provide a method for updating a face image training set, including:

[0006] Acquire multiple face images;

[0007] For any one of the aforementioned face images, determine the label of the face image;

[0008] For any one of the aforementioned face images, determine the quality level of the face image;

[0009] Generate corresponding sample data based on the labels and quality levels of each face image;

[0010] A training set is generated based on the sample data described above;

[0011] The face model to be trained is trained based on the training set to obtain the trained face model;

[0012] Upon obtaining the evaluation results of the trained face model, a decision is made on whether to update the training set based on the evaluation results. The evaluation results are obtained based on the client's application feedback on the trained face model, and the application feedback includes information indicating whether the trained face model needs to be retrained.

[0013] In this embodiment, after acquiring multiple face images, the server determines the label and quality level of each face image, generates corresponding sample data based on the generated labels and quality levels, generates a training set based on the generated sample data, and trains the face model to be trained using the training set to obtain the trained face model. If the server receives the evaluation result of the trained face model sent by the client, it selects whether to update the training set based on the information included in the evaluation result indicating whether the trained face model needs to be retrained. Since updating the training set only occurs when the evaluation result indicates that the face model needs to be retrained, unnecessary updates are reduced when updating the training set in the above manner. Furthermore, since the evaluation result is determined based on the feedback obtained after the client runs the face model, the evaluation result reflects the actual running effect of the client, thereby improving the accuracy of the updated training set when updating the training set based on the evaluation result. Furthermore, since a higher quality level indicates a higher quality of sample data, meaning a higher probability that the sample data can be identified, when the sample data also includes quality level information, generating a training set based on this sample data and using the generated training set to train the face model to be trained is beneficial for selecting sample data with a more suitable quality level for the face model to be trained.

[0014] Optionally, the step of selecting whether to update the training set based on the evaluation result includes:

[0015] If the evaluation result indicates that the trained face model needs to be retrained, the update direction is determined based on the evaluation result;

[0016] If the update direction includes increasing the proportion of sample data with higher quality levels in the training set, then based on the quality level distribution of the sample data in the training set, the proportion of sample data with higher quality levels in the training set is increased, and / or the proportion of sample data with lower quality levels in the training set is decreased, to obtain the updated training set, wherein the proportion of sample data with higher quality levels in the updated training set is higher than the proportion of sample data with higher quality levels in the training set before the update.

[0017] The sample data with higher quality level refers to sample data with a quality level higher than a preset quality level threshold, and the sample data with lower quality level refers to sample data with a quality level not higher than the quality level threshold.

[0018] Optionally, the step of selecting whether to update the training set based on the evaluation result includes:

[0019] If the evaluation result indicates that the trained face model needs to be retrained, the update direction is determined based on the evaluation result;

[0020] If the update direction includes increasing the proportion of sample data with a specified label in the training set, then based on the label of the sample data in the training set, the proportion of sample data corresponding to the specified label is increased, wherein the specified label is used to indicate the category to which the sample data belongs.

[0021] Optionally, acquiring multiple face images includes:

[0022] Obtain multiple facial images from the video;

[0023] Before determining the label of the face image, the method further includes:

[0024] Obtain multi-dimensional information related to the face image from the video, the multi-dimensional information including at least one of the following: audio information, subtitle information, and character information;

[0025] The process of determining the label of the face image includes:

[0026] The label of the face image is determined based on the face image and the multi-dimensional information associated with the face image.

[0027] Optionally, obtaining multiple face images from the video includes:

[0028] The video is subjected to frame extraction to obtain multiple image frames;

[0029] Clustering is performed on the multiple image frames to obtain the clustering results;

[0030] If the clustering result includes a face cluster, the face cluster is extracted to obtain multiple face images.

[0031] Optionally, after extracting the face cluster to obtain multiple face images, the method further includes:

[0032] If the number of face images contained in the same face cluster is less than a preset threshold, the video is re-processed by frame extraction until the number of face images contained in the same face cluster is not less than the threshold.

[0033] Optionally, determining the label of the face image based on the face image and multi-dimensional information related to the face image includes:

[0034] A minimum spanning tree is generated based on the face images contained in the face cluster, and the nodes of the minimum spanning tree correspond one-to-one with the face images;

[0035] The label of the face image is determined based on the face image corresponding to the minimum spanning tree and the multi-dimensional information related to the face image.

[0036] Optionally, generating a minimum spanning tree based on the face images contained in the face cluster includes:

[0037] Calculate the face heat corresponding to the face image in the face cluster, wherein the face heat is related to the number of times the face image appears;

[0038] The face images are filtered based on the face heat, and the minimum spanning tree is generated based on the filtered face images.

[0039] Optionally, generating a minimum spanning tree based on the face images contained in the face cluster includes:

[0040] Calculate the weight density corresponding to the face images in the face cluster, wherein the weight density is used to reflect whether the face image is an outlier;

[0041] The face images are filtered according to the weight density, and the minimum spanning tree is generated based on the filtered face images.

[0042] Secondly, embodiments of this application provide a face image training set update apparatus, comprising:

[0043] The face image acquisition module is used to acquire multiple face images;

[0044] The label determination module is used to determine the label of any one of the face images;

[0045] The quality level module is used to determine the quality level of any one of the face images.

[0046] The sample data generation module is used to generate corresponding sample data based on the labels and quality levels of each of the face images;

[0047] The training set generation module is used to generate a training set based on each of the sample data;

[0048] The face model training module is used to train the face model to be trained based on the training set to obtain the trained face model.

[0049] The evaluation result acquisition module is used to select whether to update the training set based on the evaluation result of the trained face model when the evaluation result is obtained. The evaluation result is obtained based on the application feedback of the client to the trained face model. The application feedback includes information indicating whether the trained face model needs to be retrained.

[0050] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any of the first aspects.

[0051] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the method described in any of the first aspects.

[0052] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to perform the method described in any of the first aspects.

[0053] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0055] Figure 1 This is a flowchart illustrating a method for updating a face image training set according to an embodiment of this application;

[0056] Figure 2 This is a schematic diagram of facial images of the same person of different qualities provided in an embodiment of this application;

[0057] Figure 3 This is a schematic diagram of an undirected graph provided in an embodiment of this application;

[0058] Figure 4 This is a schematic diagram of different face images provided in one embodiment of this application;

[0059] Figure 5 This is a flowchart illustrating another method for updating a face image training set according to another embodiment of this application;

[0060] Figure 6This is a schematic diagram of the system framework involved in a face image training set update method provided in another embodiment of this application;

[0061] Figure 7 This is a schematic diagram of the structure of a face image training set update device provided in an embodiment of this application;

[0062] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0063] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0064] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0065] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0066] Furthermore, in the description of this application and the appended claims, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0067] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0068] As the barriers to internet content production have decreased, the upload and publication volume of videos and news (such as text and images) has grown exponentially.

[0069] Information feed products (such as videos and news) can provide users with appropriate content in suitable scenarios. For example, they can distribute categorized videos, videos with added tags, and videos with supplementary explanatory text. The content distributed to users is usually obtained by machines processing uploaded content through algorithms. Since information feed distribution mainly involves video content, and much of this video content involves specific people, it is usually necessary to perform facial recognition on the video content (such as recognizing facial features, facial expressions, and facial attributes) to improve the accuracy of video distribution, and then distribute video content to users based on the recognition results.

[0070] Because the content sources of news feed products include a wide variety of content creation organizations, such as self-media and professionally generated content (PGC) and user-generated content (UGC), the scenarios corresponding to news feed products are complex due to the diverse content sources. If a training set is obtained from an open dataset, and then a face model trained on that dataset is used for face recognition in news feed products, the faces in the open dataset are usually from specific scenarios (such as security check scenarios). However, security check scenarios differ from the scenarios corresponding to news feeds (face images in security check scenarios are usually provided by the public security department, while faces in news feeds often involve various self-produced dramas, self-produced variety shows, and signed artists, meaning the range of faces is broader and the scenarios are more complex). Therefore, the face model trained using an open dataset is difficult to accurately recognize faces in the video content of news feeds, making it difficult to apply directly to news feeds.

[0071] To obtain a more suitable training set, embodiments of this application provide a method for updating a face image training set. In this method, after training a face model, an evaluation result of at least one face model is obtained, and the training set is updated when the evaluation result indicates that the face model needs to be retrained.

[0072] The face image training set update method provided in the embodiments of this application is described below with reference to the accompanying drawings.

[0073] Figure 1 This document illustrates a flowchart of a face image training set update method provided in an embodiment of this application. This method can be applied to an electronic device, which can be a server. The following description uses a server as an example to illustrate the face image training set update method provided in this application, detailed below:

[0074] S11, acquire multiple face images.

[0075] Here, "face image" refers to an image containing a human face, which can be obtained from various data sources such as videos (e.g., short videos), image datasets, and open-source face datasets.

[0076] S12, for any one of the above-mentioned face images, determine the label of the face image.

[0077] The labels here are determined according to the requirements. For example, if we need to train a face model that can recognize gender, then the labels here would include gender. For example, if we need to train a face model that can recognize age, then the labels here would include age. Of course, the above labels can also include classification information, such as classification information to indicate whether the face belongs to a celebrity or a non-celebrity, which will not be elaborated on here.

[0078] S13, for any one of the above-mentioned face images, determine the quality level of the face image.

[0079] In this context, a higher quality level indicates higher quality sample data and a higher probability of being recognized; conversely, a lower quality level indicates lower quality sample data. In this embodiment, the quality of a face image does not refer to whether the image is clear or blurry, but rather to the recognizability of the face image. Higher recognizability corresponds to higher face quality, i.e., a higher quality level. Figure 2 This illustration shows a diagram of facial images of the same person of varying quality. Figure 2 In the image, from left to right, the quality of the facial images is sorted from low to high. When the quality of a facial image is too low, it will be difficult to determine the identity information, which will lead to recognition errors. Therefore, it is necessary to clean facial images with low quality.

[0080] Specifically, the image can be processed by at least one of image enhancement, background removal, and illumination normalization. Then, face detection is performed on the processed image using a face key point detection method. If a face is detected in the processed image, a face quality score is calculated, which corresponds to the quality level of the face image.

[0081] Image enhancement methods include at least one of the following: contrast enhancement, sharpening, and noise suppression. These methods are used to improve the visual effect of an image in order to extract useful information from the image.

[0082] Among them, the background removal method is used to remove all or part of the background from an image, while preserving the foreground object.

[0083] Illumination normalization is used to reduce the impact of illumination variations on image feature extraction. There are many methods for illumination normalization, such as histogram equalization, which equalizes the histogram of the image to make the brightness distribution of the image more uniform, thereby reducing the influence of illumination.

[0084] Among them, face quality scoring can be achieved through a face quality assessment model. The image of the detected face is used as the input of the face quality assessment model to obtain the face quality score or quality level output by the face quality assessment model.

[0085] In this embodiment of the application, the face quality assessment model can be trained in a supervised manner or in an unsupervised manner.

[0086] Alternatively, considering the high labor and time costs of manual annotation, a face quality assessment model can be trained in an unsupervised manner.

[0087] Optionally, the face quality assessment model trained in an unsupervised manner includes at least two recognition modules, each employing a different recognition algorithm to improve the accuracy of the model's output. For example, suppose the face quality assessment model includes three recognition modules (let's say a first recognition module, a second recognition module, and a third recognition module). The first recognition module is trained using a model perturbation training method, the second recognition module is trained using a face similarity distribution distance training method, and the third recognition module is trained using an adaptive feature length training method.

[0088] Among them, the training method of model perturbation refers to randomly discarding a portion of neurons during the training process, forcing the model to learn more robust feature representations. After discarding neurons, the variance of a set of features is calculated, and the magnitude of the variance represents the judgment of the first recognition module on the quality of the face image.

[0089] The purpose of training a face similarity distribution distance model is to train a model to evaluate the similarity between two face images. Specifically, it involves learning a metric space in which similar face images are closer together, while dissimilar images are farther apart.

[0090] The core idea of ​​the adaptive feature length training method is to introduce an adaptive mechanism during the training of the recognition model to dynamically adjust the feature length (i.e., the length of the feature vector) to enhance the model's response to important features while suppressing unimportant features. Since this training method maps quality factors to the feature length, the quality is positively correlated with the length of the feature length.

[0091] When the face quality assessment model includes a first recognition module, a second recognition module, and a third recognition module, and the output results of the first recognition module, the second recognition module, and the third recognition module are respectively the first recognition result, the second recognition result, and the third recognition result are normalized and assigned corresponding weights. The resulting comprehensive result is then used as the output result of the face quality assessment model.

[0092] S14, Generate corresponding sample data based on the labels and quality levels of each of the aforementioned face images.

[0093] Specifically, face images, including quality ratings and labels, are used as sample data in the training set.

[0094] S15, Generate a training set based on the above sample data.

[0095] Specifically, the set corresponding to each sample data can be used as the training set mentioned above, or the sample data can be filtered (e.g., high-quality sample data can be selected) and the set corresponding to the selected sample data can be used as the training set mentioned above.

[0096] S16, Train the face model to be trained based on the above training set to obtain the trained face model.

[0097] The aforementioned face models include, but are not limited to: face detection models and face attribute recognition models. Face attributes include gender and / or age, etc.

[0098] S17, after obtaining the evaluation results of the trained face model, select whether to update the training set based on the evaluation results. The evaluation results are obtained based on the client's application feedback on the trained face model. The application feedback includes information indicating whether the trained face model needs to be retrained.

[0099] Specifically, after obtaining the trained face model, if the trained face model is applied to the client, the client can send the evaluation results to the server.

[0100] The evaluation results are determined based on feedback information from the applied face model, including information indicating whether the face model needs retraining. Optionally, the need for retraining can be indicated by user satisfaction with the face model (e.g., a low satisfaction level indicates retraining). Alternatively, the need for retraining can be indicated by user evaluation scores of the face model's output (e.g., a low score indicates retraining). Furthermore, the need for retraining can be based on recognition accuracy; for example, if the recognition accuracy is below a preset accuracy threshold, retraining is required.

[0101] In this embodiment, the server trains a face model based on sample data from the training set. The trained face model is then applied to the client. After applying the face model, the client can directly send the corresponding evaluation results to the server, or selectively send the corresponding evaluation results to the server. For example, the client only chooses to send the corresponding evaluation results to the server if the application effect of the face model is unsatisfactory.

[0102] Optionally, when the client sends the evaluation results to the server, it can choose to send them in real time (e.g., after applying the face model) or at intervals (e.g., after the interval time has elapsed), without any limitation here.

[0103] It should be noted that when a client applies multiple face models, the client may choose to send the evaluation results corresponding to some face models to the server. In this case, the server will receive the evaluation results of at least one face model.

[0104] After receiving the evaluation results, the server can analyze them and determine how to update the corresponding training set based on the analysis. For example, if the evaluation results indicate that the face model's recognition accuracy is low, updating the training set used to train the face model will increase the proportion of sample data that is beneficial to improving recognition accuracy. Because the proportion of sample data that is beneficial to improving recognition accuracy is increased, retraining the face model based on the updated training set will improve the recognition accuracy of the retrained face model.

[0105] Updating the training set here includes replacing some or all of the original sample data in the training set with new sample data, or it may include adding the new sample data to the training set to expand the training set.

[0106] In this embodiment, after acquiring multiple face images, the server determines the label and quality level of each face image, generates corresponding sample data based on the generated labels and quality levels, generates a training set based on the generated sample data, and trains the face model to be trained using the training set to obtain the trained face model. If the server receives the evaluation result of the trained face model sent by the client, it selects whether to update the training set based on the information included in the evaluation result indicating whether the trained face model needs to be retrained. Since updating the training set only occurs when the evaluation result indicates that the face model needs to be retrained, unnecessary updates are reduced when updating the training set in the above manner. Furthermore, since the evaluation result is determined based on the feedback obtained after the client runs the face model, the evaluation result reflects the actual running effect of the client, thereby improving the accuracy of the updated training set when updating the training set based on the evaluation result. Furthermore, since a higher quality level indicates a higher quality of sample data, meaning a higher probability that the sample data can be identified, when the sample data also includes quality level information, generating a training set based on this sample data and using the generated training set to train the face model to be trained is beneficial for selecting sample data with a more suitable quality level for the face model to be trained.

[0107] In some embodiments, S17 includes:

[0108] If the above evaluation results indicate that the trained face model needs to be retrained, the update direction shall be determined based on the evaluation results.

[0109] If the above update direction includes increasing the proportion of high-quality sample data in the training set, then based on the quality level distribution of the sample data in the training set, the proportion of high-quality sample data in the training set is increased, and / or the proportion of low-quality sample data in the training set is decreased, to obtain the updated training set, wherein the proportion of high-quality sample data in the updated training set is higher than the proportion of high-quality sample data in the training set before the update.

[0110] Among them, the sample data with higher quality level refers to sample data with a quality level higher than the preset quality level threshold, and the sample data with lower quality level refers to sample data with a quality level not higher than the aforementioned quality level threshold.

[0111] To increase the proportion of high-quality sample data, after evaluating the quality level of a face image, a quality level comparison step can be added to filter out face images with lower quality levels. Specifically, the quality level can be compared with a preset quality level threshold. If it is not higher than the quality level threshold, the face image can be discarded, and no sample data will be generated based on the face image.

[0112] In this embodiment, the update direction is used to indicate the adjustment direction of the sample data in the training set. Optionally, when the evaluation result contains textual information, semantic parsing is performed on the evaluation result, and the update object and update intensity are determined based on the semantic parsing result. The update direction is then updated based on the update object and update intensity. Optionally, when the evaluation result contains numerical information, considering that numerical information does not have a clearly defined update object, the update direction can be determined based on the evaluation result and the functional description of the face model. For example, the update object can be determined based on the functional description of the face model, and the update intensity can be determined based on the numerical information in the evaluation result.

[0113] In this embodiment of the application, since the evaluation results are obtained based on the client's application feedback on the face model, determining the update direction based on the evaluation results can improve the matching degree between the update direction and the application feedback. Therefore, when updating the training set according to the update direction, it is beneficial to improve the matching degree between the obtained training set and the application feedback, that is, to improve the accuracy of the training set.

[0114] In some embodiments, considering that the faces involved in the information stream may be celebrities or non-celebrities (i.e., ordinary people or internet celebrities), but because celebrities have higher exposure rates, the probability of including celebrities in the training set sample data is higher. Conversely, because non-celebrities have lower exposure rates, the probability of including non-celebrities in the training set sample data is lower. Therefore, the face model trained based on the training set may have difficulty effectively recognizing non-celebrities, and in this case, the training set needs to be updated. Conversely, when the face model trained based on the training set may have difficulty effectively recognizing celebrities, the training set also needs to be updated. That is, in this embodiment, the classification of faces in the sample data can be marked by labels (such as whether they belong to the celebrity category or the ordinary person category, etc.), and the training set can be updated subsequently according to the labels in the sample data. In this case, S17 above includes:

[0115] If the above evaluation results indicate that the trained face model needs to be retrained, the update direction shall be determined based on the above evaluation results;

[0116] If the above update direction includes increasing the proportion of sample data with specified labels in the training set, then based on the labels of the sample data in the training set, the proportion of sample data corresponding to the specified labels is increased, wherein the specified labels are used to indicate the category to which the sample data belongs.

[0117] Optionally, the proportion of sample data corresponding to a specified label can be adjusted by changing the data source. For example, when it is necessary to increase the proportion of sample data corresponding to a specified label, a data source more likely to contain that specified label can be selected, such as selecting face images from short videos. Since selecting a data source more likely to contain the specified label to obtain face images is beneficial to increasing the proportion of sample data corresponding to the specified label in the training set.

[0118] Optionally, the proportion of sample data corresponding to a specified label can be adjusted by filtering out labels that indicate facial images as specified labels and constructing sample data based on the filtering results. For example, when it is necessary to increase the proportion of sample data corresponding to a specified label, sample data can be constructed based on the facial images corresponding to the filtered labels that indicate facial images as specified labels, thereby increasing the proportion of sample data corresponding to the specified label.

[0119] In this embodiment of the application, since the proportion of sample data corresponding to a specified label in the training set can be adjusted, a face model that can recognize the face corresponding to the specified label can be trained based on the updated training set.

[0120] Optionally, after obtaining the facial model, corresponding recognition processing can be performed based on the facial model. For example, if the facial model is capable of recognizing a celebrity, facial recognition can be performed on the information stream based on the facial model to find the video stream (or information) corresponding to the celebrity. These video streams (or information) may be the video stream (or information) corresponding to the celebrity in the current time period, or they may be the video stream (or information) corresponding to the celebrity when they were an ordinary person.

[0121] Furthermore, the facial recognition results can be applied in various ways, such as: (a) casting: video character recognition can directly select clips of programs previously participated in by celebrities, avoiding manual screening and thus assisting producers in casting; (b) advertising placement: when a spokesperson appears in a video of a drama, video character recognition can be performed first to identify and save all the locations where the spokesperson appears in the video, then advertising can be matched, and finally the matched advertising clips can be placed in the clips where the spokesperson appears in the video, thus achieving a natural connection; (c) viewing only a specific person (this person can be a real person or an intellectual property (IP) character): selecting clips of a specific person from the video so that the selected clips can be presented to the user later.

[0122] Additionally, considering that continuously recommending videos featuring the same face to a user might negatively impact their viewing experience, videos featuring different faces can be interspersed when the user is scrolling through videos continuously. That is, after the face model identifies videos corresponding to different faces, the videos featuring the same face are broken up and then distributed to the user.

[0123] It should be noted that, due to the numerous application scenarios involving people, and the fact that different application scenarios typically involve different facial models, it is necessary to construct different facial models in order to effectively identify people in the large amount of content that includes people in the information stream. At the same time, as the content, source, and popular trends in the information stream change, the facial models also need to be continuously supplemented, updated, and iterated to improve the accuracy of the facial models used in the application.

[0124] In the above embodiment, the label corresponding to the face image is determined based on the face image itself, that is, based on the image information. However, considering that there are other dimensions of information besides image information, such as video information, the label of the face image can also be determined by combining information from other dimensions.

[0125] In some embodiments, S11 includes: acquiring a plurality of the aforementioned face images from a video.

[0126] Correspondingly, before determining the label of the face image in S12 above, the process further includes:

[0127] Obtain multi-dimensional information related to the aforementioned facial images from the video. This multi-dimensional information includes at least one of the following: audio information, subtitle information, and character information.

[0128] Correspondingly, the labels used to determine the aforementioned facial images include:

[0129] Based on the aforementioned face image and the multi-dimensional information related to the aforementioned face image, the label of the aforementioned face image is determined.

[0130] The aforementioned audio information refers to the audio information of the person corresponding to the facial image. For example, when person A in a video makes a sound, the audio information corresponding to that sound is the audio information corresponding to person A.

[0131] The aforementioned subtitle information can be the text information corresponding to the aforementioned audio information, or it can be the introduction information of the video in which the face image is located (such as the title of the TV series when the video is a video clip of a TV series), etc., and there is no limitation here.

[0132] The aforementioned character information can be the character information corresponding to a real person, or the character information corresponding to an IP character (i.e., a virtual character). This character information is the name of the character corresponding to the face image in the face image. For example, assuming that the character B plays the role of "Potato" in the face image, then the character information corresponding to character B is "Potato".

[0133] In this embodiment of the application, when the face image is obtained from a video, at least one of the audio information, subtitle information, and character information corresponding to the face image is obtained from the video. For example, the moment when the face image appears in the video is determined, and the audio information and subtitle information corresponding to that moment are taken as the audio information corresponding to the face image, and the character in the video corresponding to the face image is taken as the character information corresponding to the face image.

[0134] Optionally, if the face image is obtained from a non-video source, such as an image, it can be first detected whether the image contains multi-dimensional information related to the face image. For example, it can be detected whether the image contains text information through Optical Character Recognition (OCR). If it does, the text information is extracted from the image as multi-dimensional information related to the face image.

[0135] In addition to the multi-dimensional information listed above, this multi-dimensional information can also include video background information, which will not be elaborated here. This video background information may include scene setting information, video shooting and production techniques, and shooting location, among other things.

[0136] In this embodiment, when determining the label of a face image, the face image and multi-dimensional information can be labeled simultaneously or separately. When the face image and multi-dimensional information are labeled separately, the label of the face image corresponding to the same face and the label of the multi-dimensional information corresponding to the same face may be different. In this case, these labels need to be uniformly processed so that the labels corresponding to the face image and the multi-dimensional information corresponding to the same face are the same. For example, the same IP person may have multiple names or aliases. After unifying them into a standardized result, the label corresponding to the face image is determined based on the standardized result.

[0137] In this embodiment of the application, when determining the label corresponding to a face image, not only the information of the face image is considered, but also the multi-dimensional information related to the face image is considered. The multi-dimensional information related to the face image can reflect information other than the image information of the face image itself. Therefore, determining the label corresponding to the face image based on the face image and the multi-dimensional information related to the face image is beneficial to improving the accuracy of the obtained label.

[0138] In some embodiments, obtaining multiple of the aforementioned face images from a video includes:

[0139] B1. Perform frame extraction on the above video to obtain multiple image frames.

[0140] Specifically, the video can be processed by extracting frames at fixed or random intervals to obtain multiple extracted image frames. These image frames do not necessarily contain human faces.

[0141] Optionally, after extracting the image frames, the extracted image frames can be subjected to low-quality filtering to improve the quality of the subsequently obtained sample data. This low-quality filtering includes one or more of the following: filtering image frames containing sensitive information, filtering image-text mismatches, and filtering image frames with low resolution.

[0142] Specifically, image frames containing sensitive information can be filtered in the following way: determine whether the text information and / or image content information in the image frame contains information that violates laws and security, or information that violates public order and good morals. If the text information and / or image content information in the image frame contains information that violates laws and security, or information that violates public order and good morals, then the image frame is determined to be an image frame containing sensitive information and needs to be filtered.

[0143] Specifically, image frames with mismatched text and image information or low resolution can be filtered in the following ways: Based on the image frame's metadata (such as category, tags, etc.), determine whether the text information and image content information in the image frame match; or, determine whether the image content of the video's cover image matches the image content information of an image frame extracted from the video. If they match, the image frame is determined to be a mismatched text and image information frame and needs to be filtered. Based on the image frame's metadata (such as resolution, sharpness), determine whether the image frame's resolution or sharpness meets the requirements. If it does not meet the requirements, the image frame is determined to be a low-resolution image frame and needs to be filtered.

[0144] B2. Perform clustering processing on the above multiple image frames to obtain the clustering results.

[0145] Clustering aims to group multiple image frames based on similarity, so that image frames within the same group have high similarity, while image frames between different groups have low similarity.

[0146] The clustering algorithm used in clustering processing can be the K-Means clustering method. The goal of the K-Means clustering method is to divide the data into K clusters, such that the samples within each cluster are as similar as possible, and the samples between clusters are as different as possible.

[0147] Optionally, considering that clustering is for obtaining face images, and that image frames extracted from videos may contain only partial faces, or faces easily affected by the background of the image frame, preprocessing of image frames containing faces can be performed before clustering to improve the accuracy of the clustering results. For example, anchor points can be added to image frames containing faces to improve the accuracy of face clustering. Here, anchor points refer to key facial feature points, such as the positions of facial features like eyes, nose, mouth, and ears.

[0148] Optionally, the preprocessing of image frames containing faces may also include similarity transformation processing on those image frames. Since similarity transformation processing includes one or more processes such as rotation angle, scaling ratio, and translation vector, it can reduce the impact of pose and / or angle on the clustering results.

[0149] B3. If the above clustering results include face clusters, extract the face clusters to obtain multiple face images.

[0150] The clustering results may contain multiple clusters. Two image frames within the same cluster contain similar features. The cluster in which an image frame containing a face belongs is the aforementioned face cluster.

[0151] In this embodiment of the application, since image frames are clustered, and clustering can group image frames with similar features into the same cluster, for example, grouping image frames corresponding to the same face into the same cluster, clustering image frames extracted from the video can improve the speed of obtaining face images.

[0152] In some embodiments, considering that different faces typically appear at different frequencies—for example, in the same video, the main character and supporting characters usually appear at different frequencies—the number of face images contained in different face clusters after clustering is likely to be different. If the number is too small, it will result in insufficient sample data corresponding to the few faces, which is detrimental to subsequent training of the face model. Therefore, to improve the accuracy of the face model obtained through subsequent training, after B3 above, the following is also included:

[0153] If the number of face images contained in the same face cluster is less than a preset threshold, the video is re-processed by frame extraction until the number of face images contained in the same face cluster is not less than the threshold.

[0154] Specifically, the number of image frames extracted can be increased by reducing the interval frame count, thereby increasing the probability of obtaining more of the desired additional face images. Alternatively, all (or most) of the image frames in the video that include the face image can be extracted to obtain more of the desired additional face images.

[0155] In this embodiment, when the number of face images contained in a face cluster is determined to be less than a preset threshold, the video is re-processed by frame extraction until the number of face images contained in the face cluster is not less than the threshold. Therefore, the number of face images subsequently extracted from each face cluster is not less than the threshold. That is, it ensures that a large number of face images can be extracted from each face cluster, and a larger number of face images is beneficial for obtaining more corresponding sample data, thereby improving the accuracy of the trained face model.

[0156] In some embodiments, considering the large number of face images contained in all face clusters, in order to reduce the amount of annotation while minimizing the impact of coverage, a minimum spanning tree can be generated based on the face images, and then the face images corresponding to the spanning tree can be annotated. That is, determining the labels of the face images based on the face images and multi-dimensional information related to the face images includes:

[0157] C1. Generate a minimum spanning tree based on the face images contained in the face cluster, wherein each node of the minimum spanning tree corresponds one-to-one with the face image.

[0158] The Minimum Spanning Tree (MST) is a classic problem in graph theory. It refers to finding a subset of edges in a weighted undirected graph such that this subset forms a tree containing all the vertices in the graph, and the sum of the weights of the edges in this tree is minimized.

[0159] In this embodiment, the Kruskal algorithm or Prim's algorithm can be used to generate the minimum spanning tree. Of course, other algorithms can also be used, which will not be elaborated here.

[0160] In this embodiment, some face images can be selected from each face cluster, and then a minimum spanning tree can be generated based on the selected face images. Since face images are selected from each face cluster, the minimum spanning tree can cover each face cluster.

[0161] In this embodiment, a corresponding face image can be selected from more than half of the face clusters, and then a minimum spanning tree can be generated based on the selected face image. Since face images are selected from more than half of the face clusters, the minimum spanning tree can cover most of the face clusters.

[0162] In some embodiments, considering that face heat may be considered when recommending videos to users, face images that need to be labeled can be filtered by face heat. In this case, C1 above includes:

[0163] C11. Calculate the face heat corresponding to the face image in the above face cluster, wherein the face heat is related to the number of times the face image appears.

[0164] Optionally, face heat = number of times the face ID appears / total number of videos in the data source.

[0165] Optionally, face popularity can be determined based on the difference between the frequency of a face ID appearing on the current day and the frequency of its appearance in the previous two days.

[0166] Optionally, face popularity can be determined based on the ratio of the frequency of a face ID appearing on the current day to the historical frequency of that face ID, in order to exclude face IDs with a particularly high frequency every day, as well as face IDs with a particularly low frequency but a very high ratio of the current day's frequency to the historical frequency.

[0167] For example, suppose face IDs H1 and H2 appeared 1000 and 18 times respectively on the statistical day, and historically appeared 500 and 3 times respectively. Directly calculating the frequency ratio of the face IDs on the current day to their historical frequency yields 2 and 6, respectively. However, this clearly leans towards judging the face IDs as popular, so the ratio needs to be adjusted. Let's assume the ratio of the frequency of a face ID on the current day to its frequency over the entire statistical period is S(w). i ):

[0168] Among them, w i T represents a face ID. j Represents the time window, F(w) i ,T j ) indicates that the face ID is within the time window T j Frequency of occurrence, S(w) i This represents the face popularity of a certain face ID.

[0169] Regarding S(w) i After correction, we obtain S. ′ (w i ):

[0170] Where m is a prior average, which can be defined as S(w) of all face IDs. i The average value of ) can be obtained by using S. average This indicates that C is a constant, which can be defined as the average frequency of all face IDs over a day. In this case, F can be used. average Let I be the total number of face IDs, and J be the total number of days. Then:

[0171]

[0172] The above formula can be viewed as each face ID having a prior average value S. average The weight of its own personality score (S(w) i )-S average The score is determined by how well a face ID performs. Higher frequency, meaning greater exposure, gives it a higher weight. When a face ID's popularity score is equal to or less than the day's average popularity score S... average This face ID is definitely not a popular face ID, so it can be filtered out.

[0173] According to the formula above, if the frequency of a face ID is particularly low, much smaller than F... averageIf the original score is particularly high, the ID weight constraint of its score will be close to 0. Even if the original score is exceptionally high, the corrected score will be close to the average score, thus filtering out face IDs with very low frequency but large variation in number. If a face ID has a particularly high frequency and its word weight is close to 1, but this type of face is often common, the original score will be close to the average score, and the personality score will also be very low. The popularity score of the face will also be close to the average score, meaning this type of face ID will also be filtered out. ′ (w i The value of ) can be seen as the frequency of appearance of the final face ID. The larger the value, the higher the frequency.

[0174] C12. Filter the face images based on the face heat index and generate the minimum spanning tree based on the filtered face images.

[0175] After obtaining the face heat, the calculated face heat can be compared with a preset face heat threshold. If it is less than the face heat threshold, the face images corresponding to these face IDs are filtered out to obtain face IDs with higher face heat.

[0176] Optionally, before generating the minimum spanning tree, construct as follows: Figure 3 The undirected graph shown has nodes representing face IDs, and the edges can be set as the first weights corresponding to two nodes.

[0177] The first weight of the two nodes (or two face images) mentioned above can be calculated based on the information content and similarity of the two face images. For example, assuming cost_ij represents the first weight corresponding to face image i and face image j, info_i and info_j represent the information content corresponding to face image i and face image j respectively, and sim_ij represents the similarity between face image i and face image j, then cost_ij = info_i * info_j / sim_ij. Here, sim_ij can be obtained by the dot product of the vectors of face image i and face image j, and info_i = face heat. Here, face ID is the unique identifier of a face, and each face ID corresponds to a person's face. Since the first weight is related not only to face heat but also to the similarity between the two face images, and there is a certain relationship between different nodes of the minimum spanning tree, calculating the first weight can improve the accuracy of the generated minimum spanning tree.

[0178] To more clearly describe the undirected graphs described in the embodiments of this application, the following is combined with... Figure 3 Describe it.

[0179] exist Figure 3The diagram shows three face images: face image 1, face image 2, and face image 3. These three face images are equivalent to three nodes in an undirected graph. The values ​​of the edges between these three nodes are first weight 12, first weight 13, and first weight 23, respectively. The first weight 12 is calculated based on the information content of face image 1 and face image 2 and their corresponding similarity. The calculation process of the first weight 13 and the first weight 23 is similar and will not be described in detail here.

[0180] After obtaining the undirected graph, select any vertex of the undirected graph as the starting point and add it to the minimum spanning tree. Among all the edges connecting the minimum spanning tree to non-tree vertices, select the edge with the smallest weight. This edge adds a new vertex to the minimum spanning tree. Repeat this step until all vertices are included in the minimum spanning tree.

[0181] In some embodiments, considering that faces may be blurred before recommending videos to users, face images can be filtered by density-weighting. In this case, C1 above includes:

[0182] C11' Calculate the weight density corresponding to the face images in the above face cluster, where the weight density is used to reflect whether the above face images are outliers.

[0183] Weight density is used to indicate whether a particular face image may be an outlier or deviate significantly from most face images. These face images are less suitable as sample data, while denser, harder-to-distinguish face images are more valuable. For example... Figure 4 As shown in the box, the bottom face image 4c has a significant deviation from the top face images 4a and 4b, meaning that the weight density of the bottom face image 4c is less than that of the top face images 4a and 4b.

[0184] Optionally, the weight density is calculated according to the following formula:

[0185] in,

[0186] In the above formula,

[0187] In the above formula, This represents the weight density; arg is an abbreviation for argument (i.e., independent variable). argmin x argmax represents the value of variable x that minimizes the following expression. x This represents the value of variable x that makes the following expression reach its maximum value; x is a specific image of a human face.

[0188] x (u) This represents the representative element of the u-th face cluster, which is also the center of the face cluster. It can be directly obtained after clustering, sim(x, x). (u) ) indicates calculating x and x (u) The similarity is used to determine the facial image's identity. A lower similarity indicates a greater difference, and the image usually needs to be annotated accordingly.

[0189] P θ (y1|x) represents the probability that x is predicted to be y1, P θ (y2|x) represents the probability that x is predicted as y2, argmin x (P θ (y1|x)-P θ (y2|x)) represents the sample x corresponding to the minimum difference between the probability of being predicted as y1 and the probability of being predicted as y2. After obtaining this x, sim(x,x) is then calculated. (u) ) and the corresponding

[0190] C12': Filter the face images according to the above weight density, and generate the minimum spanning tree based on the filtered face images.

[0191] After obtaining the weight density, the weight density can be compared with a preset weight density threshold to select a weight density that is not less than the preset weight density threshold. An undirected graph is then generated based on the face image corresponding to the selected weight density, and a corresponding minimum spanning tree is generated based on the undirected graph.

[0192] C2. Based on the face image corresponding to the minimum spanning tree and the multi-dimensional information related to the face image, determine the label of the face image.

[0193] In this embodiment, since the minimum spanning tree is equivalent to finding a representative subset of face images from all face images in a face cluster as nodes of the minimum spanning tree (ignoring other parts of the permutation), determining the label corresponding to a face image based on the face image corresponding to the minimum spanning tree and the multi-dimensional information associated with that face image can reduce the number of face images that need to be labeled and reduce the impact on coverage. Furthermore, because the minimum spanning tree algorithm has low complexity, it can process large-scale graphs in a shorter time; that is, determining the label corresponding to a face image by generating the minimum spanning tree helps save the time required for label generation.

[0194] To more clearly describe the face image training set update method provided in the embodiments of this application, the following is combined with... Figure 5 and Figure 6 Please provide an explanation.

[0195] Figure 5 The diagram illustrates a flowchart of another face image training set update method provided in an embodiment of this application. Figure 6 The diagram illustrates the system framework of the face image training set update method provided in this application embodiment.

[0196] S51. Obtain raw face training data from different data sources.

[0197] like Figure 6 As shown, there are multiple data sources for raw face training data, including: open-source face datasets, short video datasets, mini-video datasets, and image datasets.

[0198] S52. Standardize the acquired raw face training data.

[0199] After obtaining raw facial training data from different data sources, these raw facial training data are usually standardized, for example, by unifying them.

[0200] S53. Preprocess the standardized raw face training data.

[0201] like Figure 6 As shown, the standardized raw face training data will enter the processing chain and be preprocessed by the data preprocessing system. This preprocessing specifically includes: parsing the video structured metadata and extracting frames from the video; obtaining the corresponding metadata based on the parsing results; and then performing low-quality filtering on the extracted image frames based on this metadata (e.g., filtering image frames containing sensitive information, filtering image-text mismatches, and filtering low-resolution image frames). The remaining image frames after filtering are then clustered to obtain the clustering results.

[0202] S54. Process the clustering results to generate a minimum spanning tree.

[0203] like Figure 6 As shown, the active learning filtering system analyzes the clustering results. If it determines that there are face clusters in the clustering results, it extracts these face clusters to obtain the corresponding face images. It analyzes whether the number of face images contained in the same face cluster is less than a preset threshold. If so, it performs secondary clustering (such as re-processing the video by extracting frames) until the number of face images contained in the same face cluster is not less than the preset threshold, at which point the secondary clustering stops. If not, no secondary clustering is required.

[0204] If the number of face images contained in the same face cluster is not less than a preset threshold, the face heat or weight density of these face images is calculated, the face images are filtered according to the calculation results, and finally the minimum spanning tree is generated based on the filtered face images.

[0205] S55. Obtain multi-dimensional information corresponding to the face image based on the face image corresponding to the minimum spanning tree, and perform multimodal iterative annotation based on the face image and the corresponding multi-dimensional information to obtain the label corresponding to the face image.

[0206] like Figure 6 As shown, the multimodal face annotation system can obtain multi-dimensional information corresponding to the face image based on the face image corresponding to the minimum spanning tree, and perform multimodal iterative annotation based on the face image and the corresponding multi-dimensional information to obtain the label corresponding to the face image.

[0207] S56. Perform unsupervised quality assessment on the face image corresponding to the minimum spanning tree to obtain the quality level corresponding to the face image. The quality level, label and face image are stored in the face dataset (i.e. the training set described above).

[0208] like Figure 6 As shown, the face quality system can perform unsupervised quality assessment based on the face image corresponding to the minimum spanning tree to obtain the quality level of the face image. The quality level, label and face image are stored in the face dataset (i.e. the training set described above).

[0209] Once a face model is trained using face data from a face dataset, if a consumer client subscribes to services related to that face model, the face model will be applied to that consumer client. This face model can be a model for face detection, face recognition, face attribute extraction, or other similar functions.

[0210] After the face model is applied to the consumer end, the evaluation result of the face model can be obtained through the face model effect evaluation. After the evaluation result is sent to the evaluation result acquisition module, the face dataset can be updated.

[0211] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0212] Corresponding to the face image training set update method described in the above embodiments, Figure 7The diagram shows a structural block diagram of a face image training set update apparatus provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0213] Reference Figure 7 The face image training set update device 7 includes:

[0214] The face image acquisition module 71 is used to acquire multiple face images.

[0215] The label determination module 72 is used to determine the label of any one of the aforementioned face images.

[0216] Quality level module 73 is used to determine the quality level of any of the aforementioned face images.

[0217] The sample data generation module 74 is used to generate corresponding sample data based on the labels and quality levels of each of the aforementioned face images.

[0218] The training set generation module 75 is used to generate a training set based on the aforementioned sample data.

[0219] The face model training module 76 is used to train the face model to be trained based on the above training set to obtain the trained face model.

[0220] The evaluation result acquisition module 77 is used to select whether to update the training set based on the evaluation result of the trained face model. The evaluation result is obtained based on the application feedback of the client to the trained face model. The application feedback includes information indicating whether the trained face model needs to be retrained.

[0221] In this embodiment, after acquiring multiple face images, the server determines the label and quality level of each face image, generates corresponding sample data based on the generated labels and quality levels, generates a training set based on the generated sample data, and trains the face model to be trained using the training set to obtain the trained face model. If the server receives the evaluation result of the trained face model sent by the client, it selects whether to update the training set based on the information included in the evaluation result indicating whether the trained face model needs to be retrained. Since updating the training set only occurs when the evaluation result indicates that the face model needs to be retrained, unnecessary updates are reduced when updating the training set in the above manner. Furthermore, since the evaluation result is determined based on the feedback obtained after the client runs the face model, the evaluation result reflects the actual running effect of the client, thereby improving the accuracy of the updated training set when updating the training set based on the evaluation result. Furthermore, since a higher quality level indicates a higher quality of sample data, meaning a higher probability that the sample data can be identified, when the sample data also includes quality level information, generating a training set based on this sample data and using the generated training set to train the face model to be trained is beneficial for selecting sample data with a more suitable quality level for the face model to be trained.

[0222] Optionally, the above evaluation result acquisition module 77 specifically includes:

[0223] If the above evaluation results indicate that the trained face model needs to be retrained, the update direction shall be determined based on the above evaluation results;

[0224] If the above update direction includes increasing the proportion of sample data with higher quality levels in the above training set, then based on the quality level distribution of the sample data in the above training set, the proportion of sample data with higher quality levels in the above training set is increased, and / or the proportion of sample data with lower quality levels in the above training set is decreased, to obtain the updated training set, wherein the proportion of sample data with higher quality levels in the updated training set is higher than the proportion of sample data with higher quality levels in the training set before the update.

[0225] Among them, the sample data with higher quality level refers to sample data with a quality level higher than the preset quality level threshold, and the sample data with lower quality level refers to sample data with a quality level not higher than the aforementioned quality level threshold.

[0226] Optionally, the above evaluation result acquisition module 77 specifically includes:

[0227] If the above evaluation results indicate that the trained face model needs to be retrained, the update direction shall be determined based on the above evaluation results;

[0228] If the above update direction includes increasing the proportion of sample data with specified labels in the training set, then based on the labels of the sample data in the training set, the proportion of sample data corresponding to the specified labels is increased, wherein the specified labels are used to indicate the category to which the sample data belongs.

[0229] Optionally, the face image acquisition module 71 described above is specifically used for:

[0230] Multiple facial images of the aforementioned individuals were obtained from the video.

[0231] The face image training set update device 7 provided in this application embodiment also includes:

[0232] Before determining the label of the aforementioned face image, multi-dimensional information related to the aforementioned face image is obtained from the video. The multi-dimensional information includes at least one of the following: audio information, subtitle information, and character information.

[0233] Correspondingly, when determining the label of the aforementioned face image, the label determination module is specifically used for:

[0234] Based on the aforementioned face image and the multi-dimensional information related to the aforementioned face image, the label of the aforementioned face image is determined.

[0235] Optionally, the above-mentioned acquisition of multiple facial images from the video includes:

[0236] The above video was processed by frame extraction to obtain multiple image frames;

[0237] Clustering is performed on the above image frames to obtain the clustering results;

[0238] If the above clustering results include face clusters, extract the face clusters to obtain the face images.

[0239] Optionally, after extracting the face clusters to obtain multiple face images, the method further includes:

[0240] If the number of face images contained in the same face cluster is less than a preset threshold, the video is re-processed by frame extraction until the number of face images contained in the same face cluster is not less than the threshold.

[0241] Optionally, determining the label of the face image based on the face image and multi-dimensional information related to the face image includes:

[0242] A minimum spanning tree is generated based on the face images contained in the face cluster mentioned above, and the nodes of the minimum spanning tree correspond one-to-one with the face images mentioned above.

[0243] Based on the face image corresponding to the minimum spanning tree and the multi-dimensional information related to the face image, the label of the face image is determined.

[0244] Optionally, generating a minimum spanning tree based on the face images contained in the face cluster includes:

[0245] Calculate the face heat corresponding to the face image in the above face cluster, wherein the face heat is related to the number of times the face image appears;

[0246] The face images are filtered based on the face heat index, and the minimum spanning tree is generated based on the filtered face images.

[0247] Optionally, generating the corresponding minimum spanning tree based on the aforementioned face cluster includes:

[0248] Calculate the weight density corresponding to the face images in the above face cluster, wherein the weight density is used to reflect whether the above face images are outliers;

[0249] The face images are filtered according to the weight density, and the minimum spanning tree is generated based on the filtered face images.

[0250] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0251] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device 8 of this embodiment includes: at least one processor 80 ( Figure 8 The diagram shows only one processor, a memory 81, and a computer program 82 stored in the memory 81 and executable on the at least one processor 80, which, when executing the computer program 82, performs the steps in any of the above method embodiments.

[0252] The electronic device 8 can be a desktop computer, laptop, handheld computer, or cloud server, etc. This electronic device may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will understand that... Figure 8 This is merely an example of electronic device 8 and does not constitute a limitation on electronic device 8. It may include more or fewer components than shown, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0253] The processor 80 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0254] In some embodiments, the memory 81 may be an internal storage unit of the electronic device 8, such as a hard disk or memory of the electronic device 8. In other embodiments, the memory 81 may be an external storage device of the electronic device 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 8. Furthermore, the memory 81 may include both internal and external storage units of the electronic device 8. The memory 81 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0255] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0256] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0257] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.

[0258] This application provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.

[0259] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0260] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0261] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0262] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0263] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0264] It should be noted that the information collection process (such as the facial image collection process) / feature extraction process involved in this application is performed with the user's knowledge and permission. That is, the information collection process / feature extraction process complies with the requirements of laws and regulations and does not constitute an act that harms the public interest.

[0265] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for updating a training set of face images, characterized in that, The method comprises: obtaining a plurality of face images; for any one of the face images, determining a label of the face image; for any one of the face images, determining a quality level of the face image; generating corresponding sample data according to the labels and the quality levels of the face images; generating a training set according to the sample data; training a face model to be trained according to the training set to obtain a trained face model; in a case where an evaluation result of the trained face model is obtained, selecting whether to update the training set according to the evaluation result, wherein the evaluation result is obtained based on application feedback of the trained face model by a client, and the application feedback comprises information indicating whether the trained face model needs to be retrained.

2. The face image training set updating method according to claim 1, characterized in that, The method comprises: in a case where the evaluation result indicates that the trained face model needs to be retrained, determining an update direction according to the evaluation result; if the update direction comprises increasing a proportion of sample data with a higher quality level in the training set, increasing the proportion of sample data with a higher quality level in the training set and / or reducing a proportion of sample data with a lower quality level in the training set based on a quality level distribution of sample data in the training set to obtain an updated training set, wherein the proportion of sample data with a higher quality level in the updated training set is higher than the proportion of sample data with a higher quality level in the training set before updating. The sample data with a higher quality level is sample data with a quality level higher than a preset quality level threshold, and the sample data with a lower quality level is sample data with a quality level not higher than the quality level threshold. 3.The method of claim 1, wherein, The method comprises: in a case where the evaluation result indicates that the trained face model needs to be retrained, determining an update direction according to the evaluation result; if the update direction comprises increasing a proportion of sample data with a specified label in the training set, increasing the proportion of sample data corresponding to the specified label based on labels of sample data in the training set, wherein the specified label is used to indicate a classification to which the sample data belongs.

4. The face image training set updating method according to claim 1, wherein the method comprises: obtaining a plurality of face images from a video; before the label of the face image is determined, the method further comprises: obtaining multi-dimensional information related to the face image from the video, wherein the multi-dimensional information comprises at least one of the following information: audio information, subtitle information, and character role information; the method comprises: determining the label of the face image according to the face image and the multi-dimensional information related to the face image. 5.The method of Claim 4, wherein, the method comprises: performing frame extraction processing on the video to obtain a plurality of image frames; performing clustering processing on the plurality of image frames to obtain a clustering result; In a case where the clustering result includes a face cluster, the face cluster is extracted to obtain a plurality of face images. 6.The method of Claim 5, wherein, After the face cluster is extracted to obtain a plurality of face images, the method further includes: In a case where the number of face images included in the same face cluster is less than a preset number threshold, the video is re-processed by frame extraction until the number of face images included in the same face cluster is not less than the number threshold.

7. The face image training set updating method according to claim 5 or 6, characterized in that, The method further includes: A minimum spanning tree is generated according to the face images included in the face cluster, and nodes of the minimum spanning tree correspond to the face images one by one. Labels of the face images are determined according to the face images corresponding to the minimum spanning tree and multi-dimensional information related to the face images. 8.The method of Claim 7, wherein, The method further includes: A face heat corresponding to a face image in the face cluster is calculated, wherein the face heat is related to a number of times the face image appears. The face images are screened according to the face heat, and the minimum spanning tree is generated according to the screened face images. 9.The method of Claim 7, wherein The method further includes: A weight density corresponding to a face image in the face cluster is calculated, wherein the weight density is used to reflect whether the face image is an abnormal point. The face images are screened according to the weight density, and the minimum spanning tree is generated according to the screened face images. 10.A face image training set updating apparatus, characterized by comprising: The method further includes: A face image acquisition module is configured to acquire a plurality of face images. A label determination module is configured to determine a label of each face image. A quality level module is configured to determine a quality level of each face image. A sample data generation module is configured to generate corresponding sample data according to the label and the quality level of each face image. A training set generation module is configured to generate a training set according to each sample data. A face model training module is configured to train a to-be-trained face model according to the training set to obtain a trained face model. An evaluation result acquisition module is configured to, in a case where an evaluation result of the trained face model is acquired, select whether to update the training set according to the evaluation result, wherein the evaluation result is obtained based on application feedback of a client to the trained face model, and the application feedback includes information indicating whether the trained face model needs to be retrained.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1 to 9.

12. A computer-readable storage medium, the computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 9.

13. A computer program product, when the computer program product runs on an electronic device, causes the electronic device to execute the method of any one of claims 1 to 9.

Images