Face image processing method and device

By clustering and labeling multi-angle face images, forming a face image group and training a face recognition model, the problem of low recognition accuracy of multi-angle face images in the prior art is solved, and higher recognition accuracy is achieved.

CN110472460BActive Publication Date: 2024-11-19BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201810453319.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2018-05-11
Publication Date
2024-11-19
Estimated Expiration
2038-05-11

AI Technical Summary

Technical Problem

The existing face recognition model has low recognition accuracy in multi-angle face image recognition scenes, and cannot effectively process complex face images captured by surveillance cameras.

Method used

By acquiring multiple face images from different angles, clustering and annotating, forming face image groups, removing error clustering results, improving data labeling accuracy, and using the face image groups obtained by clustering to train the face recognition model .

Benefits of technology

The accuracy of multi-angle face image recognition is improved, ensuring that the face images in the face image group correspond to the same person, avoiding the labeling errors of the training data, and improving the recognition accuracy of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110472460B_ABST
    Figure CN110472460B_ABST
Patent Text Reader

Abstract

The present disclosure provides a method and device for processing facial images, the method comprising: obtaining multiple facial images, the multiple facial images including facial images of at least one person at different angles; clustering the multiple facial images to obtain multiple facial image groups; and removing part of the facial images in at least one facial image group according to user input and / or preset rules, so that each facial image in each facial image group in the at least one facial image group corresponds to the same person. Using the solution provided by the present disclosure, multiple facial image groups can be obtained, wherein multiple facial images of different angles in a facial image group correspond to one person, so as to facilitate the use of facial image groups to train a facial recognition model that can recognize facial images of different angles.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a method and device for processing facial images. Background Art

[0002] In offline unmanned retail and smart retail scenarios, facial recognition for customer payment needs to be realized, and its technical basis is high-precision recognition and detection of customer faces. In the above scenarios, facial information is generally collected by surveillance cameras installed in the store, which is different from standard facial data captured in experimental environments, such as ID photos, which have the following characteristics: frontal face, no downward angle, single background, clear facial features, no obstruction, etc. However, the facial images captured by surveillance cameras in the above open environments are often extremely complex and far from the standard of ID photos.

[0003] The training data sets of existing face recognition algorithms are generally standard face recognition data sets. In these data sets, the face shooting scenes are too simple compared to the scenes shot by surveillance cameras, resulting in the inability to cover the multi-angle face images captured by surveillance cameras. This results in low recognition accuracy when the model trained with the standard face recognition data set is used to recognize face images taken by surveillance cameras.

[0004] In the process of realizing the concept disclosed in the present invention, the inventors found that there are at least the following problems in the prior art: in scenarios such as smart retail, the facial images of users collected are at various angles, and the recognition accuracy of facial images taken by surveillance cameras using a model trained using an existing standard facial recognition data set is very low. Summary of the invention

[0005] In view of this, the present disclosure provides a facial image processing method and device to solve the problem of low recognition accuracy in scenarios where facial images with multiple angles are recognized using a model trained using a standard facial recognition data set in the prior art.

[0006] One aspect of the present disclosure provides a method for processing facial images, including: first, obtaining multiple facial images, in order to obtain multiple facial images of different angles for each person, the multiple facial images are required to include facial images of different angles for at least one person; then, clustering the multiple facial images obtained to obtain multiple facial image groups. Since each person has his or her own special facial features, the clustering result can make the facial images of different angles for each person be clustered in the same facial image group after clustering, so that the function of automatically annotating the multiple facial images is actually realized, for example, facial image group 1 and multiple facial images of different angles corresponding to facial image group 1. In addition, after obtaining the multiple facial image groups, according to user input and / or preset rules, some facial images in at least one facial image group are removed, so that each facial image in each facial image group in the at least one facial image group corresponds to the same person, so as to correct the erroneous clustering results and improve the accuracy of data annotation.

[0007] According to an embodiment of the present disclosure, the acquisition of multiple facial images may include the following operations: acquiring a video, which may be a video captured by a camera in a facial recognition environment; and acquiring facial images in the video in frames. Since people are often active in videos, the multiple frames captured often contain facial images of the person at different angles. Specifically, the acquisition of facial images in the video in frames may include the following operations: based on experience, experimental results, etc., acquiring video frames of the video frame by frame or at intervals of a set number of frames, so that the video frames contain high-quality facial images of the same person at different angles; and then acquiring facial images of the video frames to obtain high-quality facial images of the same person at different angles.

[0008] According to an embodiment of the present disclosure, the method may further include the following operations: after obtaining the multiple face image groups, respectively obtaining the group center of at least one face image group, and then using the group center as the identifier of the corresponding face image group, so that the face image groups can be labeled using the group center of the human image group, that is, the labeling information of each face image group can be the group center of the face image group. For example, the face image group includes: the group center and multiple face images at different angles corresponding to the group center.

[0009] According to an embodiment of the present disclosure, the group center of the at least one face image group may be obtained by the following scheme, for example, including the following operations: for each face image group in the at least one face image group:

[0010] First, the distances between each face image and other face images in the face image group are calculated respectively, and then the face image with the smallest sum of distances from other face images in the face image group is taken as the group center.

[0011] According to an embodiment of the present disclosure, the method may further include the following operations: after obtaining the group center, for the face image group, respectively calculating the rotation angle of each face image in the face image group relative to the group center; and then using the obtained rotation angle to perform angle annotation on the corresponding face image. That is, the face image group in this embodiment may also include annotation information of the rotation angle, and the annotation information may be applied in scenarios where rotation angle annotation information is required.

[0012] According to an embodiment of the present disclosure, a specific scheme for calculating the rotation angle is also provided, which may include the following operations for each facial image: first, obtaining a specified number of key points of the facial image and a specified number of key points of the group center, then obtaining the positional relationship between the key points of the facial image and the positional relationship between the key points of the group center, and then determining the rotation angle of the facial image relative to the group center based on the positional relationship between the key points of the facial image and the positional relationship between the key points of the group center.

[0013] According to an embodiment of the present disclosure, the method may further include the following operations: for the facial image group, sorting the facial images in the facial image group according to the rotation angle and / or distance between the facial images and the center of the group; and displaying the sorted facial images.

[0014] According to an embodiment of the present disclosure, the method may further include the following operations: after obtaining multiple face image groups, using the multiple face image groups to train a face recognition model; specifically, after obtaining the multiple face image groups, the face features of face images at different angles corresponding to the face image groups, and the face image groups to which they belong are input into the first face recognition model for training to obtain model parameter values.

[0015] Accordingly, another aspect of the present disclosure provides a facial image processing device, which may include the following modules: an acquisition module, configured to acquire a plurality of facial images, wherein the plurality of facial images include facial images of at least one person at different angles;

[0016] a clustering module, configured to cluster the plurality of face images to obtain a plurality of face image groups; and

[0017] The correction module is used to remove part of the face images in at least one face image group according to user input and / or preset rules, so that each face image in each face image group in the at least one face image group corresponds to the same person.

[0018] According to an embodiment of the present disclosure, the acquisition module may include the following units:

[0019] A video acquisition unit, used for acquiring video;

[0020] A video frame acquisition unit, configured to acquire video frames of the video frame by frame or at intervals of a set number of frames; and

[0021] The face image acquisition unit is used to acquire the face image of the video frame.

[0022] According to an embodiment of the present disclosure, the device may also include the following modules: a group center acquisition module connected to the clustering module, and an identification module, wherein the group center acquisition module is used to respectively obtain the group center of at least one face image group; and the identification module is used to use the group center as the identification of the corresponding face image group.

[0023] According to an embodiment of the present disclosure, the group center acquisition module may include the following units: a distance calculation unit and a center acquisition unit, wherein the distance calculation unit is used to calculate the distance between each face image and other face images in each face image group in the at least one face image group; and the center acquisition unit is used to take the face image with the smallest sum of distances from other face images in the face image group as the group center.

[0024] According to an embodiment of the present disclosure, the device may further include the following modules: a rotation angle calculation module connected to the group center acquisition module, and an angle marking module. The rotation angle calculation module is used to calculate the rotation angle of each face image in the face image group relative to the group center; the angle marking module is used to use the obtained rotation angle to mark the angle of the corresponding face image.

[0025] According to an embodiment of the present disclosure, the rotation angle calculation module may include the following units: a key point acquisition unit, a position relationship acquisition unit and a rotation angle calculation unit, wherein the key point acquisition unit is used to obtain, for each face image, a specified number of key points of the face image and a specified number of key points of the group center; the position relationship acquisition unit is used to obtain the position relationship between the key points of the face image and the position relationship between the key points of the group center; the rotation angle calculation unit is used to determine the rotation angle of the face image relative to the group center based on the position relationship between the key points of the face image and the position relationship between the key points of the group center.

[0026] According to an embodiment of the present disclosure, the device may also include the following modules: a sorting module and a display module, wherein the sorting module is used to sort the facial images in the facial image group according to the rotation angle and / or distance between the facial images and the center of the group; and the display module is used to display the sorted facial images.

[0027] According to an embodiment of the present disclosure, the device may further include the following modules: a face recognition model training module, configured to train a face recognition model using the multiple face image groups after obtaining the multiple face image groups.

[0028] Another aspect of the present disclosure provides a face image processing device for face image processing, which may include:

[0029] One or more processors; and a storage device for storing executable instructions, wherein when the executable instructions are executed by the processors, any of the methods described above is implemented.

[0030] Another aspect of the present disclosure provides a non-volatile storage medium storing computer executable instructions, which are used to implement any of the above methods when executed.

[0031] Another aspect of the present disclosure provides a computer program, which includes computer executable instructions, and the instructions are used to implement the method described above when executed.

[0032] According to the embodiments of the present disclosure, since each person has his or her own special facial features, the clustering results can make the facial images of each person at different angles be clustered in the same facial image group after clustering, so that the function of automatically annotating the multiple facial images is actually realized, for example, facial image group 1 and multiple facial images at different angles corresponding to facial image group 1, so that the facial image group obtained by clustering can be used to train the facial recognition model. For example, the facial features of the multi-angle facial images of one or more people and the facial image groups to which they belong can be input into a pre-built facial recognition model for model training, and a model that can meet the multi-angle facial image recognition can be trained, which can at least partially solve the problem that when the model trained by using the existing standard facial recognition data set performs facial recognition on the facial images taken by the surveillance camera, the recognition accuracy is very low due to the presence of facial images at different angles, and thus the technical effect of improving the recognition accuracy of multi-angle facial images can be achieved.

[0033] According to the embodiments of the present disclosure, since it is further checked whether the multi-angle facial images corresponding to each person are correct, if they are incorrect, the accuracy of the facial images in the facial image group is improved by removing the incorrect facial images. This can ensure as much as possible that the facial images in the facial image group correspond to the same person, avoid labeling errors in the training data, and achieve the technical effect of improving the recognition accuracy of multi-angle facial images.

[0034] According to an embodiment of the present disclosure, a specific method for obtaining multiple facial images is further provided. The method can simulate an actual facial recognition environment, thereby acquiring a video that is the same or similar to the actual facial recognition environment, and further acquiring facial images that are the same as the actual facial recognition environment. This helps to improve the similarity between the facial images in the facial image group with annotated information obtained using the present disclosure and the facial images collected in the actual usage environment, so that the model recognition accuracy of the facial recognition model trained using the facial image group provided by the present embodiment is higher.

[0035] According to the embodiments of the present disclosure, after obtaining the clustering results, the group centers of the face image groups are respectively obtained, and the group centers can be used as identifiers of the corresponding face image groups, which makes it easy to check whether the face image groups obtained by clustering are correct.

[0036] According to the embodiments of the present disclosure, after obtaining the group center of the face image group, the rotation angle of each face image in the current face image group relative to the group center is calculated, and then the current face image is angle-labeled using the obtained rotation angle. In this way, the training set obtained by the present disclosure also includes the rotation angle information of individual face images. Therefore, in specific scenarios, such as in a specific payment environment, when there is a certain rotation angle between the camera and the payer, face images with specific rotation angles can be selected from the training set to optimize the model, which helps to improve the recognition accuracy of the model.

[0037] According to an embodiment of the present disclosure, a training method for multiple face recognition models is also provided. The training method can simply and efficiently train the face recognition model. The trained face recognition model can recognize face images from multiple angles with high recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0039] Figure 1 An exemplary system architecture to which face image processing can be applied according to an embodiment of the present disclosure is schematically shown;

[0040] Figure 2Schematically showing facial images at different angles according to an embodiment of the present disclosure;

[0041] Figure 3A A flowchart of a method for processing a face image according to an embodiment of the present disclosure is schematically shown;

[0042] Figure 3B A flowchart of a method for processing a face image according to another embodiment of the present disclosure is schematically shown;

[0043] Figure 3C A schematic diagram schematically shows a face image group according to an embodiment of the present disclosure;

[0044] Figure 4A A flowchart of a method for acquiring multiple face images according to an embodiment of the present disclosure is schematically shown;

[0045] Figure 4B A flowchart of a method for acquiring multiple face images according to another embodiment of the present disclosure is schematically shown;

[0046] Figure 4C A flowchart of a method for acquiring a face image from a video frame according to an embodiment of the present disclosure is schematically shown;

[0047] Figure 5 A flowchart of a method for processing a face image according to another embodiment of the present disclosure is schematically shown;

[0048] Figure 6 A flowchart of a method for obtaining a group center of a face image group according to an embodiment of the present disclosure is schematically shown;

[0049] Figure 7 A flowchart of a method for processing a face image according to another embodiment of the present disclosure is schematically shown;

[0050] Figure 8 A flow chart of a method for calculating a rotation angle according to an embodiment of the present disclosure is schematically shown;

[0051] Fig. 9 A schematic diagram schematically shows a face image and key points in a video frame according to an embodiment of the present disclosure;

[0052] Fig.10 A schematic diagram schematically illustrates key points according to an embodiment of the present disclosure;

[0053] Fig.11 A flowchart of a method for processing a face image according to another embodiment of the present disclosure is schematically shown;

[0054] Fig.12A flowchart of a method for processing a face image according to another embodiment of the present disclosure is schematically shown;

[0055] Fig.13 A block diagram schematically shows a face image processing device according to an embodiment of the present disclosure;

[0056] Fig.14 A block diagram schematically shows a face image processing device according to another embodiment of the present disclosure;

[0057] Fig.15 A block diagram schematically shows a face image processing device according to another embodiment of the present disclosure;

[0058] Fig.16 A block diagram schematically shows a face image processing device according to another embodiment of the present disclosure;

[0059] Fig.17 A block diagram schematically shows a face image processing device according to another embodiment of the present disclosure;

[0060] Fig.18 A block diagram schematically shows a face image processing device according to another embodiment of the present disclosure;

[0061] Fig.19 A block diagram schematically shows a face image processing device according to another embodiment of the present disclosure;

[0062] Fig. 20 A block diagram schematically shows a face image processing device according to another embodiment of the present disclosure;

[0063] Fig.21 A block diagram schematically shows a face image processing device according to another embodiment of the present disclosure;

[0064] Fig. 22 The block diagram schematically shows a computer system suitable for implementing the face image processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0065] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0066] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0067] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0068] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). Those skilled in the art should also understand that any transitional conjunctions and / or phrases that substantially indicate two or more optional items, whether in the specification, claims, or drawings, should be understood to give the possibility of including one of these items, either of these items, or both of these items. For example, the phrase "A or B" should be understood to include the possibilities of "A" or "B", or "A and B".

[0069] The embodiments of the present disclosure provide a method and device for facial image processing, which can automatically cluster multiple facial images to obtain multiple human image groups corresponding to the multiple facial images. Since each person has unique facial features, when the facial images of the same person are at different angles, the facial features of the facial images of the same person at different angles still have a strong correlation. Therefore, the multiple facial images can be clustered accordingly, so that the clustering result is that the facial images of the same person at different angles are clustered in the same facial image group. In this way, it is possible to automatically annotate multiple facial images: which facial image group a facial image should correspond to, wherein the facial images in each facial image group correspond to the same person. The obtained facial image group can be used to train a facial recognition model. Since the facial image group contains facial images of the same person at different angles, the recognition accuracy of the facial recognition model trained thereby is higher than that of the existing facial recognition model trained based on standard facial images in the scene of multi-angle facial recognition.

[0070] Figure 1 The exemplary system architecture 100 to which face image processing can be applied according to an embodiment of the present disclosure is schematically shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0071] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The system architecture 100 can be applied to a company's facial attendance punch-in system, community management, building access control system, etc. Since multi-angle facial image recognition can be achieved, it is more convenient for users to use it, and it is not necessary to face the camera directly, which can improve the user experience; in addition, it can also be applied to offline store systems, such as member identification, face payment, etc. For example, when a user enters a store, the surveillance camera collects and identifies whether it is a store member, and the surveillance camera identifies the payer to confirm the payer's identity information, which facilitates the implementation of the face payment method.

[0072] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as image acquisition applications, payment applications, shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0073] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc., wherein terminal devices 101, 102, 103 can also include information acquisition devices, for example, image acquisition devices, sound acquisition devices, etc. Specifically, the image acquisition device can be an external camera, such as a surveillance camera, or an airborne camera, such as a camera built into a laptop computer, etc., and the sound acquisition device can be a microphone, etc.

[0074] The server 105 may be a server that provides various services, such as clustering the facial images collected by the user using the terminal devices 101, 102, and 103, performing frame processing on the collected video, identifying and obtaining facial images from video frames, obtaining the group center of the facial image group, calculating the rotation angle of each facial image in the facial image group relative to the group center, constructing a facial recognition model, training the facial recognition model, and performing facial recognition using the trained facial recognition model. The backend management server may analyze and process the received user request and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device.

[0075] It should be noted that the face image processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the clustering module of the face image processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The face image processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the face image processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0076] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0077] Figure 2The face images at different angles according to the embodiment of the present disclosure are schematically shown. As can be seen from the figure, Figure 2 The five face images in the figure correspond to the same person, but the angle of the face in each image is different. When the face recognition model trained by the prior art is used to perform face recognition on the above five face images, the five face images in the figure may be judged as five different people. This is because the training data set of the prior art is generally a standard face recognition data set, such as: ALFW, CASIA-WebFace, MS-Celeb-1M and other standard face recognition data sets, and the face images in the standard face recognition data set usually include one or more of the following features: front face, no depression angle, single background, clear facial features or no occlusion, such as ID photos. The face recognition model trained by these standard face recognition data sets has a good effect when recognizing standard faces. However, in the prior art, the demand for face recognition is not only for standard face images. For example, in the scenarios of offline unmanned retail and smart retail, the face recognition authentication of customers' face payment needs to be realized, and its technical basis is the high-precision recognition and detection of customer faces. In the above scenarios, the collection of face information is generally through surveillance cameras deployed in the store. Different from the standard face images captured in the standard face image acquisition environment, the face images captured by the surveillance cameras in the above open environment often cannot meet the acquisition requirements of the standard face images. For example, the faces in the collected face images may be frontal, sideways, or crooked, etc. In addition, the cameras are often placed on the wall to produce a low angle of shooting, etc. In summary, the existing face recognition models trained based on the standard face recognition dataset cannot meet the face recognition needs in scenarios such as offline unmanned retail and smart retail.

[0078] The embodiments of the present disclosure provide a facial image processing method and device. Considering that each person has his or her own special facial features, even when the facial image is at a certain angle relative to the frontal face image, there is still a large correlation between the facial features of the facial images of the same person at different angles. Therefore, the clustering result can make the facial images of each person at different angles be clustered in the same facial image group after clustering, so that the function of automatically annotating the multiple facial images is actually realized, for example, facial image group 1 and multiple facial images at different angles corresponding to facial image group 1. In this way, the facial image group obtained by clustering can be used to train a facial recognition model. For example, the facial features of the multi-angle facial images of one or more people and the facial image group to which they belong can be input into a pre-built facial recognition model for model training, and a model that can meet the multi-angle facial image recognition can be trained. This can at least partially solve the problem that when the model trained by using the existing standard facial recognition data is trained on the facial images taken by the surveillance camera, the recognition accuracy is very low due to the presence of facial images at different angles, and thus the technical effect of improving the multi-angle facial image recognition accuracy can be achieved.

[0079] Figure 3A The flowchart of the method for processing a face image according to an embodiment of the present disclosure is schematically shown.

[0080] like Figure 3A As shown, the method includes, in operation S301, acquiring a plurality of face images, wherein the plurality of face images include face images of at least one person at different angles.

[0081] Specifically, the multiple facial images may be photos of at least one person at multiple different angles taken by a camera device, or videos of at least one person at different angles recorded by a video recording device, and then facial images at different angles obtained from the video frames. In addition, it may also be multiple pictures of at least one person captured from the Internet, for example, first obtain multiple facial images and corresponding identity information from the Internet, and then select a specified number of identity information as candidate facial images based on identity information, such as mobile phone number, name, identification number, etc., and then obtain multiple pictures of at least one person including facial images at different angles through one or more operations such as identity information comparison, removal of duplicate pictures, and selection of facial images at different angles, and then obtain facial images from the pictures.

[0082] Then, in operation S302, the plurality of face images are clustered to obtain a plurality of face image groups.

[0083] Specifically, since each person has his or her own special facial features, the facial features of the face images of the same person at different angles are correlated. Therefore, the clustering result can make the face images of each person at different angles be clustered in the same face image group after clustering, so that the function of automatically annotating the multiple face images is actually realized, for example, face image group 1 and the multiple face images at different angles corresponding to face image group 1, ... face image group n and the multiple face images at different angles corresponding to face image group n, where n is a positive integer ≥ 1. The clustering method can be the same as the existing technology, for example, first extracting facial features from each face image, and then clustering the facial features to obtain multiple face image groups, and the face images in each face image group correspond to the same person.

[0084] It should be noted that if Figure 3B As shown, a flowchart of a method for processing a face image according to another embodiment of the present disclosure is schematically shown. In this embodiment, the method for processing a face image may further include the following operations:

[0085] In operation S303, part of the facial images in at least one facial image group are removed according to user input and / or preset rules, so that each facial image in each facial image group in the at least one facial image group corresponds to the same person.

[0086] Since the clustering results may contain errors, for example, facial images of different persons are clustered in the same facial image group, in order to improve the accuracy, the facial images in the same facial image group can be checked and corrected manually and / or based on certain rules to ensure that the facial images in each facial image group correspond to the same person. Figure 3C As shown, a schematic diagram of a face image group according to an embodiment of the present disclosure is schematically shown. Specifically, each face image group can be displayed by the above-mentioned terminal devices 101, 102, 103, etc., wherein the face image groups can be displayed in multiple rows or columns in the candidate area, wherein the identifier of the face image group can be the face image group name, for example, face image group 1, ... face image group n, etc., and can also be a face image in the face image group as the face image group identifier, for example, the face image closest to the front face, the face image with the smallest sum of distances from other face images in the face image group, etc. In addition, the displayed content can also include an identification result area for displaying one or more specific face image groups, and can also be used to correct and review face images that are automatically labeled incorrectly, so as to better improve the accuracy of the face image group.

[0087] The face image processing method provided by the embodiments of the present disclosure can use the face image group obtained by clustering to train the face recognition model. For example, the face features of multi-angle face images of one or more persons and the face image group to which they belong can be input into a pre-built face recognition model for model training. A model that can meet the requirements of multi-angle face image recognition can be trained. This can at least partially solve the problem that when the model trained using the existing standard face recognition data set performs face recognition on face images taken by surveillance cameras, the recognition accuracy is very low due to the presence of face images at different angles. Therefore, the technical effect of improving the recognition accuracy of multi-angle face images can be achieved.

[0088] Reference below Figure 4A to Figure 4C , combined with specific embodiments Figure 3A The method shown is further explained. Figure 4A In the figure, a flowchart of a method for acquiring multiple face images according to an embodiment of the present disclosure is schematically shown.

[0089] In this embodiment, the acquiring of multiple face images may include the following operations:

[0090] In operation S401, a video is obtained. Among them, a camera can be used to rotate around a person to shoot, and a video containing multiple facial images of the person at different angles can be obtained; in addition, a fixed camera can be used to let a person rotate at a certain angle in front of the camera, so that the camera can shoot a video containing multiple facial images of the person at different angles. In addition, in order to make each facial image in the final facial image group more similar to the facial image obtained in the actual use environment, if the layout of the specific use environment is known, the same camera arrangement as the actual use environment can be used, or the camera of the actual use environment can be directly used, and then similar crowd flow situations, such as crowd flow situations for face payment, can be used to shoot multiple videos containing at least one person at different angles. The facial images in the facial image group obtained thereby are more consistent with the facial images obtained in the actual application scenario, and the recognition accuracy of the facial recognition model trained thereby is higher.

[0091] In operation S402, a face image in the video is acquired in units of frames.

[0092] In this embodiment, the face image in each frame can be extracted from each of the at least two frames. Since different moments correspond to faces at different angles during video shooting, the angles of the face images in different frames can be different, and face images at different angles can be extracted from different frames. Figure 4B In the figure, a flowchart of a method for acquiring multiple facial images according to another embodiment of the present disclosure is schematically shown.

[0093] In operation S401, the video is obtained, which may be the same as in the previous embodiment and will not be described in detail here.

[0094] In operation S403, video frames of the video are acquired frame by frame or at intervals of a set number of frames. In this embodiment, video frames frame1, frame2, ..., frameN can be acquired frame by frame or at intervals of a set number of frames, where N is a positive integer greater than or equal to 2. Since the payer does not appear in the video for a long time in the case of face payment, in order to obtain as many facial images of the payer from different angles as possible, the video frames of the video can be acquired frame by frame.

[0095] In operation S404, a face image of the video frame is obtained. In this embodiment, an existing method for obtaining a face image from an image may be used.

[0096] exist Figure 4C In the figure, a flow chart of a method for obtaining a face image from a video frame according to an embodiment of the present disclosure is schematically shown. In this embodiment, the following operations may be included:

[0097] In operation S4041, a face detection frame is set. Specifically, the size of the face detection frame can be determined according to the distance between the camera and the face, or according to other rules, so that the window of the face detection frame is a face window if and only if it just frames a face, that is, the size of the window is consistent with the size of the face, and the window basically fits the outer contour of the face. In addition, when multiple face images appear in the video, and the distance between each person and the camera is different, the sizes of the face images in the frame image will be different. At this time, the face images cannot be detected with the same size of face detection frames. At this time, face detection frames of different sizes can be used to detect faces, or the same size of face detection frames can be used and the frame image can be scaled to detect the face images in the frame image.

[0098] In operation S4042, the facial features of the portion of the video frame covered by the face detection frame are extracted. In this embodiment, facial features are extracted, and the facial features correspond to the above-mentioned facial features. The facial features are vector representations of facial features obtained by extracting features from a face image by a computer, i.e., feature vectors, such as X=(x1, x2, ..., xd), where d is a positive integer greater than or equal to 1, and each dimension is a numerical value, which is calculated based on the input (image area), such as summing, subtracting, comparing sizes, etc. The feature extraction process is a process of transforming the original input data (a matrix composed of the color values ​​of the image area) to the corresponding feature vector.

[0099] In operation S4043, the facial features are input into a pre-trained facial model to obtain a recognition result. The facial model may be a classifier, a neural network, a random forest model, etc. Taking a classifier as an example, the classifier takes a feature vector as input and uses a category as output through mathematical calculation. Each category corresponds to a numerical code, which is called a label corresponding to the category. For example, a face window is coded as 1, and a non-face window is coded as -1. A classifier is a function that transforms a feature vector into a category label, as shown in formula (1):

[0100]

[0101] Among them, the parameter t can be obtained by model training. For example, the correct face image is labeled as 1, and the incorrect face image is labeled as -1, and then the above-mentioned face image with labeled information is input into the above-mentioned model until the consistency between the model output result and the labeled information reaches a set threshold.

[0102] In operation S4044, a face image is determined according to the recognition result. Specifically, an image in a frame corresponding to a face detection frame of a face image whose output result is a model is taken as the face image.

[0103] In a specific embodiment, clustering the plurality of face images includes: clustering the face features of the face images based on a K-means or hierarchical clustering method to obtain at least one face image group and a face image corresponding to each face image group. In this way, clustering the plurality of face images can be achieved to obtain at least one face image group.

[0104] like Figure 5 FIG. 2 schematically shows a flow chart of a method for processing facial images according to another embodiment of the present disclosure. In this embodiment, after obtaining the plurality of facial image groups, the method may further include the following operations:

[0105] In operation S501, the group center of at least one face image group is obtained respectively. Specifically, the group center may be a face image with the smallest sum of distances from other face images in the face image group, wherein the distance may be a cosine distance between face features (feature vectors) of two face images, etc. The larger the distance, the greater the difference between the two face images, and the smaller the distance, the more similar the two face images are.

[0106] In operation S502, the group center is used as an identifier of the corresponding face image group.

[0107] Since the group center is the face image with the smallest sum of distances from other face images in the face image group, that is, the group center can better represent other face images in the face recognition group, it is more representative to use it as the identifier of the face image group.

[0108] like Figure 6 FIG. 1 schematically shows a flow chart of a method for obtaining a group center of a face image group according to an embodiment of the present disclosure. In this embodiment, for each face image group in the at least one face image group, the following operations may be included:

[0109] In operation S601, the distance between each face image and other face images in the face image group is calculated respectively. Specifically, assuming that there are m face images in a face image group, there are m corresponding face features: X1, X2, ...Xm, where X1 is the face feature of the first face image, which is an n-dimensional feature vector, X1 = (a1, a2, ...an), X2 is the face feature of the second face image, X2 = (b1, b2, ...bn), then the distance between the first face image (face 1) and the second face image (face 2) is as shown in formula (2):

[0110]

[0111] Wherein, i is a positive integer ≥1.

[0112] In operation S602, the face image having the smallest sum of distances from other face images in the face image group is taken as the group center.

[0113] First, the average distance between each face image in the face image group and other face images in the face image group is calculated; taking a face image group with only 3 face images as an example, the above average distance can be obtained by formula (3):

[0114]

[0115] Among them, L_face1 is the average distance of face image 1 relative to other face images in the face image group, L_face2 is the average distance of face image 2 relative to other face images in the face image group, and L_face3 is the average distance of face image 3 relative to other face images in the face image group.

[0116] Accordingly, the center face can be determined by formula (4):

[0117] Center face = min(L_face1, L_face2, L_face3) (4)

[0118] The group center can be obtained through the above operations.

[0119] like Figure 7 , schematically shows a flow chart of a method for processing a face image according to another embodiment of the present disclosure. In this embodiment, after obtaining the group center, the method may further include the following operations:

[0120] In operation S701, for the face image group, the rotation angle of each face image in the face image group relative to the group center is calculated respectively. Specifically, the rotation angle of the face image relative to the group center can be determined based on the position relationship between the key points in the face image and the position relationship between the key points in the group center, for example, the rotation angle of the face image relative to the group center in the binocular direction is determined based on the distance between the eyes in the group center and the distance between the eyes in the face image. Of course, the rotation angle of the face image relative to the group center in the horizontal and vertical directions can be determined based on the position relationship between more key points.

[0121] In operation S702, the obtained rotation angle is used to mark the angle of the corresponding face image. Specifically, the rotation angle of each face image in the face image group relative to the group center can be obtained through the previous operation, and then the rotation angle information can be marked for each face image in the face image. The rotation angle can be used as a basis for screening training data in a specific application scenario. For example, in the face payment scenario, since the camera usually uses a surveillance camera, and the surveillance camera is usually at a certain angle to the front face of the payer, and the angle is basically within a rough range, if a face image group with face images of various angles is used as a training data set for the face recognition model, the face recognition model can recognize a larger angle range of face images, but, correspondingly, since there is no optimization for face images at a specific angle, the accuracy of face recognition at that specific angle cannot be further improved. In the present embodiment, since the rotation angle information of the face image is also marked in the face image group, face images with specific rotation angles can be selected from multiple face image groups as training data sets for the face recognition model. This can help improve the recognition accuracy of the face recognition model at a specific rotation angle, or train a face recognition model for a specific rotation angle according to a specific application scenario.

[0122] like Figure 8 As shown, a flowchart of a method for calculating a rotation angle according to an embodiment of the present disclosure is schematically shown. In this embodiment, for each face image, the following operations may be included:

[0123] In operation S801, a specified number of key points of the face image is obtained, and a specified number of key points of the group center are obtained. The key points include but are not limited to: one or more of the eye center point, the mouth corner point, the nose vertex, the cheekbone vertex or the eyebrow vertex. Fig. 9As shown, a schematic diagram of a face image and key points in a video frame according to an embodiment of the present disclosure is schematically shown, wherein the image in the box is a face image recognized from a video frame, and the rectangular area in the picture is represented as [x, y, w, h], (x, y) represents the position of the upper left corner of the picture relative to the face image, and (w, h) represents the width and height of the box; the bright spots in the face image are key points, which can be represented as kp_1, kp_2, ..., kp_N, wherein N represents the number of key points; kp_N is a two-dimensional data, representing the position coordinates relative to the upper left corner of the picture, and N key points are extracted for each face.

[0124] The key point detection technology can be a face key point detection technology based on SDM and / or a face key point detection technology based on MTCNN. In other words, face detection can be performed first, and then key point detection can be performed separately on the detected face detection frame, such as using SDM technology; face detection and key point recognition can also be performed simultaneously, such as using MTCNN technology.

[0125] For SDM technology, the following operations may be included: first, modeling is performed, the input of the model is a face image, and the output of the model is the key points in the face image. The model training process may include the following operations: inputting a face image with manually annotated information into the model, and adjusting the model parameters so that the feature points output by the model are close to the key points of the manually annotated face, wherein the manually annotated information is the manually annotated key points.

[0126] For the multi-task convolutional neural network (MTCNN) technology, three neural networks can be used to form the MTCNN, including: Proposal Network (P-Net), RefineNetwork (R-Net) and Output Network (O-Net). Among them, the network structure of P-Net mainly obtains the regression vector of the candidate window and bounding box of the face image. The bounding box is used for regression, the candidate window is calibrated, and then the highly overlapping candidate windows are merged through non-maximum suppression (NMS). The network structure of R-Net still removes false positive areas through bounding box regression and NMS, but the network structure of R-Net is different from that of P-Net, and there is an additional fully connected layer, so it will achieve a better effect of suppressing false positive areas. The network structure of O-Net has one more convolutional layer than the network structure of R-Net, so the processing result will be more refined. The role of O-Net is the same as that of R-Net, but O-Net supervises the face image more and also outputs 5 landmarks, i.e. key points, such as Fig.10 As shown, a schematic diagram of key points according to an embodiment of the present disclosure is schematically shown, wherein each face image has 5 key points. Of course, more or fewer key points can be set according to actual needs.

[0127] In operation S802, the positional relationship between the key points of the face image and the positional relationship between the key points in the center of the group are obtained. The above positional relationship includes but is not limited to: the coordinates of the key points relative to the upper left corner of the face image, the relative coordinates between the key points of the same type (such as the relative coordinates between the centers of two eyes), the distance between the key points, and any one or more of the above.

[0128] In operation S803, the rotation angle of the face image relative to the group center is determined based on the positional relationship between the key points of the face image and the positional relationship between the key points of the group center. Specifically, the rotation angle of the face image relative to the group center in the horizontal direction and / or the vertical direction is obtained by calculation.

[0129] like Fig.11 As shown, a flowchart of a method for processing a face image according to another embodiment of the present disclosure is schematically shown. In this embodiment, the method may also include the following operations:

[0130] In operation S111, for a face image group, the face images in the face image group are sorted according to the rotation angle and / or distance between the face image and the group center. The rotation angle is the rotation angle of each face image in the face image group relative to the group center, and the distance can be a cosine distance, that is, representing the similarity between each face image and the group center, specifically, it can be a cosine distance between the face features of the face image and the face features of the group center.

[0131] In operation S112, the sorted facial images are displayed. In this embodiment, it may be necessary to manually check the facial images in the facial image group to ensure that the facial images in each facial image group correspond to the same person. If there are many facial images in the facial image group and the rotation angle of some facial images relative to the group center is large, it is difficult to manually directly compare the facial images and the group center. If the facial images in the facial image group are sorted according to similarity and / or rotation angle, a gradual process can be shown. If it is difficult to determine whether the non-clustering is correct by directly comparing the group center and the facial image, the gradual process shown after sorting can be referred to to assist in determining whether the non-clustering is correct. This helps to improve the convenience of manual verification and is less likely to result in misjudgment or omission.

[0132] like Fig.12 As shown, a flowchart of a method for processing a face image according to another embodiment of the present disclosure is schematically shown. In this embodiment, the method may also include the following operations:

[0133] In operation S121, a face recognition model is trained using multiple face image groups. The face recognition model may be a newly constructed model; or it may be a model trained using an existing standard training data set, and then the trained model is retrained using multiple face image groups, that is, the trained model is fine-tuned using multiple face image groups, which can greatly improve the face recognition accuracy of the face recognition model in the prior art in the multi-angle face recognition scenario.

[0134] In a specific embodiment, a first face recognition model is pre-built, and after obtaining the plurality of face image groups, the face features of face images at different angles corresponding to the face image groups and the face image groups to which they belong are input into the first face recognition model for training to obtain model parameter values. The trained first face recognition model can be used to realize face recognition of face images at multiple angles.

[0135] In another specific embodiment, for the second face recognition model pre-trained with standard face data, after obtaining the plurality of face image groups, the face features of face images at different angles corresponding to the face image groups and the face image groups to which they belong are input into the second face recognition model for training to obtain adjusted model parameter values. The trained second face recognition model can be used to realize face recognition of face images at multiple angles with high recognition accuracy.

[0136] In another specific embodiment, a third face recognition model is pre-constructed, and after obtaining the plurality of face image groups and the rotation angle between the face image and the group center of the face image group to which it belongs, first, the face rotation angle of the application scenario is obtained; then, the model parameter values ​​of the third face recognition model are obtained using the face image with angle annotation information, the face image group to which the face image belongs, and the face rotation angle. It should be noted that the use of the face image with angle annotation information, the face image group to which the face image belongs, and the face rotation angle to obtain the model parameter values ​​of the third face recognition model can be implemented separately or in combination in one of the following two ways: the first way is to first select face images of specific angles from the face image group, and then input the face images with angle annotation information and the face image group to which the face images belong into the third face recognition model for training to obtain model parameters; the second way is to directly input the face image with angle annotation information, the face image group to which the face image belongs, and the face rotation angle into the third face recognition model for training to obtain model parameters.

[0137] In other specific embodiments, for the fourth face recognition model pre-trained with standard face data, after obtaining the multiple face image groups and the rotation angle between the face image and the group center of the face image group to which it belongs, the face rotation angle of the application scenario is first obtained; then, the face image with angle annotation information, the face image group to which the face image belongs, and the face rotation angle are used to obtain the adjusted model parameter values ​​of the fourth face recognition model.

[0138] Accordingly, the present disclosure also provides a face image processing device 1300 corresponding to the above-mentioned face image processing method, Fig.13 The block diagram schematically shows a face image processing device according to an embodiment of the present disclosure.

[0139] like Fig.13 As shown, the face image processing device includes: an acquisition module 131 and a clustering module 132.

[0140] The acquisition module 131 is used to acquire multiple facial images, where the multiple facial images include facial images of at least one person at different angles.

[0141] The clustering module 132 is used to cluster the multiple face images to obtain multiple face image groups.

[0142] also, Fig.14 A block diagram of a face image processing device according to another embodiment of the present disclosure is schematically shown, and the device may further include:

[0143] The correction module 141 is used to remove part of the facial images in at least one facial image group according to user input and / or preset rules, so that each facial image in each facial image group in the at least one facial image group corresponds to the same person.

[0144] Wherein, the acquisition module 141 includes:

[0145] The video acquisition unit is used to acquire video.

[0146] The video frame acquisition unit is used to acquire the video frames of the video frame by frame or at intervals of a set number of frames.

[0147] The face image acquisition unit is used to acquire the face image of the video frame.

[0148] In a specific embodiment, the face image acquisition unit includes:

[0149] The setting subunit is used to set the face detection frame.

[0150] The feature extraction subunit is used to extract the facial features of the portion of the video frame covered by the face detection frame.

[0151] The model training subunit is used to input the facial features into a pre-trained facial model to obtain a recognition result.

[0152] The image recognition unit is used to determine the face image according to the recognition result.

[0153] The clustering module 132 is specifically used to cluster the facial features of the facial images based on a K-means or hierarchical clustering method to obtain at least one facial image group and a facial image corresponding to each facial image group.

[0154] In another embodiment, Fig.15 A block diagram of a face image processing device according to another embodiment of the present disclosure is schematically shown, and the device may further include the following modules:

[0155] The group center acquisition module 151 connected to the clustering module 132 is used to respectively acquire the group center of at least one face image group.

[0156] The identification module 152 is used to use the cluster center as the identification of the corresponding face image group.

[0157] The group center acquisition module 151 includes:

[0158] The distance calculation unit is used to calculate the distance between each face image in each face image group and other face images in the face image group.

[0159] The center acquisition unit is used to take the face image with the smallest sum of distances from other face images in the face image group as the group center.

[0160] In yet another embodiment, Fig.16 A block diagram of a face image processing device according to another embodiment of the present disclosure is schematically shown, and the device may further include the following modules:

[0161] The rotation angle calculation module 161 connected to the group center acquisition module 151 is used to calculate the rotation angle of each face image in the face image group relative to the group center.

[0162] The angle marking module 162 is used to use the obtained rotation angle to perform angle marking on the corresponding face image.

[0163] Wherein, the rotation angle calculation module 161 includes:

[0164] The key point acquisition unit is used to acquire, for each face image, a specified number of key points of the face image, and acquire a specified number of key points of the group center.

[0165] The position relationship acquisition unit is used to acquire the position relationship between the key points of the face image and the position relationship between the key points of the group center.

[0166] The rotation angle calculation unit is used to determine the rotation angle of the face image relative to the group center according to the positional relationship between the key points of the face image and the positional relationship between the key points of the group center.

[0167] Specifically, the key point acquisition unit is specifically used to obtain the key points based on SDM and / or MTCNN facial key point detection.

[0168] In other embodiments, Fig.17 A block diagram of a face image processing device according to another embodiment of the present disclosure is schematically shown, and the device may further include the following modules:

[0169] The sorting module 171 is used to sort the facial images in the facial image group according to the rotation angle and / or distance between the facial images and the group center.

[0170] The display module 172 is used to display the sorted face images.

[0171] In yet another embodiment, the apparatus may further include the following modules: a face recognition model training module, configured to train a face recognition model using the multiple face image groups after obtaining the multiple face image groups.

[0172] In a specific embodiment, Fig.18 A block diagram of a face image processing device according to another embodiment of the present disclosure is schematically shown, and the device may further include the following modules:

[0173] The first face recognition model construction module 181 is used to pre-construct a first face recognition model.

[0174] The first face recognition model training module 182 is used to input the face features of face images at different angles corresponding to the face image groups and the face image groups to which they belong into the first face recognition model after obtaining the multiple face image groups, so as to obtain model parameter values.

[0175] In yet another specific embodiment, Fig.19 A block diagram of a face image processing device according to another embodiment of the present disclosure is schematically shown, and the device may further include the following modules:

[0176] The second face recognition model construction module 191 is used to pre-construct a second face recognition model;

[0177] The second face recognition model training module 192 is used to train the pre-built second face recognition model using standard face data to obtain a second face recognition model;

[0178] The second face recognition model adjustment module 193 is used to input the face features of face images at different angles corresponding to the face image groups and the face image groups to which they belong into the second face recognition model after obtaining the multiple face image groups, so as to obtain adjusted model parameter values.

[0179] In yet another specific embodiment, Fig. 20 A block diagram of a face image processing device according to another embodiment of the present disclosure is schematically shown, and the device may further include the following modules:

[0180] The third face recognition model construction module 201 is used to pre-construct a third face recognition model;

[0181] The rotation angle acquisition module 202 is used to acquire the face rotation angle of the application scenario;

[0182] The third face recognition model parameter acquisition module 203 is used to obtain the model parameter values ​​of the third face recognition model by using the face image with angle annotation information, the face image group to which the face image belongs, and the face rotation angle after obtaining the multiple face image groups and the rotation angle between the face image and the group center of the face image group to which it belongs.

[0183] In yet another specific embodiment, Fig.21 A block diagram of a face image processing device according to another embodiment of the present disclosure is schematically shown, and the device may further include the following modules:

[0184] The fourth face recognition model construction module 211 is used to pre-construct a fourth face recognition model;

[0185] The fourth face recognition model training module 212 is used to train the pre-built fourth face recognition model using standard face data to obtain a fourth face recognition model;

[0186] The rotation angle acquisition module 213 is used to acquire the face rotation angle of the application scenario;

[0187] The fourth face recognition model parameter acquisition module 214 is used to obtain the adjusted model parameter values ​​of the fourth face recognition model by using the face image with angle annotation information, the face image group to which the face image belongs, and the face rotation angle after obtaining the multiple face image groups and the rotation angle between the face image and the group center of the face image group to which it belongs.

[0188] Since the face recognition device corresponds to the face recognition method, the face recognition device will not be elaborated in detail, and the details may refer to the relevant parts of the face recognition method.

[0189] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0190] For example, any multiple of the acquisition module 131, the clustering module 132, the correction module 141, the group center acquisition module 151, the identification module 152, the sorting module 171, and the angle marking module 162 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the acquisition module 131, the clustering module 132, the correction module 141, the group center acquisition module 151, the identification module 152, the sorting module 171, and the angle marking module 162 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the acquisition module 131, clustering module 132, correction module 141, group center acquisition module 151, identification module 152, sorting module 171, and angle labeling module 162 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is executed.

[0191] Fig. 22 A block diagram of a computer system suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown. Fig. 22The computer system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0192] like Fig. 22 As shown, the computer system 2200 according to an embodiment of the present disclosure includes a processor 2201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 2202 or a program loaded from a storage part 1408 into a random access memory (RAM) 2203. The processor 2201 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 2201 may also include an onboard memory for caching purposes. The processor 2201 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0193] In RAM 2203, various programs and data required for the operation of system 2200 are stored, for example, for storing executable instructions. Processor 2201, ROM 2202 and RAM 2203 are connected to each other through bus 2204. Processor 2201 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 2202 and / or RAM 2203. It should be noted that the program can also be stored in one or more memories other than ROM 2202 and RAM 2203. Processor 2201 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories, for example, enabling the one or more processors to obtain multiple facial images, the multiple facial images including facial images of at least one person at different angles; clustering the multiple facial images to obtain multiple facial image groups, and also removing part of the facial images in at least one facial image group according to user input and / or preset rules, so that each facial image in each facial image group in the at least one facial image group corresponds to the same person.

[0194] According to an embodiment of the present disclosure, the system 2200 may further include an input / output (I / O) interface 2205, which is also connected to the bus 2204. The system 2200 may further include one or more of the following components connected to the I / O interface 2205: an input section 2206 including a keyboard, a mouse, etc.; an output section 2207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 2208 including a hard disk, etc.; and a communication section 2209 including a network interface card such as a LAN card, a modem, etc. The communication section 2209 performs communication processing via a network such as the Internet. A drive 2210 is also connected to the I / O interface 2205 as needed. A removable medium 2211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 2210 as needed, so that a computer program read therefrom is installed into the storage section 2208 as needed.

[0195] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 2209, and / or installed from the removable medium 2211. When the computer program is executed by the processor 2201, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0196] The present disclosure also provides a computer-readable medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed, various methods according to the embodiments of the present disclosure are implemented.

[0197] According to an embodiment of the present disclosure, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, optical cable, radio frequency signal, etc., or any suitable combination of the above.

[0198] For example, according to an embodiment of the present disclosure, the computer-readable medium may include the ROM 2202 and / or the RAM 2203 described above and / or one or more memories other than the ROM 2202 and the RAM 2203 .

[0199] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present disclosure can be combined or / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0200] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A face image processing method, include: Acquire a plurality of facial images, wherein the plurality of facial images include facial images of at least one person at different angles; Clustering the plurality of face images to obtain a plurality of face image groups, wherein each face image group includes face images of the same person at different angles; Removing some facial images from at least one facial image group according to user input and / or preset rules, so that each facial image in each facial image group in the at least one facial image group corresponds to the same person, wherein the some facial images in the at least one facial image group do not belong to the same person as other facial images in the at least one facial image group; For each face image group in the plurality of face image groups, respectively calculating a rotation angle of each face image in the face image group relative to a group center, wherein the group center is used to represent a face image having a minimum sum of distances from other face images in the face image group, wherein the distances include a cosine distance between face features of two face images; Using the obtained rotation angle, the corresponding face image is angle-marked to obtain a face image with angle-marked information; A face recognition model is trained based on the facial features of the face images at different angles corresponding to each of the multiple face image groups, the face images at different angles with angle annotation information corresponding to each of the multiple face image groups, and the face image groups to which the face images at different angles belong.

2. The method according to claim 1, in, The acquiring of multiple face images comprises: Get the video; Acquire video frames of the video frame by frame or at intervals of a set number of frames; Obtain a face image of the video frame.

3. The method according to claim 1, further comprising, after obtaining the plurality of face image groups: respectively obtaining a group center of at least one face image group; The group center is used as the identifier of the corresponding face image group.

4. The method according to claim 3, in, The obtaining of the group center of at least one face image group comprises, for each face image group in the at least one face image group: Calculate the distance between each face image and other face images in the face image group respectively; The face image with the smallest sum of distances from other face images in the face image group is taken as the group center.

5. The method according to claim 1, in, The calculating of the rotation angle of each face image in the face image group relative to the group center comprises, for each face image: Obtaining a specified number of key points of the face image, and obtaining a specified number of key points of the group center; Obtaining the positional relationship between the key points of the face image, and obtaining the positional relationship between the key points of the group center; The rotation angle of the face image relative to the group center is determined based on the positional relationship between the key points of the face image and the positional relationship between the key points of the group center.

6. The method according to claim 1, further comprising: include: For the facial image group, sorting the facial images in the facial image group according to the rotation angle and / or distance between the facial image and the center of the group; as well as Display the sorted face images.

7. A facial image processing device, include: An acquisition module, used to acquire a plurality of face images, wherein the plurality of face images include face images of at least one person at different angles; A clustering module, used for clustering the plurality of face images to obtain a plurality of face image groups, wherein each face image group includes face images of the same person at different angles; a correction module, configured to remove part of the face images in at least one face image group according to user input and / or preset rules, so that each face image in each face image group in the at least one face image group corresponds to the same person, wherein the part of the face images in the at least one face image group and other face images in the at least one face image group do not belong to the same person; a rotation angle calculation module, configured to calculate, for each face image group in the plurality of face image groups, a rotation angle of each face image in the face image group relative to a group center, wherein the group center is used to represent a face image having a minimum sum of distances from other face images in the face image group, wherein the distances include a cosine distance between face features of two face images; An angle marking module is used to use the obtained rotation angle to mark the corresponding face image with angle marking information to obtain the face image; A face recognition model training module is used to train a face recognition model based on the facial features of the face images at different angles corresponding to each of the multiple face image groups, the face images at different angles with angle annotation information corresponding to each of the multiple face image groups, and the face image groups to which the face images at different angles belong.

8. The device according to claim 7, in, The acquisition module comprises: A video acquisition unit, used for acquiring video; A video frame acquisition unit, used for acquiring video frames of the video frame by frame or at intervals of a set number of frames; The face image acquisition unit is used to acquire the face image of the video frame.

9. The device according to claim 7, further comprising: include: A group center acquisition module connected to the clustering module, used to respectively acquire the group center of at least one face image group; The identification module is used to use the group center as the identification of the corresponding face image group.

10. The device according to claim 9, in, The group center acquisition module includes: a distance calculation unit, configured to calculate, for each face image group in the at least one face image group, the distance between each face image in the face image group and other face images; The center acquisition unit is used to take the face image with the smallest sum of distances from other face images in the face image group as the group center.

11. The device according to claim 7, in, The rotation angle calculation module comprises: A key point acquisition unit, used for acquiring, for each face image, a specified number of key points of the face image, and acquiring a specified number of key points of the group center; A position relationship acquisition unit, used to acquire the position relationship between the key points of the face image, and to acquire the position relationship between the key points of the group center; The rotation angle calculation unit is used to determine the rotation angle of the face image relative to the group center according to the position relationship between the key points of the face image and the position relationship between the key points of the group center.

12. The device according to claim 7, further comprising: include: a sorting module, configured to sort the facial images in the facial image group according to a rotation angle and / or a distance between the facial image and the center of the group; as well as The display module is used to display the sorted face images.

13. A device for processing facial images, include: one or more processors; A storage device for storing executable instructions, wherein when the executable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

14. A computer-readable medium having executable instructions stored thereon, wherein the instructions, when executed by a processor, implement the method according to any one of claims 1 to 6.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.