Method, device and electronic device for handling illegal users

Abstract

The embodiments of the present invention provide a method, device and electronic device for processing illegal users, which relate to the field of face recognition technology. The method includes: obtaining a first picture of a live picture stream; performing preset processing on the first picture to obtain a first facial feature of the first picture; calculating the similarity between the first facial feature and each facial feature in a first database of illegal users, and saving the first facial feature to the first database; determining a processing strategy for the first facial feature based on the similarity between the first facial feature and each facial feature in the first database. The above scheme can not only update the first database in real time to ensure the accuracy of the data, but also process habitual illegal users of live broadcasts more accurately to improve efficiency.

Description

Technical Field

[0001] The present invention relates to the field of face recognition technology, and in particular to a method, device and electronic device for handling illegal users. Background Art

[0002] With the development of the Internet information age, live streaming has gradually become a mainstream way for the public to share life and show creativity, which can enrich life. But at the same time, live streaming is also a double-edged sword like the Internet. Some live streaming violators often use multiple accounts to broadcast in order to expand the scope of live streaming, becoming habitual violators, which has a bad impact. For habitual live streaming violators, the platform needs to spend a lot of manpower to review them. Summary of the invention

[0003] The present invention provides a method, device and electronic device for processing illegal users, so as to solve the problem that the prior art needs to consume a lot of manpower to review the users who often violate the rules in live broadcasts to a certain extent.

[0004] In a first aspect of the present invention, a method for handling illegal users is provided, the method comprising:

[0005] Get the first picture of the live picture stream;

[0006] Performing a preset process on the first image to obtain a first facial feature of the first image;

[0007] Calculating the similarity between the first facial feature and each facial feature in the first database of illegal users;

[0008] A processing strategy for the first facial feature is determined according to a similarity between the first facial feature and each facial feature in the first database.

[0009] In a second aspect of the present invention, a device for processing illegal users is provided, the device comprising:

[0010] A first acquisition module, used to acquire a first picture of a live picture stream;

[0011] A first processing module, used for performing a preset process on the first image to obtain a first facial feature of the first image;

[0012] A second processing module, used for calculating the similarity between the first facial feature and each facial feature in the first database of illegal users;

[0013] The first determination module is used to determine a processing strategy for the first facial feature according to a similarity between the first facial feature and each facial feature in the first database.

[0014] In a third aspect of the present invention, there is also provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0015] Memory, used to store computer programs;

[0016] The processor is used to implement the steps in the above-mentioned method for handling illegal users when executing the program stored in the memory.

[0017] In a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for handling illegal users as described above is implemented.

[0018] In a fifth aspect of the embodiments of the present invention, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute the above-mentioned method for handling illegal users.

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] In an embodiment of the present invention, a first image of a live broadcast image stream is obtained and the first image is subjected to preset processing to obtain a first facial feature of the first image, and a similarity between the first facial feature and each facial feature in a first database of violating users is calculated. Based on the similarity between the first facial feature and each facial feature in the first database, a processing strategy for the first facial feature is determined, thereby more accurately processing users who frequently violate live broadcast rules and improving efficiency.

[0021] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for describing the embodiments are briefly introduced below.

[0023] Figure 1 A flow chart of a method for handling illegal users provided by an embodiment of the present invention;

[0024] Figure 2 A specific flow chart of the method for handling illegal users provided by an embodiment of the present invention;

[0025] Figure 3 A structural block diagram of a device for processing illegal users provided by an embodiment of the present invention;

[0026] Figure 4 This is a structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] At present, face recognition is one of the main research topics in the field of computer vision and biometrics. According to the different feature extraction methods, the existing methods can be divided into the following two categories: face recognition methods based on manual features and face recognition methods based on deep features.

[0030] Among them, the former relies on the combination of artificially designed features (such as texture, etc.) and machine learning techniques (such as principal component analysis, linear discriminant analysis or support vector machine, etc.). However, face images vary greatly, such as head posture, facial expression, age, occlusion, lighting, etc. It is difficult to manually design features that are robust to different changes in an unconstrained environment, which makes dedicated methods for each type of change, such as methods for different ages, different postures or different lighting conditions, have great limitations.

[0031] The face recognition method based on deep features is based on the Deep Convolution Neural Network (DCNN), combined with the back gradient propagation algorithm, and uses large-scale data sets to train DCNN, so as to learn the best features to characterize these data. These large-scale data sets contain various changes, so the DCNN trained with them can learn more robust features in face images. However, there is no automated method to distinguish habitual violators and ban them for the behavior of live broadcast violators.

[0032] Therefore, the embodiments of the present invention provide a method, device and electronic device for handling illegal users, which extract and compare the second facial features of illegal users, and automatically mine live broadcast users who are frequent illegal users; and add the second facial features of the mined frequent illegal users to the first database, and for the live broadcast picture stream, extract the first facial features and compare them with the first database, so as to achieve the purpose of cracking down on users who are frequent illegal users in live broadcasts.

[0033] Specifically, Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a method for handling illegal users, and the method specifically includes:

[0034] Step 101, obtaining the first picture of the live picture stream.

[0035] Specifically, during the live broadcast process of the user, a live broadcast picture stream of the user during the live broadcast process is obtained, and the first picture is one of the pictures in the live broadcast picture stream.

[0036] Step 102: Perform a preset process on the first image to obtain a first facial feature of the first image.

[0037] Specifically, the first image in the live image stream is subjected to preset processing, that is, the first facial feature of the first image after the preset processing can be extracted. The preset processing is a combination of image processing, portrait processing, facial feature processing, and other processing methods for the first image. By performing the preset processing on the first image, the first facial feature obtained can be a high-quality facial feature, thereby improving the accuracy of face recognition.

[0038] Step 103: Calculate the similarity between the first facial feature and each facial feature in the first database of illegal users.

[0039] Specifically, the first facial feature is compared with each facial feature in the first database to calculate the similarity between the first facial feature and each facial feature in the first database, thereby determining whether the facial feature in the first database is similar to the first facial feature and the degree of similarity.

[0040] The facial features in the first database are facial features of users who frequently violate the rules (eg, users who violate the rules multiple times), and each facial feature in the first database is recorded with a corresponding user identity identification number (IDentity, ID).

[0041] Step 104: Determine a processing strategy for the first facial feature according to the similarity between the first facial feature and each facial feature in the first database.

[0042] Specifically, based on the similarity between the first facial feature and each facial feature in the first database, it can be known whether the facial feature in the first database is similar to the first facial feature, and the degree of similarity, so that the processing strategy for the first facial feature can be determined, that is, whether to ban the live picture stream corresponding to the first facial feature, etc., to improve the blocking accuracy.

[0043] In the above embodiment of the present invention, a first picture of a live picture stream is obtained, and the first picture is subjected to preset processing to obtain a first facial feature of the first picture, and a similarity between the first facial feature and each facial feature in a first database of violating users is calculated; based on the similarity between the first facial feature and each facial feature in the first database, a processing strategy for the first facial feature is determined, so that habitual violating users of live broadcasts can be processed more accurately to improve efficiency.

[0044] Optionally, before calculating the similarity between the first facial feature and each facial feature in the first database of illegal users in step 103, the method may further include:

[0045] like Figure 2 As shown, step A1, obtain the illegal user's picture.

[0046] Specifically, firstly, the illegal user picture is obtained, and the method of obtaining is not limited. Among them, the illegal user picture can be a picture of a user who has violated the rules on a live broadcast platform.

[0047] Step A2: performing preset processing on the illegal user image to obtain a second facial feature of the illegal user image.

[0048] Specifically, the image of the offending user is subjected to a preset processing, that is, the second facial features of the image of the offending user after the preset processing can be extracted. The preset processing is a combination of image processing, portrait processing, facial feature processing, etc. for the image of the offending user. By performing the preset processing on the image of the offending user, the obtained second facial features can be high-quality facial features, thereby improving the accuracy of face recognition.

[0049] like Figure 2 As shown, in step A3, the similarity between the second facial feature and each facial feature in the second database of the violating user is calculated, and the second facial feature is saved in the second database.

[0050] Specifically, the second facial feature is compared with each facial feature in the second database, so as to calculate the similarity between the second facial feature and each facial feature in the second database, thereby knowing whether the facial feature in the second database is similar to the second facial feature, and the degree of similarity. The facial features in the second database are facial features of illegal users, and the facial features in the second database (i.e., the database for mining users with habitual illegal behavior) can be updated based on the second facial features, thereby keeping the second database updated and improving the accuracy of facial recognition. In addition, each facial feature in the second database is recorded with a corresponding user ID.

[0051] like Figure 2 As shown, step A4, based on the similarity between the second facial feature and each facial feature in the second database, determines whether the second facial feature is the first target facial feature, that is, the second facial feature is compared with each facial feature in the second database for similarity to determine whether the second facial feature is the first target facial feature.

[0052] Specifically, based on the similarity between the second facial feature and each facial feature in the second database, it can be known whether the facial feature in the second database is similar to the second facial feature, and the degree of similarity, so that it can be determined whether the second facial feature is the first target facial feature, that is, whether the second facial feature is a habitual violator, which can improve the accuracy of habitual violators.

[0053] Specifically, for the second facial feature, it is compared with each facial feature in the second database, and the cosine similarity sim (sim∈[0,1]) between the two is calculated; when the second database is large in size, locality sensitive hashing (LSH), inverted file system product quantization (IVFPQ), hierarchical navigable small world (HNSW) and other methods can be used to further improve the retrieval speed of similar features.

[0054] Step A5: If the second facial feature is the first target facial feature (ie, the facial feature of a user who frequently violates the rules), the first target facial feature is saved in the first database.

[0055] Specifically, if the second facial feature is the first target facial feature, that is, the second facial feature is the facial feature of a habitual violator, then the first target facial feature needs to be saved as the facial feature of the habitual violator in the first database to update the database of habitual violators, thereby ensuring real-time updating of the first database.

[0056] Optionally, the step A4 determines whether the second facial feature is the first target facial feature according to the similarity between the second facial feature and each facial feature in the second database, and specifically includes:

[0057] According to the similarity between the second facial feature and each facial feature in the second database, obtaining a second target facial feature in the second database whose similarity with the second facial feature is greater than a first similarity threshold;

[0058] If the first number of the second target facial features is greater than the first value, the two facial features are determined to be the first target facial features.

[0059] Specifically, according to the similarity between the second facial feature and each facial feature in the second database, the second target facial feature in the second database whose similarity with the second facial feature is greater than the first similarity threshold can be obtained, that is, the second target facial feature in the second database with a higher similarity to the second facial feature can be obtained; in other words, if the similarity sim between the input second facial feature and the second target facial feature in the first database is greater than the first similarity threshold sim thd (sim thd ∈[0, 1]), the second facial feature and the second target facial feature are considered to belong to the same face.

[0060] If the first number of the second target facial features is less than or equal to the first value, the judgment of the second facial features is stopped, that is, the second facial features are ignored; if the first number of the second target facial features is greater than the first value, the two facial features are determined to be the first target facial features, that is, whether the second facial features are habitual violators is determined, which can improve the accuracy of habitual violators.

[0061] It should be noted that the first similarity threshold is a similarity limit for judging whether the similarity between the facial feature in the second database and the second facial feature is relatively high, which can be set as needed and is not specifically limited here. Among them, the first value can be used to judge whether the number of the second target facial features meets the condition for judging the second facial features as a habitual offender, that is, the first number of the second target facial features is greater than the first value (such as: 2), that is, the input second facial features hit the first number of user IDs, and the second facial features can be judged as a habitual offender. Otherwise, the second facial features can be judged as an ordinary offender. The first value can be set as needed and is not specifically limited here.

[0062] Optional, such as Figure 2 As shown, the preset processing may include:

[0063] Step B1, rotating the target image to obtain a rotated target image; the target image is the first image or the illegal user image.

[0064] Specifically, Figure 2 As shown, the target image is obtained through step 101 (such as obtaining a live image stream), and the rotation processing is the face rotation detection processing, that is, the target image is used as the input image to find the position of the face in the target image. If no face is detected, the rotation processing is stopped; if a face is detected, the coordinates of the face frame (i.e., the face boundary frame) containing each face and the coordinates of the face key points (such as eyes, nose, mouth, etc.) can be detected, and the target image is rotated according to the face frame coordinates and the face key point coordinates, so as to obtain the rotated target image; wherein the coordinates of the face frame are:

[0065] [x0, y0, x1, y1]

[0066] Among them, x0 and y0 are the coordinates of the upper left corner of the face frame;

[0067] x1 and y1 are the coordinates of the lower right corner of the face frame.

[0068] Among them, the target image can be the first image or the illegal user image, that is, the first image can be rotated to obtain the rotated first image; the illegal user image can also be rotated to obtain the rotated illegal user image.

[0069] like Figure 2 As shown, step B2 is to determine whether the face in the rotated target image meets the first preset condition, that is, to determine the completeness of the target image.

[0070] Specifically, it is determined whether the face in the rotated target image meets the first preset condition; if so, the process proceeds to step B3; if not, the target image is filtered out, that is, no further processing is performed on the target image.

[0071] like Figure 2 As shown, in step B3, if the face in the rotated target image meets the first preset condition, face alignment processing is performed on the rotated target image to obtain a target image with a positive face image.

[0072] Specifically, if the face in the rotated target image meets the first preset condition, it is necessary to perform face alignment on the rotated target image to obtain a target image with a positive face image, that is, to adjust the face posture to face forward, so as to achieve frontalization of the face. The features extracted from the aligned face are more robust.

[0073] Step B4, extracting facial features of the forward facial image to obtain third target facial features of the target image;

[0074] Among them, when the target picture is the first picture, the third target facial feature is the first facial feature; when the target picture is the illegal user picture, the third target facial feature is the second facial feature.

[0075] Specifically, facial features are extracted from the target image with a positive face image obtained after the face alignment process to obtain a third target facial feature related to the target image. If the target image is the first image, the third target facial feature obtained is the first facial feature related to the first image; if the target image is a picture of an illegal user, the third target facial feature obtained is the second facial feature related to the picture of the illegal user.

[0076] Among them, facial feature extraction is usually based on a convolutional neural network, such as a residual network (ResidualNetwork, ResNet), plus a loss function (such as the first loss function tripletloss, the second loss function cosinface, the third loss function arcface, etc.), and is trained using a large-scale data set with classification as the goal. The trained neural network can extract stable facial features, and the feature dimension can be 512 dimensions.

[0077] Optionally, the step B1 rotates the target image to obtain a rotated target image, which specifically includes:

[0078] Performing face detection on the target image to obtain the face frame confidence of the target image;

[0079] If the face frame confidence is less than or equal to the confidence threshold, the target image is rotated to obtain a rotated target image, and the face frame confidence of the rotated target image is greater than the confidence threshold.

[0080] At present, the commonly used face detection algorithms in the prior art are generally based on large-scale training data and can cope with changes in lighting, blur, angle, etc. In addition, in actual live broadcast scenarios, when the user rotates the phone or the user's body posture changes, the face direction in the live broadcast image will change accordingly, thereby affecting the detection accuracy; therefore, the embodiment of the present invention adopts a rotation detection strategy to cope with such changes in face direction, which can be specifically as follows:

[0081] First, perform face detection on the original input image (i.e., the target image) to obtain the face frame confidence of the target image; if the face frame confidence conf is greater than the confidence threshold conf thd , the face in the target image is considered to be positive, and face detection ends; if the face frame confidence conf is less than or equal to the confidence threshold conf thd , rotate the target image by a first angle (e.g. 90 degrees clockwise) and then perform face detection. If the face frame confidence conf is still less than or equal to the confidence threshold conf thd , rotate the target image by the first angle and then perform face detection, and so on, until the face frame confidence conf of the rotated target image is greater than the confidence threshold conf thd End face detection, that is, consider the face in the target image to be positive.

[0082] If the face box confidence conf is always less than or equal to the confidence threshold conf thd , it is considered that the target image does not contain a face, the face detection is terminated, and no further processing is performed on the target image.

[0083] Optionally, the first preset condition may include:

[0084] The horizontal coordinate of the nose in the rotated target image is greater than or equal to the horizontal coordinate of the left eye, and the horizontal coordinate of the nose is less than or equal to the horizontal coordinate of the right eye; and

[0085] The minimum value of the horizontal coordinate of the face key point in the rotated target image is greater than or equal to zero, and the maximum value of the horizontal coordinate of the key point is less than or equal to the width value of the rotated target image, and the minimum value of the vertical coordinate of the key point is greater than or equal to zero, and the maximum value of the vertical coordinate of the key point is less than or equal to the height value of the rotated target image.

[0086] Specifically, since the face changes dynamically in the live broadcast scene, when the side face or incomplete face is detected and the facial features are further extracted and stored, it is easy to cause a lot of misjudgments. Therefore, we need to judge the completeness of the face after face detection. Specifically, the coordinates of the key points of the face output by rotation processing are used to judge the completeness of the face. The coordinates of the key points of the face are:

[0087] [[x0,y0],[x1,y1],[x2,y2],[x3,y3],[x4,y4]]

[0088] Among them, the subscripts 0-4 in the above facial key point coordinates represent the coordinates of the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth, that is, [x0, y0] is the left eye coordinate, [x1, y1] is the right eye coordinate, [x2, y2] is the nose coordinate, [x3, y3] is the left corner of the mouth coordinate, and [x4, y4] is the right corner of the mouth coordinate. First, for the side face, use the following first formula to filter:

[0089] x0≤x2≤x1

[0090] That is, if the first formula is satisfied, the nose of the target image is considered to be between the left and right eyes, that is, it is not a profile face; if the first formula is not satisfied, the face of the target image is considered to be a profile face and needs to be filtered without further processing; for an incomplete face, the following second formula can be used for filtering:

[0091]

[0092] Among them, w is the width value of the target image;

[0093] h is the height of the target image;

[0094] That is, if the above second formula is satisfied, the target image is considered to be a complete face, that is, the facial key points of the complete face should be located in the target image; if the above second formula is not satisfied, the face of the target image is considered to be an incomplete face and needs to be filtered without subsequent processing.

[0095] Optionally, the step B3 performs face alignment processing on the rotated target image to obtain a target image with a positive face image, including:

[0096] The coordinates of the facial key points in the rotated target image are affine transformed with the preset key point coordinates to obtain a target image with a positive facial image.

[0097] Specifically, the face alignment process can be to use a set of reference points located at fixed positions in the target image to scale and crop the face image to achieve the purpose of face alignment. This process can use a feature point detector to find a set of face feature points (i.e., body key point coordinates), and perform an affine transformation on the face key point coordinates in the rotated target image with the preset key point coordinates to obtain a target image with a positive face image after face alignment.

[0098] Optionally, the step 104 determines a processing strategy for the first facial feature according to a similarity between the first facial feature and each facial feature in the first database, including:

[0099] like Figure 2 As shown, step C1 is to obtain a fourth target facial feature in the first database whose similarity with the first facial feature is greater than a second similarity threshold, that is, to obtain the fourth target facial feature through similarity comparison.

[0100] Specifically, based on the similarity between the first facial feature and each facial feature in the first database, it can be known whether the facial feature in the first database is similar to the first facial feature, and the degree of similarity, thereby obtaining the fourth target facial feature in the first database. The fourth target facial feature is a facial feature in the first database whose similarity to the first facial feature is greater than a second similarity threshold.

[0101] It should be noted that the second similarity threshold is a similarity limit for determining whether the similarity between the facial features in the first database and the first facial features is high, and can be set as needed, and is not specifically limited here. The first similarity threshold and the second similarity threshold can be the same value or different values, and are not specifically limited here.

[0102] Step C2: determining the target similarity according to the second number of the fourth target facial features.

[0103] Step C3: determining a processing strategy for the first facial feature according to the target similarity.

[0104] Specifically, in the above steps C2 and C3, the target similarity can be determined according to the second number of fourth target facial features, so that different processing strategies can be used to process the first facial features, that is, different numbers of fourth target facial features can have different processing strategies.

[0105] Optionally, the step C2 determines the target similarity according to the second number of the fourth target facial features, including:

[0106] If the second number is one, determining the product of the similarity between the fourth target facial feature and the first facial feature and the single image hit suppression coefficient as the target similarity;

[0107] If the second number is multiple, then determine the similarity between the fifth target facial feature and the first facial feature as the target similarity, and the fifth target facial feature is the facial feature with the greatest similarity to the first facial feature among the multiple fourth target facial features.

[0108] like Figure 2 As shown, in step 201, the accuracy of the ban can be improved by the following single-image hit suppression strategy. Specifically, if the number of the fourth target facial feature is 1, the fourth target facial feature is downgraded, that is, the target similarity sim i is the product of the similarity between the fourth target facial feature and the first facial feature and the single image hit suppression coefficient; if the number of the fourth target facial feature is greater than 1, the target similarity sim i The maximum value is taken, that is, the maximum similarity between the plurality of fourth target facial features and the first facial feature. Specifically, it can be calculated by the following third formula:

[0109]

[0110] Among them, sim i is the target similarity;

[0111] α is the single-image hit suppression coefficient;

[0112] n is the second number.

[0113] Optional, such as Figure 2 As shown, the step C3 determines the processing strategy of the first facial feature according to the target similarity, including:

[0114] Step 202: If the target similarity is within a first preset range, the live picture stream corresponding to the first facial feature is blocked.

[0115] Specifically, if a high threshold is set and low threshold When the following formula is satisfied, the target similarity sim is determined i In the first preset range:

[0116]

[0117] At this time, the live picture stream corresponding to the first facial feature is automatically blocked.

[0118] Optionally, the step C3 of determining a processing strategy for the first facial feature according to the target similarity further includes:

[0119] If the target similarity is within the second preset range, obtaining a review result of whether the first facial feature and the fourth target facial feature are the same face, the review result including: a correct hit result of the same face and an incorrect hit result of a different face;

[0120] When the review result is a correct hit result, the live image stream corresponding to the first facial feature is blocked.

[0121] Specifically, when the following formula is satisfied, the target similarity sim is determined i In the second preset range:

[0122]

[0123] At this time, it is necessary to push the live image stream corresponding to the first facial feature to the review department (such as manual review) to obtain the review result of whether the reviewed first facial feature and the fourth target facial feature are the same face. It can be reviewed as a correct hit result of the same face, or it can be reviewed as an incorrect hit result of not being the same face, so as to avoid erroneous bans that will cause a very bad experience for users.

[0124] And, if the target similarity sim i Satisfies the following formula:

[0125]

[0126] At this time, the live picture stream can be ignored and no blocking is performed.

[0127] Optionally, after the live broadcast image stream corresponding to the first facial feature is blocked, the method further includes:

[0128] Update the first database according to the live picture stream corresponding to the first facial feature.

[0129] Specifically, in step 202, if the target similarity is within the first preset range, the live picture stream corresponding to the first facial feature is blocked, and the first facial feature is added to the first database, thereby maintaining the real-time update of the first database and improving the accuracy of face recognition. If the target similarity is within the second preset range, the audit result of whether the first facial feature and the fourth target facial feature are the same face is obtained. If the audit result is a correct hit result, the live picture stream corresponding to the first facial feature is blocked, and the first facial feature is added to the first database, thereby maintaining the real-time update of the first database and improving the accuracy of face recognition.

[0130] Optionally, after obtaining the review result of whether the first facial feature and the fourth target facial feature are the same face, the method further includes:

[0131] Obtaining the number of audits of the fourth target facial feature;

[0132] Obtaining a hit rate of the fourth target facial feature according to the number of correct hits and the number of incorrect hits in the historical review results of the fourth target facial feature;

[0133] If the number of audits is greater than or equal to a first threshold and the hit rate is less than or equal to a second threshold, the fourth target facial feature is deleted from the first database.

[0134] Specifically, after obtaining the review result, the number of reviews of the fourth target facial feature is obtained. Based on the review result and the historical review results, the number of correct hits and the number of incorrect hits of the fourth target facial feature can be known, thereby knowing the hit rate of the fourth target facial feature; if the number of reviews is greater than or equal to the first threshold and the hit rate is less than or equal to the second threshold, the fourth target facial feature will be deleted from the first database, so that the facial features in the first database can be filtered to ensure the accuracy of the ban.

[0135] Specifically, since the faces in the live picture stream change in real time, there are cases where blurred faces are added to the first database (i.e., the database for banning users who frequently violate regulations), which can easily lead to misjudgment. In order to further standardize the first database and ensure the accuracy of the ban, the facial features in the first database can be automatically cleaned up. Among them, for the fourth target facial feature in the first database, if its similarity with the live picture stream is sim iIf the value is between the high threshold and the low threshold, it means that the confidence level of whether the two facial features (the fourth target facial feature and the first facial feature) belong to the same person is not high enough, so they are pushed to manual review. The manual review result can be used to automatically clean up the fourth target facial feature. Set the number of reviews of the fourth target facial feature count i , after manual review, the number of correct hits is The number of false hits is in, Then the hit rate precision of the fourth target face feature is i for:

[0136]

[0137] When the fourth target face feature review count i is greater than or equal to the first threshold, and the hit rate precision i If the fourth target facial feature is less than or equal to the second threshold, the fourth target facial feature is deleted from the first database to further ensure the blocking accuracy.

[0138] To summarize, in the mining stage, the embodiments of the present invention can ensure the detection rate of faces through rotation processing, and use the first preset condition and face alignment processing to ensure that the detected faces are high-quality faces, and automatically mine live broadcast users who frequently violate the rules by extracting the facial features of the positive face image; and the facial features of the mined users who frequently violate the rules are added to the first database; in the banning stage, for the live broadcast picture stream, the single image hit suppression strategy is used to improve the banning accuracy; the extracted facial features are compared with the facial features in the first database, so as to achieve the purpose of cracking down on live broadcast users who frequently violate the rules; the banning accuracy is further guaranteed by counting the facial feature hit rate of the first database and performing automatic cleaning.

[0139] like Figure 3 As shown, an embodiment of the present invention provides a device 300 for processing illegal users, the device comprising:

[0140] A first acquisition module 301 is used to acquire a first picture of a live picture stream;

[0141] A first processing module 302, configured to perform a preset process on the first image to obtain a first facial feature of the first image;

[0142] The second processing module 303 is used to calculate the similarity between the first facial feature and each facial feature in the first database of illegal users;

[0143] The first determination module 304 is used to determine a processing strategy for the first facial feature according to a similarity between the first facial feature and each facial feature in the first database.

[0144] In the above embodiment of the present invention, a first picture of a live picture stream is obtained, and the first picture is subjected to preset processing to obtain a first facial feature of the first picture, and a similarity between the first facial feature and each facial feature in a first database of violating users is calculated; based on the similarity between the first facial feature and each facial feature in the first database, a processing strategy for the first facial feature is determined, so that habitual violating users of live broadcasts can be processed more accurately to improve efficiency.

[0145] Optionally, before calculating the similarity between the first facial feature and each facial feature in the first database of illegal users, the device further includes:

[0146] The second acquisition module is used to obtain the illegal user's pictures;

[0147] A third processing module is used to perform a preset process on the illegal user picture to obtain a second facial feature of the illegal user picture;

[0148] A fourth processing module, configured to calculate the similarity between the second facial feature and each facial feature in the second database of the violating user, and save the second facial feature in the second database;

[0149] A first judging module, configured to judge whether the second facial feature is a first target facial feature according to a similarity between the second facial feature and each facial feature in the second database;

[0150] The first saving module is used to save the second target facial feature into the first database if the second facial feature is the first target facial feature.

[0151] Optionally, the first judgment module includes:

[0152] A first acquisition unit is used to acquire, according to the similarity between the second facial feature and each facial feature in the second database, a second target facial feature in the second database whose similarity to the second facial feature is greater than a first similarity threshold;

[0153] A determination unit is configured to determine that the second target facial feature is a first target facial feature if the first quantity of the second target facial feature is greater than a first value.

[0154] Optionally, the preset processing includes:

[0155] Rotate the target image to obtain a rotated target image;

[0156] Determining whether the face in the rotated target image meets a first preset condition;

[0157] If the face in the rotated target image meets the first preset condition, performing face alignment processing on the rotated target image to obtain a target image with a positive face image;

[0158] Extracting facial features of the forward facial image to obtain third target facial features of the target image;

[0159] Wherein, the target picture is the first picture or the illegal user picture; when the target picture is the first picture, the third target facial feature is the first facial feature;

[0160] When the target image is the illegal user image, the third target facial feature is the second facial feature.

[0161] Optionally, rotating the target image to obtain a rotated target image includes:

[0162] Performing face detection on the target image to obtain the face frame confidence of the target image;

[0163] If the face frame confidence is less than or equal to the confidence threshold, the target image is rotated to obtain a rotated target image, and the face frame confidence of the rotated target image is greater than the confidence threshold.

[0164] Optionally, the first preset condition includes:

[0165] The horizontal coordinate of the nose in the rotated target image is greater than or equal to the horizontal coordinate of the left eye, and the horizontal coordinate of the nose is less than or equal to the horizontal coordinate of the right eye; and

[0166] The minimum value of the horizontal coordinate of the face key point in the rotated target image is greater than or equal to zero, and the maximum value of the horizontal coordinate of the key point is less than or equal to the width value of the rotated target image, and the minimum value of the vertical coordinate of the key point is greater than or equal to zero, and the maximum value of the vertical coordinate of the key point is less than or equal to the height value of the rotated target image.

[0167] Optionally, performing face alignment processing on the rotated target image to obtain a target image having a positive face image includes:

[0168] The coordinates of the facial key points in the rotated target image are affine transformed with the preset key point coordinates to obtain a target image with a positive facial image.

[0169] Optionally, the first determining module 304 includes:

[0170] A second acquisition unit, configured to acquire a fourth target facial feature in the first database whose similarity to the first facial feature is greater than a second similarity threshold;

[0171] A first determining unit, configured to determine a target similarity according to a second number of the fourth target facial features;

[0172] The second determining unit is used to determine a processing strategy for the first facial feature according to the target similarity.

[0173] Optionally, the first determining unit includes:

[0174] If the second number is one, determining the product of the similarity between the fourth target facial feature and the first facial feature and the single image hit suppression coefficient as the target similarity;

[0175] If the second number is multiple, the similarity between the fifth target facial feature and the first facial feature is determined as the target similarity, and the fifth target facial feature is the facial feature with the greatest similarity to the first facial feature among the multiple fourth target facial features.

[0176] Optionally, the second determining unit includes:

[0177] If the target similarity is within a first preset range, the live picture stream corresponding to the first facial feature is blocked.

[0178] Optionally, the second determining unit further includes:

[0179] If the target similarity is within the second preset range, obtaining a review result of whether the first facial feature and the fourth target facial feature are the same face, the review result including: a correct hit result of the same face and an incorrect hit result of a different face;

[0180] When the review result is a correct hit result, the live image stream corresponding to the first facial feature is blocked.

[0181] Optionally, after the second determining unit blocks the live picture stream corresponding to the first facial feature, the first determining module 304 further includes:

[0182] An updating unit is used to update the first database according to the live picture stream corresponding to the first facial feature.

[0183] Optionally, the device further comprises:

[0184] A third acquisition module, used to obtain the number of audits of the fourth target facial feature;

[0185] A fourth acquisition module, used for acquiring a hit rate of the fourth target facial feature according to the number of correct hits and the number of incorrect hits in the historical review results of the fourth target facial feature;

[0186] A deletion module is used to delete the fourth target facial feature from the first database if the number of audit times is greater than or equal to a first threshold and the hit rate is less than or equal to a second threshold.

[0187] It should be noted that the embodiment of the device for processing illegal users is a device corresponding to the above-mentioned method for processing illegal users. All implementation methods of the above-mentioned method embodiments are applicable to the embodiment of the device and can achieve the same technical effects, which will not be repeated here.

[0188] To summarize, in the mining stage, the embodiments of the present invention can ensure the detection rate of faces through rotation processing, and use the first preset condition and face alignment processing to ensure that the detected faces are high-quality faces, and automatically mine live broadcast users who frequently violate the rules by extracting the facial features of the positive face image; and the facial features of the mined users who frequently violate the rules are added to the first database; in the banning stage, for the live broadcast picture stream, the single image hit suppression strategy is used to improve the banning accuracy; the extracted facial features are compared with the facial features in the first database, so as to achieve the purpose of cracking down on live broadcast users who frequently violate the rules; the banning accuracy is further guaranteed by counting the facial feature hit rate of the first database and performing automatic cleaning.

[0189] The embodiment of the present invention also provides an electronic device. Figure 4 As shown, it includes a processor 401 , a communication interface 402 , a memory 403 and a communication bus 404 , wherein the processor 401 , the communication interface 402 , and the memory 403 communicate with each other via the communication bus 404 .

[0190] The memory 403 is used to store computer programs.

[0191] When the processor 401 is used to execute the program stored in the memory 403, part or all of the steps in the method for handling illegal users provided in the embodiment of the present invention are implemented.

[0192] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0193] The communication interface is used for communication between the above terminal and other devices.

[0194] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0195] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0196] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the method for handling illegal users described in the above embodiment.

[0197] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the method for handling illegal users described in the above embodiment.

[0198] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0199] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for handling illegal users, characterized in that: The method comprises: Get the first picture of the live picture stream; Performing a preset process on the first image to obtain a first facial feature of the first image; Calculating the similarity between the first facial feature and each facial feature in the first database of illegal users; Determining a processing strategy for the first facial feature according to a similarity between the first facial feature and each facial feature in the first database; The preset processing includes: Rotate the target image to obtain a rotated target image; Determining whether the face in the rotated target image meets a first preset condition; If the face in the rotated target image meets the first preset condition, performing face alignment processing on the rotated target image to obtain a target image with a positive face image; Extracting facial features of the forward facial image to obtain third target facial features of the target image; Among them, the target picture includes the first picture or the illegal user picture; when the target picture is the first picture, the third target facial feature is the first facial feature.

2. The method according to claim 1, characterized in that: Before calculating the similarity between the first facial feature and each facial feature in the first database of illegal users, the method further includes: Obtain the picture of the violating user; The illegal user picture is processed by a preset method to obtain a second facial feature of the illegal user picture; Calculating the similarity between the second facial feature and each facial feature in the second database of the violating user, and saving the second facial feature in the second database; Determining whether the second facial feature is a first target facial feature based on a similarity between the second facial feature and each facial feature in the second database; If the second facial feature is the first target facial feature, the second target facial feature is saved in the first database.

3. The method according to claim 2, characterized in that The determining, based on the similarity between the second facial feature and each facial feature in the second database, whether the second facial feature is the first target facial feature includes: According to the similarity between the second facial feature and each facial feature in the second database, obtaining a second target facial feature in the second database whose similarity with the second facial feature is greater than a first similarity threshold; If the first number of the second target facial features is greater than the first value, the second facial features are determined to be the first target facial features.

4. The method according to claim 2, characterized in that: The preset processing includes: When the target image is the illegal user image, the third target facial feature is the second facial feature.

5. The method according to claim 1, characterized in that The step of rotating the target image to obtain a rotated target image includes: Performing face detection on the target image to obtain the face frame confidence of the target image; If the face frame confidence is less than or equal to the confidence threshold, the target image is rotated to obtain a rotated target image, and the face frame confidence of the rotated target image is greater than the confidence threshold.

6. The method according to claim 1, characterized in that The first preset condition includes: The horizontal coordinate of the nose in the rotated target image is greater than or equal to the horizontal coordinate of the left eye, and the horizontal coordinate of the nose is less than or equal to the horizontal coordinate of the right eye; and The minimum value of the horizontal coordinate of the face key point in the rotated target image is greater than or equal to zero, and the maximum value of the horizontal coordinate of the key point is less than or equal to the width value of the rotated target image, and the minimum value of the vertical coordinate of the key point is greater than or equal to zero, and the maximum value of the vertical coordinate of the key point is less than or equal to the height value of the rotated target image.

7. The method according to claim 1, characterized in that The step of performing face alignment processing on the rotated target image to obtain a target image having a positive face image includes: The coordinates of the facial key points in the rotated target image are affine transformed with the preset key point coordinates to obtain a target image with a positive facial image.

8. The method according to claim 1, characterized in that The step of determining a processing strategy for the first facial feature according to a similarity between the first facial feature and each facial feature in the first database includes: Acquire a fourth target facial feature in the first database whose similarity with the first facial feature is greater than a second similarity threshold; determining a target similarity according to a second number of the fourth target facial features; A processing strategy for the first facial feature is determined according to the target similarity.

9. The method according to claim 8, characterized in that The determining the target similarity according to the second number of the fourth target facial features includes: If the second number is one, determining the product of the similarity between the fourth target facial feature and the first facial feature and the single image hit suppression coefficient as the target similarity; If the second number is multiple, the similarity between the fifth target facial feature and the first facial feature is determined as the target similarity, and the fifth target facial feature is the facial feature with the greatest similarity to the first facial feature among the multiple fourth target facial features.

10. The method according to claim 8, characterized in that The step of determining a processing strategy for the first facial feature according to the target similarity includes: If the target similarity is within a first preset range, the live picture stream corresponding to the first facial feature is blocked.

11. The method according to claim 8, characterized in that The step of determining the processing strategy of the first facial feature according to the target similarity further includes: If the target similarity is within the second preset range, obtaining a review result of whether the first facial feature and the fourth target facial feature are the same face, the review result including: a correct hit result of the same face and an incorrect hit result of a different face; When the review result is a correct hit result, the live image stream corresponding to the first facial feature is blocked.

12. The method according to claim 10 or 11, characterized in that: After the live picture stream corresponding to the first facial feature is blocked, the method further includes: Update the first database according to the live picture stream corresponding to the first facial feature.

13. The method according to claim 11, characterized in that After obtaining the review result of whether the first facial feature and the fourth target facial feature are the same face, the method further includes: Obtaining the number of audits of the fourth target facial feature; Obtaining a hit rate of the fourth target facial feature according to the number of correct hits and the number of incorrect hits in the historical review results of the fourth target facial feature; If the number of audits is greater than or equal to a first threshold and the hit rate is less than or equal to a second threshold, the fourth target facial feature is deleted from the first database.

14. A device for processing illegal users, characterized in that: The device comprises: A first acquisition module, used to acquire a first picture of a live picture stream; A first processing module, used for performing a preset process on the first image to obtain a first facial feature of the first image; A second processing module, used for calculating the similarity between the first facial feature and each facial feature in the first database of illegal users; A first determination module, configured to determine a processing strategy for the first facial feature according to a similarity between the first facial feature and each facial feature in the first database; The preset processing includes: Rotate the target image to obtain a rotated target image; Determining whether the face in the rotated target image meets a first preset condition; If the face in the rotated target image meets the first preset condition, performing face alignment processing on the rotated target image to obtain a target image with a positive face image; Extracting facial features of the forward facial image to obtain third target facial features of the target image; Among them, the target picture includes the first picture or the illegal user picture; when the target picture is the first picture, the third target facial feature is the first facial feature.

15. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus; wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the method for handling illegal users as described in any one of claims 1 to 13 when executing the program stored in the memory.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for handling illegal users as described in any one of claims 1 to 13 is implemented.

Images