Face image selection method and device, equipment and storage medium

Abstract

The invention is suitable for the technical field of image processing, and provides a face image selection method, device and equipment and a storage medium, and the method comprises the steps: obtaining an upper feature point and a lower feature point of an eye in a to-be-discriminated face image and an associated cooperative judgment feature point, and carrying out the recognition of the to-be-discriminated face image based on the upper feature point, the lower feature point and the cooperative judgment feature point; determining the opening degree data of the eyes; and under the condition that the opening degree data indicates that the eyes are in the target state, determining the face image as a target image. According to the scheme, the calculation complexity of the eye opening degree can be reduced, the representation accuracy of the eye opening degree is improved, and the image selection effect is improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a method, device, equipment and storage medium for selecting a face image. Background Art

[0002] Face recognition of the collected images can be applied in many scenarios. For example, taking photos through facial expression recognition, identity authentication through facial feature recognition, and personnel information display in video conferences. In order to ensure the effect of facial images in different application scenarios, it is necessary to select facial images with eyes in a normal open state in combination with the detection of eye opening degree, so as to perform image processing or further use based on the appropriate images.

[0003] Existing methods for detecting eye openness usually calculate the enclosed area of ​​the upper and lower eyelids to determine whether a person's eyes are in a normally open state. However, the calculation and processing of the eyelid enclosed area is relatively complicated and the accuracy of the representation of the eye openness is insufficient, resulting in poor image selection results. Summary of the invention

[0004] The embodiments of the present application provide a facial image selection method, apparatus, device and storage medium to reduce the computational complexity of the degree of eye opening and improve the accuracy of the representation of the degree of eye opening, thereby improving the image selection effect.

[0005] A first aspect of an embodiment of the present application provides a method for selecting a face image, comprising:

[0006] Acquire upper feature points, lower feature points, and associated collaborative judgment feature points of the eyes in the first face image to be judged, wherein the collaborative judgment feature points include nose feature points and / or eye corner feature points;

[0007] Determining the eye opening degree data based on the upper feature point, the lower feature point and the collaborative judgment feature point;

[0008] When the eye opening degree data indicates that the eyes are in a target state, the first face image is determined as a target image.

[0009] A second aspect of an embodiment of the present application provides a facial image selection device, comprising:

[0010] An acquisition module, used to acquire upper feature points, lower feature points and associated collaborative judgment feature points of the eyes in the first face image to be identified, wherein the collaborative judgment feature points include nose feature points and / or eye corner feature points;

[0011] A determination module, configured to determine the eye opening degree data based on the upper feature point, the lower feature point and the collaborative judgment feature point;

[0012] A selection module is used to determine the first face image as a target image when the eye opening and closing degree data indicates that the eyes are in a target state.

[0013] A third aspect of an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the computer program.

[0014] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] The fifth aspect of the present application provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code is executed in an electronic device, the processor in the electronic device executes the steps in the method described in the first aspect above.

[0016] The above scheme in the embodiment of the present application, through the upper feature points, lower feature points and related collaborative judgment feature points of the eyes, calculates the degree of eye opening based on the feature points, and then selects a suitable face image as the target image, reduces the calculation complexity of the degree of eye opening and improves the accuracy of the representation of the degree of eye opening, thereby improving the image selection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Moreover, the same reference numerals are used throughout the drawings to represent the same components. In the drawings:

[0018] Figure 1 The process of the face image selection method of some embodiments of the present application is as follows Figure 1 ;

[0019] Figure 2 is a schematic diagram of facial feature points in some embodiments of the present application;

[0020] Figure 3 Schematic diagram of calculation of eye opening degree in some embodiments of the present application Figure 1 ;

[0021] Figure 4 Schematic diagram of calculation of eye opening degree in some embodiments of the present application Figure 2 ;

[0022] Figure 5 The process of the face image selection method of some embodiments of the present application is as follows Figure 2 ;

[0023] Figure 6 is a display schematic diagram of a screen display window in some embodiments of the present application;

[0024] Figure 7 is a structural diagram of a face image selection device in some embodiments of the present application;

[0025] Figure 8 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0028] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0029] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0030] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0031] In some application scenarios, face detection is required, such as smiling face capture, online conferencing, video chat, or access control identity recognition.

[0032] In the smiling face capture scenario, usually only whether the expression is a smiling face is detected, without detecting the state of the eyes, so it is easy to capture a closed-eye image.

[0033] In application scenarios such as online meetings or video chats, video software systems mostly display the other party's dynamic video images in real time based on video streams. In some cases, AI technology can be used to appropriately process portraits to maintain privacy. While maintaining the ability to distinguish individual portraits, it will also perform image segmentation and splicing for the video portraits of each participant and display the video on the user interface of the video conference or video call.

[0034] In some cases, compared to the commonly used real-time display of video streams, there will also be situations where the interface is displayed based on avatars. A set avatar is used to replace a person. The avatar is displayed to refer to the user during online meetings or video calls. The avatar can be a better image selected from the video stream as the display avatar to meet special usage scenarios, especially those involving privacy. Participants may not want the video to dynamically display their real-time picture scenes, but at the same time need to let the other party see their participation status image.

[0035] In order to ensure the effect of facial images in different application scenarios, in some implementations, the degree of eye opening is determined by calculating the enclosed area of ​​the upper and lower eyelids, and facial images with eyes in a normal open state are selected for image processing or further use. When calculating the enclosed area of ​​the upper and lower eyelids, it is necessary to combine feature detection, edge detection, numerical calculation and other processing. The calculation processing is relatively complex and the enclosed area of ​​the upper and lower eyelids is not accurate enough in representing the degree of eye opening, resulting in poor image selection effect.

[0036] The embodiment of the present application proposes a facial image selection method, which uses the upper feature points, lower feature points and related collaborative judgment feature points of the eyes to calculate the degree of eye openness based on the feature points, and then selects a suitable facial image as the target image, thereby reducing the calculation complexity of the degree of eye openness and improving the accuracy of the representation of the degree of eye openness, thereby improving the image selection effect.

[0037] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0038] Combination Figure 1 As shown, in some embodiments, a method for selecting a face image is proposed, comprising:

[0039] Step 101, obtaining upper feature points, lower feature points and associated collaborative judgment feature points of the eyes in a first face image to be judged.

[0040] The collaboratively determined feature points include nose feature points and / or eye corner feature points.

[0041] The acquisition of the upper feature points, the lower feature points and the associated collaborative judgment feature points is achieved by performing facial feature point recognition on the first facial image to be judged.

[0042] In one embodiment, the first facial image to be identified is input into an artificial intelligence model, and the artificial intelligence model is used to detect upper feature points, lower feature points, eye corner feature points, nose feature points, etc.

[0043] The upper feature points of the eye are feature points located on the upper side of the eye, and the lower feature points of the eye are feature points located on the lower side of the eye. A group of upper feature points and lower feature points corresponds to the same eye.

[0044] Combination Figure 2 As shown, the upper feature point of the eye can be selected as the feature point in the middle of the upper eyelid, the lower feature point of the eye can be selected as the feature point in the middle of the lower eyelid, and the feature point of the eye corner can be selected as the feature point of the inner eye corner or the feature point of the outer eye corner. The feature point of the nose is located on the bridge of the nose, for example, and can be selected as two feature points in the length direction of the nose.

[0045] Step 102, determining the eye opening degree data based on the upper feature point, the lower feature point and the collaborative judgment feature point.

[0046] Here, a small number of feature points at the upper and lower positions of the eyes, and associated nose feature points and / or eye corner feature points are selected, and the degree of eye opening is calculated using the feature points selected at different positions to detect and determine the degree of eye opening, thereby determining whether the eyes are open or closed.

[0047] Optionally, the lines between the upper feature point and the lower feature point and the same collaborative judgment feature point can be obtained, and the eye opening degree can be determined based on the angle of the lines. Alternatively, the length of the line between two collaborative judgment feature points (the distance between the two collaborative judgment feature points) can be selected as a reference, and the length of the line between the upper feature point and the lower feature point (the distance between the upper and lower feature points) and the reference can be calculated to determine the eye opening degree, wherein the two collaborative judgment feature points can be selected as two feature points in the upper and lower directions.

[0048] In an optional implementation, step 102 determines the eye opening degree data based on the upper feature point, the lower feature point and the collaborative judgment feature point, including:

[0049] In the case where the collaboratively determined feature points include eye corner feature points, the angle between the first connecting line and the second connecting line is determined as the eye opening degree data.

[0050] The first connecting line is a connecting line between the upper feature point and the eye corner feature point; the second connecting line is a connecting line between the lower feature point and the eye corner feature point.

[0051] In one calculation method of eye opening data, combined with Figure 2 and Figure 3 As shown, based on the upper feature point A, lower feature point B and corner feature point C of the eye, the line connecting the upper feature point A and the corner feature point C and the line connecting the lower feature point B and the corner feature point C are used to determine the angle ∠ACB as the eye opening data, which is used to measure the degree of eye opening and determine whether the eyes are in an open or closed state.

[0052] Since the position of feature points around the eyes will change with the degree of eye opening and closing, the wider the eyes are opened, the larger the angle between the upper and lower feature points of the eyes and the corner of the eyes will be, and vice versa. Here, angle calculation is used instead of eye area calculation to simplify feature usage and data calculation. Figure 2 As shown in FIG. 1 , the angle formed by the upper and lower feature points of the eyes and the feature points of the corners of the eyes better represents the degree of eye openness.

[0053] Combination Figure 3 As shown in the figure, when the eye opening angle is large, the angle formed is as follows Figure 3 As shown on the left side of the middle, when the eye opening angle is small, the angle formed is as follows Figure 3 Shown on the right.

[0054] When calculating the angle ∠ACB, split the opening and closing angle ∠ACB into two upper and lower angles ∠1 and ∠2 based on the horizontal axis. Figure 3 In the case on the left side, ∠1 is the angle between the horizontal axis and the line connecting the feature points A and C, and this ∠1 is in the first quadrant, and ∠2 is the angle between the horizontal axis and the line connecting the feature points C and B, and this ∠2 is in the fourth quadrant. Figure 3 In the case on the middle right side, ∠1 is the angle between the line connecting the feature points C and A and the horizontal axis, and ∠2 is the angle between the line connecting the feature points C and B and the horizontal axis. Both ∠1 and ∠2 are in the first quadrant.

[0055] When calculating, assume that the coordinates of the upper feature point A are (x1, y1), the coordinates of the lower feature point B are (x2, y2), and the coordinates of the eye corner feature point C are (x3, y3), then:

[0056] The first step is to calculate the arc tangent of the first and second lines:

[0057] a1=atan2(x1-x3, y1-y3), where a1 is the radian value of ∠1, corresponding to the angle between the vector from point C to point A and the positive direction of the x-axis.

[0058] a2=atan2(x2-x3, y2-y3), where a2 is the radian value of ∠2, corresponding to the angle between the vector from point C to point B and the positive direction of the x-axis.

[0059] atan2 is the inverse tangent function in C programming. Since x and y in atan2(x, y) have positive and negative signs, if the signs of x and y are different, the calculated angle will fall in different quadrants.

[0060] The second step is to calculate the inverse tangent difference, which is the inverse tangent value of ∠ACB:

[0061] a=a1-a2=atan2(x1-x3, y1-y3)-atan2(x2-x3, y2-y3). a is the arc tangent value of ∠ACB, specifically the radian value of ∠ACB.

[0062] The third step is to take the acute angle of the included angle ∠ACB based on the inverse tangent value a of ∠ACB:

[0063] If a>π, then a=a–2×π; if a<-π, then a=a+2×π.

[0064] The fourth step is to convert the arc tangent value a of ∠ACB after taking the acute angle into the final angle:

[0065] ∠ACB=abs(a×180.0÷π), convert radians to degrees.

[0066] In this case, the angle value of ∠ACB is compared with the preset angle threshold angle_th. If ∠ACB>angle_th, it can be considered that the eyes are in an open state, otherwise it is considered that the eyes are in a closed state, thereby realizing the eye opening and closing judgment.

[0067] Optionally, if the angle is greater than a threshold, it is determined that the eye opening degree data indicates that the eyes are in an open state, and the current first face image can be used as a suitable target image. Otherwise, it is determined that the eye opening degree data indicates that the eyes are in a closed state.

[0068] Differently, the embodiment of the present application also provides a method for judging the opening and closing state of the eyes, using the ratio of the distance between the upper and lower feature points of the eyes to the distance between the two feature points in the length direction of the nose as the basis for judgment. Here, the ratio calculation is used instead of the closed area calculation around the eyes, which simplifies the feature usage and data calculation amount, and better represents the degree of eye opening.

[0069] In an optional implementation, step 102 determines the eye opening degree data based on the upper feature point, the lower feature point and the collaborative judgment feature point, including:

[0070] In the case where the collaboratively determined feature points include the nose feature point, a ratio of the third connecting line to the fourth connecting line is determined as the eye opening degree data.

[0071] Among them, combined Figure 2 and Figure 4 As shown, the third connecting line is the connecting line between the upper feature point A and the lower feature point B, and the fourth connecting line is the connecting line between the first feature point M and the second feature point N in the length direction of the nose.

[0072] Combination Figure 4 As shown, the length of the line between A and B is taken as the distance between the two feature points, and the length of the line between C and D is taken as the distance between the two feature points.

[0073] In one example, the ratio of the distance between the upper and lower feature points of the eye to the distance between the upper and lower feature points of the nose bridge is used as the eye opening data to measure the degree of eye opening and determine whether the eyes are in an open or closed state.

[0074] Assuming the coordinates of the upper feature point A of the eye (x1, y1), the coordinates of the lower feature point B (x2, y2), the coordinates of the first feature point M in the length direction of the nose (x3, y3) and the coordinates of the second feature point N (x4, y4), then:

[0075] The distance between the upper and lower feature points of the eye D1 = abs((x1-x2)*(x1-x2)+(y1-y2)*(y1-y2));

[0076] The distance between the two feature points of the nose D2 = abs((x3-x4)*(x3-x4)+(y3-y4)*(y3-y4));

[0077] Distance ratio rate = D1 / D2.

[0078] In this case, the ratio rate is compared with a preset ratio threshold angle_th. If rate>rate_th, it can be considered that the eyes are in an open state, otherwise it is considered that the eyes are in a closed state, thereby realizing the eye opening and closing judgment.

[0079] Optionally, if the ratio is greater than a threshold, it is determined that the eye opening degree data indicates that the eyes are in an open state, and the current first face image can be used as a suitable target image. Otherwise, it is determined that the eye opening degree data indicates that the eyes are in a closed state.

[0080] Optionally, in order to improve the detection accuracy, in the embodiment of the present application, the above-mentioned ratio judgment method and the angle judgment method can be combined and used together as a basis for determining the opening and closing state of the eyes to improve the accuracy of the detection results.

[0081] Step 103: When the eye opening degree data indicates that the eyes are in a target state, the first face image is determined as a target image.

[0082] Optionally, the target state may be an open state or a closed state, or a specific state in which the degree of opening is within a set range, and finally the corresponding first facial image is determined as a suitable target image to achieve detection and selection of the best facial image.

[0083] In this process, based on the upper and lower feature points of the eyes, associated collaborative judgment feature points are introduced, and collaborative judgment is performed in combination with the feature points of the corners of the eyes and the nose. The degree of eye opening is calculated based on the feature points, and then a suitable face image is selected as the target image, which reduces the calculation complexity of the degree of eye opening, improves the accuracy of the representation of the degree of eye opening, and improves the image selection effect.

[0084] The above-mentioned face image selection method of the present application can be used in large-scale image analysis systems to achieve real-time image quality screening, and trigger related commands or execute the use of the target image after selecting a suitable target image, etc. It is suitable for use in various application scenarios and has application flexibility.

[0085] In an optional embodiment, in combination Figure 5 As shown, in step 103, when the eye opening and closing degree data indicates that the eyes are in the target state, after determining the first face image as the target image, the step further includes:

[0086] Step 104: Display the target image as a user screen display image in a target screen display window associated with the target user ID.

[0087] The user screen display image may be a user avatar or an image that replaces the user screen in the video window.

[0088] This process can be applied to similar scenarios such as video calls or video chats. After selecting a suitable target image based on the human eye opening degree data, the target image is displayed as the user screen display image of the participating user in the screen display window corresponding to the ID (Identity Document) of the current participating user, thereby realizing effective application of the target image and meeting special usage scenarios, especially those involving privacy, where the participating users do not want the video to dynamically display their real-time screen scenes, but at the same time need to let the other party see the display needs of their status images.

[0089] In an optional embodiment, before step 101 acquires the upper feature points, the lower feature points and the associated collaborative judgment feature points of the eyes in the first face image to be identified, it also includes:

[0090] Get the video image of the target user ID;

[0091] When a frontal face image of a human face is detected from a video image, specific expression detection is performed on the frontal face image;

[0092] If the confidence level of the specific expression in the frontal face image is higher than the threshold, the frontal face image is used as the first face image to be identified.

[0093] The frontal face image refers to a straight face image of a person facing the camera directly and appearing in the image.

[0094] The specific expression may be a happy expression, and the happy expression here is, for example, a smiling face expression.

[0095] The video image may be a video image of a user participating in a video call or video chat or other similar scenario. The video software system uses a camera to obtain real-time video images of the participating users, and on this basis, captures and displays facial images.

[0096] In one embodiment, the detection of whether a video image contains a face image, whether the face image is a frontal face image, and a specific facial expression can be achieved with the help of an artificial intelligence model. The artificial intelligence model can be a trained retinaface model.

[0097] The video image is input into the retinaface model. If the retinaface model detects a face from the input image, a face detection frame is obtained. The face detection frame defines the position and size of all faces detected in the input image. Then, the face image can be cropped according to the face detection frame after expansion at a certain ratio for the next step (in order to ensure the integrity of the face information, the actual cropping area will be slightly larger than the detected face detection frame).

[0098] Subsequently, based on the cropped face image, whether the face is a frontal face is detected based on information such as the face's position, size, and feature points. After detecting that it is a frontal face image, expression recognition is performed on it. The expression feature points distributed in various facial areas are detected. For example, the positions of multiple feature points (for example, 468) in detailed organ areas such as the eyes, nose, and mouth in the face are detected to identify whether the face image is a specific expression.

[0099] Expression recognition can use model detection to identify the expression category of the face in the frontal face image. The model is a classification model. For example, the model is trained to have a discrimination output of 6 expression categories. For the sake of ease of explanation, all expressions are classified into two categories in this case, "happy" expressions (i.e. specific expressions) and "other" expressions. After the face image is processed by the model, a confidence score for each expression category output by the model will be obtained, including a score for the "happy" expression and a score for the "other" expression, thereby realizing the detection and discrimination of specific expressions in the frontal face image.

[0100] When detecting facial feature points of expressions on a facial image, the feature points required in step 101 can also be detected simultaneously, so as to facilitate fast and efficient detection of facial feature points and improve image processing efficiency.

[0101] When a facial image is detected based on a video image and it is detected that the facial image is a frontal facial image and the face in the frontal facial image has a specific expression, the facial image is used as the first facial image to be identified, and the eye opening degree detection and identification is performed. When the eye opening degree meets the requirements, the facial image is determined as the best image, that is, a suitable target image.

[0102] In this case, based on the existing video image, the face is captured and facial state detection is performed. When the face is judged to be a frontal face, facial expression recognition is performed. When the recognition result is a "happy" expression, eye openness detection is performed. When it is detected that the eyes are not closed, the image can be considered to be an optimal image.

[0103] During the implementation process, the judgment of the best image needs to meet the following conditions: the face detection is a frontal face, the expression recognition is "happy", and the eyes are detected to be open to a certain degree. Based on the above three features, an image can be determined to be the best image.

[0104] In an embodiment of the present application, the best image is displayed as a user screen display image in a target screen display window associated with a target user ID, that is, a best image selected from video images of participating users is displayed instead of displaying a real-time video image.

[0105] When a better image is subsequently detected, it can be used to replace the best image displayed in the display frame for updated display.

[0106] In an optional implementation, step 104 displays the target image as a user screen display image in a target screen display window associated with the target user ID, including:

[0107] If the target screen display window displays the first user screen display image, the first user screen display image in the target screen display window is replaced with the target image for display.

[0108] Here, the first user screen display image may be the first display image in the target screen display window, or an image with poor display effect under special circumstances.

[0109] In these situations, the currently determined best image (ie, the target image) has a better image effect, and is directly replaced with the target image for display update, thereby ensuring a rapid improvement in image display effect.

[0110] Alternatively, step 104 includes:

[0111] If the target screen display window displays a first user screen display image, then when it is determined that the confidence of a specific expression in the target image is greater than the confidence of the specific expression in the first user screen display image, the first user screen display image in the target screen display window is replaced with the target image for display.

[0112] Here, the first user screen display image may be a previously selected best image or a better image that has been displayed in the target screen display window.

[0113] In these cases, the confidence level of a specific expression in a facial image is introduced, and the image effects between the currently determined best image and the previously determined best image or a better image are compared with the confidence level.

[0114] The confidence score of a specific expression in a face image is used as a reference value for the effect of the same expression category in different images. Optionally, the confidence score of a specific expression can be used to determine the range of expression movements. For example, if the confidence score of a "happy" expression is greater than a threshold when the face images in different video frames are all detected to have a "happy" expression, but the one with a higher confidence score is a more "happy" expression (i.e., a larger smile), or an image with a more obvious "happy" feature, it can be considered that its display effect is better.

[0115] This process introduces the confidence of specific expressions in facial images. Based on the best image, a confidence score is added to compare the image effects. When the confidence score is more reasonable than the previous best image while meeting the best image requirements, the image can be considered to be a better image.

[0116] When it is determined that the confidence of a specific expression in the current target image is greater than the confidence of a specific expression in the first user screen display image in the target screen display window, it can be considered that the current target image has a better image effect, and the first user screen display image in the target screen display window is replaced with the current target image to ensure continuous updating and improvement of the image display effect.

[0117] The confidence score of the better image must be greater than the score of the best image that has been displayed, and must be less than the upper limit score. The confidence score of the newly determined target image can be considered more reasonable than the previous best image. By comparing the confidence scores of high-quality images, we can find better images that are better than the best image, while preventing the appearance of overly exaggerated expression images.

[0118] Corresponding to the above implementation process, in an optional implementation manner, the display mode of the first user screen display image includes:

[0119] When a second face image to be identified is detected from the video image, determining whether the user ID corresponding to the picture display window displayed in the user interface includes the target user ID;

[0120] If not included, the target screen display window is displayed, and the second face image is displayed in the target screen display window as the first user screen display image.

[0121] The second face image is an image frame containing a face in the video image. When determining that the image frame in the video image is a face image, corresponding determination processing needs to be performed on it.

[0122] This process corresponds to the situation where the first user screen display image is the first display image in the user interface.

[0123] Alternatively, if included, when it is determined that the user screen display image in the target screen display window is the first display image and the face in the second face image has a specific expression, the second face image is displayed as the first user screen display image to replace the first display image in the target screen display window.

[0124] This process corresponds to the situation where the first user screen display image is not the first display image in the user interface.

[0125] Combination Figure 6 As shown, the display windows corresponding to the participating user IDs can be spliced ​​into a display image of the faces of the participating users in the current screen for the participating users to view each other. One participating user ID corresponds to a fixed display window. Optionally, when the target image corresponding to a participating user ID disappears from the display window and is not updated in time, the display window is closed.

[0126] The user screen display image in the display window corresponding to each participating user ID is displayed in the following specific manner:

[0127] When a facial image is detected from the video image of participating user A, the image is the aforementioned second facial image. First, the user ID of user A is matched with the screen display windows of each user being displayed in the video software system. If they match, it means that the screen display window of user A is already being displayed, and the process proceeds as if the ID match was successful. Otherwise, it is displayed as a new facial image.

[0128] The display mode of the newly added face image is as follows:

[0129] The newly added face image is the first time that the face image of user A is displayed, so a face image needs to be selected and displayed in the corresponding screen display window as soon as possible. At this time, it is not necessary to perform high-quality detection on the detected face image. Therefore, the newly added face image may not be a face with a "happy" expression, or even a frontal face image, but may be a face image with eyes closed. In order to quickly fill an empty screen display window, the face image recognized from the video image still needs to be displayed to form the first display image in the window.

[0130] The specific ways for ID matching to succeed are:

[0131] The ID matching is successful, indicating that the face image of user A has been displayed in the corresponding screen display window, but it may be in the following two situations: 1. Only the newly added face image is displayed in the window, and any image quality may not be determined, or only part of the image quality is determined; 2. The best image of the face is displayed.

[0132] When the image 1 is currently displayed in the screen display window of user A, in order to quickly replace the existing face image without quality judgment (the first image displayed in the screen display window), it is only necessary to detect whether the face is frontal and whether the facial expression is a specific expression of "happy". If the face image is detected to be frontal and with a "happy" expression, the eye opening degree can be ignored and the first image can be directly replaced. In this case, the screen display window of user A is updated to display the face image with only partial quality judgment.

[0133] When the facial image with only partial quality judgment mentioned above is currently displayed in the picture display window of user A, then based on the facial image identified from the video image of user A, it is detected whether the facial expression is positive and whether it is "happy". If the facial image is detected to be positive and with a "happy" expression, the degree of eye opening is continued to be detected to obtain the best image (for the best image detection process, see the description in the aforementioned embodiment), and the best image is used to replace the facial image with only partial quality judgment displayed in the picture display window of user A to update the display.

[0134] When the image 2 mentioned above is displayed in the picture display window of the current user A, it is necessary to continue to perform the best image detection process for the face image detected from the video image of the participating user A (see the description of the best image detection process in the aforementioned embodiment), and when the face image is determined to be the best image, it is compared with the best image currently displayed in the picture display window of user A based on the confidence score of the "happy" expression. If the best image currently displayed in the window is judged to be a better image, it will not be replaced. Otherwise, the newly determined best image will be regarded as the better image and will replace the best image currently displayed in the window to achieve display update.

[0135] In some cases, when the second facial image meets certain features, it is treated as the first facial image and corresponding image processing is performed on it.

[0136] For example: if it is determined that the user ID corresponding to the screen display window displayed in the user interface includes the target user ID, then when it is determined that the user screen display image in the target screen display window is not the first displayed image, it is determined whether the second facial image is a frontal image of a face. If it is a frontal image of a face, it is possible to perform a step of detecting a specific expression on the frontal image when a frontal image of a face is detected from a video image, and if the confidence level of the specific expression in the frontal image is higher than a threshold, taking the frontal image as the first facial image to be identified, and then perform various image processing operations on the first facial image in the aforementioned embodiments.

[0137] The above processing process can be applied to different image display processing situations, meet the image display update requirements in various scenes, and improve the image display effect.

[0138] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0139] Based on the same inventive concept, the embodiment of the present application also provides a facial image selection device. The facial image selection device provided in the embodiment of the present application can implement each process of the embodiment of the facial image selection method described above, and can achieve the same technical effect. Therefore, the specific limitations in one or more facial image selection device embodiments provided below can refer to the limitations on the facial image selection method described above. To avoid repetition, they will not be repeated here.

[0140] In this embodiment, the computing side can be divided into functional modules according to the above method. For example, each function can be divided into functional modules, or two or more functions can be integrated into one processing module.

[0141] See also Figure 7 , Figure 7 This is a structural diagram of a facial image selection device provided in an embodiment of the present application. For the sake of ease of explanation, only the parts related to the embodiment of the present application are shown.

[0142] The face image selection device 700 includes:

[0143] An acquisition module 701 is used to acquire upper feature points, lower feature points and associated collaborative judgment feature points of the eyes in the first face image to be identified, wherein the collaborative judgment feature points include nose feature points and / or eye corner feature points;

[0144] A determination module 702, configured to determine the eye opening degree data based on the upper feature point, the lower feature point and the collaborative judgment feature point;

[0145] The selection module 703 is used to determine the first face image as a target image when the eye opening and closing degree data indicates that the eyes are in a target state.

[0146] Optionally, the determination module 702 is specifically configured to:

[0147] In a case where the collaborative judgment feature point includes the eye corner feature point, determining an angle between the first connecting line and the second connecting line as the eye opening degree data;

[0148] The first connecting line is a connecting line between the upper feature point and the eye corner feature point; the second connecting line is a connecting line between the lower feature point and the eye corner feature point.

[0149] Optionally, the selection module 703 is specifically used for:

[0150] If the angle is greater than a threshold, it is determined that the eye opening degree data indicates that the eyes are in an open state, and the first face image is determined as the target image.

[0151] Optionally, the determination module 702 is specifically configured to:

[0152] In a case where the collaborative determination feature point includes the nose feature point, determining a ratio of the third connecting line to the fourth connecting line as the eye opening degree data;

[0153] The third connecting line is a connecting line between the upper feature point and the lower feature point, and the fourth connecting line is a connecting line between the first feature point and the second feature point in the length direction of the nose.

[0154] Optionally, the selection module 703 is specifically used for:

[0155] If the ratio is greater than a threshold, it is determined that the eye opening degree data indicates that the eyes are in an open state, and the first facial image is determined as the target image.

[0156] Optionally, the device further comprises:

[0157] The display module is used to display the target image as a user screen display image in a target screen display window associated with a target user ID.

[0158] Optionally, the acquisition module 701 is further used for:

[0159] Acquire a video image of the target user ID;

[0160] When a frontal face image of a human face is detected from the video image, performing specific expression detection on the frontal face image;

[0161] If the confidence level of the specific expression in the frontal face image is higher than a threshold, the frontal face image is used as the first facial image to be identified.

[0162] Optionally, the display module is specifically used for:

[0163] If the target screen display window displays a first user screen display image, then the first user screen display image in the target screen display window is replaced with the target image for display; or,

[0164] If a first user screen display image is displayed in the target screen display window, then when it is determined that the confidence of the specific expression in the target image is greater than the confidence of the specific expression in the first user screen display image, the first user screen display image in the target screen display window is replaced with the target image for display.

[0165] Optionally, the display module is further specifically used for:

[0166] In the case where a second face image to be identified is detected from the video image, determining whether the user ID corresponding to the picture display window displayed in the user interface includes the target user ID;

[0167] If not, the target screen display window is displayed, and the face image is displayed as the first user screen display image in the target screen display window; or,

[0168] If included, when it is determined that the user screen display image in the target screen display window is the first display image and the face in the second face image has the specific expression, the second face image is displayed as the first user screen display image to replace the first display image in the target screen display window.

[0169] The above integrated modules can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0170] It should be noted that the relevant contents of each step involved in the above method embodiment can all be referred to the functional description of the corresponding functional module, which will not be repeated here.

[0171] In one embodiment, Figure 8 As shown, a computer device is provided. The computer device 8 of this embodiment includes: at least one processor 800 ( Figure 8 Only one is shown), a memory 801, and a computer program 802 stored in the memory 801 and executable on the at least one processor 800, wherein the processor 800 implements the steps of any of the above-mentioned method embodiments when executing the computer program 802.

[0172] The computer device 8 may be a computing device such as a desktop computer, a notebook, a palm computer, etc. The computer device 8 may include, but is not limited to, a processor 800 and a memory 801. Those skilled in the art will understand that Figure 8 It is only an example of the computer device 8 and does not constitute a limitation of the computer device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0173] The processor 800 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0174] The memory 801 may be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. The memory 801 may also be an external storage device of the computer device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 8. Further, the memory 801 may also include both an internal storage unit and an external storage device of the computer device 8. The memory 801 is used to store the computer program and other programs and data required by the computer device. The memory 801 may also be used to temporarily store data that has been output or is to be output.

[0175] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0176] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0177] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0178] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the apparatus / computer equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of the apparatus or unit, which can be electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0181] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0182] The present application implements all or part of the processes in the above-mentioned embodiment methods, and may also be implemented through a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0183] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A facial image selection method, characterized in that: include: Acquire upper feature points, lower feature points, and associated collaborative judgment feature points of the eyes in the first face image to be judged, wherein the collaborative judgment feature points include nose feature points and / or eye corner feature points; Determining the eye opening degree data based on the upper feature point, the lower feature point and the collaborative judgment feature point; When the eye opening degree data indicates that the eyes are in a target state, the first face image is determined as a target image.

2. The method according to claim 1, characterized in that The determining of the eye opening degree data based on the upper feature point, the lower feature point and the collaborative judgment feature point includes: In a case where the collaborative judgment feature point includes the eye corner feature point, determining an angle between the first connecting line and the second connecting line as the eye opening degree data; The first connecting line is a connecting line between the upper feature point and the eye corner feature point; the second connecting line is a connecting line between the lower feature point and the eye corner feature point.

3. The method according to claim 2, characterized in that When the eye opening / closing degree data indicates that the eyes are in a target state, determining the first face image as a target image includes: If the angle is greater than a threshold, it is determined that the eye opening degree data indicates that the eyes are in an open state, and the first face image is determined as the target image.

4. The method according to claim 1, characterized in that The determining of the eye opening degree data based on the upper feature point, the lower feature point and the collaborative judgment feature point includes: In a case where the collaborative determination feature point includes the nose feature point, determining a ratio of the third connecting line to the fourth connecting line as the eye opening degree data; The third connecting line is a connecting line between the upper feature point and the lower feature point, and the fourth connecting line is a connecting line between the first feature point and the second feature point in the length direction of the nose.

5. The method according to claim 4, characterized in that When the eye opening / closing degree data indicates that the eyes are in a target state, determining the first face image as a target image includes: If the ratio is greater than a threshold, it is determined that the eye opening degree data indicates that the eyes are in an open state, and the first facial image is determined as the target image.

6. The method according to any one of claims 1 to 5, characterized in that: In the case where the eye opening / closing degree data indicates that the eyes are in the target state, after determining the first face image as the target image, the method further includes: The target image is displayed as a user screen display image in a target screen display window associated with the target user ID.

7. The method according to claim 6, characterized in that Before obtaining the upper feature points, the lower feature points and the associated collaborative judgment feature points of the eyes in the first face image to be identified, the method further includes: Acquire a video image of the target user ID; When a frontal face image of a human face is detected from the video image, performing specific expression detection on the frontal face image; If the confidence level of the specific expression in the frontal face image is higher than a threshold, the frontal face image is used as the first facial image to be identified.

8. The method according to claim 7, characterized in that The step of displaying the target image as a user screen display image in a target screen display window associated with a target user ID includes: If the target screen display window displays a first user screen display image, then the first user screen display image in the target screen display window is replaced with the target image for display; or, If a first user screen display image is displayed in the target screen display window, then when it is determined that the confidence of the specific expression in the target image is greater than the confidence of the specific expression in the first user screen display image, the first user screen display image in the target screen display window is replaced with the target image for display.

9. The method according to claim 8, characterized in that The display mode of the first user screen display image includes: In the case where a second face image to be identified is detected from the video image, determining whether the user ID corresponding to the picture display window displayed in the user interface includes the target user ID; If not, the target screen display window is displayed, and the second face image is displayed as the first user screen display image in the target screen display window; or, If included, when it is determined that the user screen display image in the target screen display window is the first display image and the face in the second face image has the specific expression, the second face image is displayed as the first user screen display image to replace the first display image in the target screen display window.

10. A facial image selection device, characterized in that: include: An acquisition module, used to acquire upper feature points, lower feature points and associated collaborative judgment feature points of the eyes in the first face image to be identified, wherein the collaborative judgment feature points include nose feature points and / or eye corner feature points; A determination module, configured to determine the eye opening degree data based on the upper feature point, the lower feature point and the collaborative judgment feature point; A selection module is used to determine the first face image as a target image when the eye opening and closing degree data indicates that the eyes are in a target state.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images