Face feature detection method, neural network model training method and electronic equipment

Abstract

The invention provides a face feature detection method, a neural network model training method and electronic equipment, and belongs to the technical field of electronic equipment. According to the face feature detection method, a first camera in the electronic equipment is used for shooting a first image, and a first face image in the first image is processed based on a neural network model, so that features associated with a face feature detection service in the first face image are reserved and even enhanced; and weakening features irrelevant to the face feature detection service in the first face image. And then executing a face feature detection service based on the image obtained after processing, for example, detecting whether a fixation point of a user is located in a display screen of the electronic equipment. Through the method, the accuracy of face feature detection can be improved.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and in particular, relates to a facial feature detection method, a neural network model training method and electronic equipment. Background Art

[0002] With the rapid development of electronic information technology and the maturity of communication technology, people have begun to explore new human-computer interaction methods that are independent of touch operation or keyboard and mouse input, that is, non-contact interaction methods, such as voice recognition, gesture recognition, face recognition, eye tracking, etc., which can provide users with more convenient and more diverse interaction methods and enhance the user experience.

[0003] However, the current technologies related to contactless interaction are not mature enough and still have some defects. For example, when implementing contactless interaction by detecting facial features, detection failures or false detections may still occur, resulting in a poor user experience. Summary of the invention

[0004] The embodiments of the present application provide a facial feature detection method, a neural network model training method and an electronic device, which can improve the accuracy of facial feature detection.

[0005] In a first aspect, a facial feature detection method is provided, which is applied to an electronic device having a first camera, and the method specifically comprises: using the first camera to obtain a first image, then processing the first facial image in the first image based on a first neural network model, and outputting a second facial image, and then performing a facial feature detection service according to the second facial image. Among them, the feature intensity of the first feature in the second facial image is greater than or equal to the feature intensity of the second feature in the first facial image, the first feature and the second feature are related to the facial feature detection service, and the first feature corresponds to the second feature; the feature intensity of the third feature in the second facial image is less than the feature intensity of the fourth feature in the first facial image, the third feature and the fourth feature are not related to the facial feature detection service, and the third feature corresponds to the fourth feature.

[0006] In the above scheme, after obtaining the first face image, the face feature detection service is not directly performed according to the first face image, but the first face image is first preprocessed to obtain the second face image: on the one hand, the features related to the face feature detection service in the first face image are retained or even strengthened, that is, the features required for subsequent detection of face features are retained or strengthened. For example, the first feature is a feature related to the face feature detection service in the first face image. When processing the first face image, the first feature can be retained or strengthened. Therefore, the feature intensity of the feature corresponding to the first feature in the second face image (i.e., the second feature) is greater than (corresponding to the scene of strengthening the first feature) or equal to (corresponding to the scene of retaining the first feature) the feature intensity of the first feature. On the other hand, the features in the first face image that are not related to the face feature detection service are weakened, that is, the noise features that are not required for subsequent detection of face features are weakened. Through the above brain stem, the interference of the face feature detection process can be reduced, so that when the face feature detection service is subsequently performed, more attention can be paid to the features related to the face feature detection service, thereby improving the detection accuracy and reducing the probability of detection failure and misdetection.

[0007] Optionally, performing a facial feature detection service according to the second facial image includes: detecting whether the user's gaze point is located within the display screen of the electronic device according to the second facial image. After performing the facial feature detection service, the method further includes: controlling the on / off state of the display screen of the electronic device according to the detection result of the facial feature detection service.

[0008] The solution provided by the embodiment of the present application can be applied to the "watching without turning off the screen" or "watching with the screen on" scenario, that is, the electronic device can detect whether the user's gaze point is located within the display screen of the electronic device based on the second face image, that is, detect whether the user is currently watching the display screen, so as to control the electronic device to switch between the screen-off state and the screen-on state. For example, in the screen-on state, if the user is not currently watching the display screen, the electronic device can be automatically switched from the screen-on state to the screen-off state, thereby saving power; if the user is currently watching the display screen, the electronic device can be kept in the screen-on state, or switched from the screen-off state to the screen-on state, thereby improving the user experience.

[0009] Optionally, the electronic device is currently in a screen-on state. At this time, using the first camera to shoot to obtain the first image includes: when the screen-off countdown is less than or equal to the first threshold, using the first camera to shoot to obtain the first image; according to the detection result of the face feature detection service, controlling the on and off state of the display screen of the electronic device, including: when the detection result indicates that the user's gaze point is within the display screen of the electronic device, maintaining the screen-on state of the display screen of the electronic device and resetting the screen-off countdown.

[0010] When the solution provided by the embodiment of the present application is applied to the "gaze without turning off the screen" scenario, when the screen-off countdown is about to end (that is, when the screen-off countdown is less than or equal to the first threshold), the first camera can be called to obtain a facial image to detect whether the user's gaze point falls within the display screen of the electronic device, and decide whether to automatically turn off the screen based on the detection result, so that it can decide whether to automatically turn off the screen based on the user's actual usage, thereby improving the user experience. If the user's gaze point is within the display screen, the screen will not be automatically turned off. In addition, the electronic device resets the screen-off countdown, that is, the screen-off countdown starts again from the beginning, so that the electronic device can still automatically turn off the screen after the user no longer uses the electronic device.

[0011] Optionally, the first feature and the second feature include eye features and / or facial contour features, and the third feature and the fourth feature include one or more of the following: light and shadow features caused by lighting, features of facial features other than eyes, biological features other than facial features, jewelry features, and makeup features.

[0012] Therefore, in the above scheme, when processing the first face image, the eye features and / or facial contour features can be retained or enhanced, and the light and shadow features caused by the influence of lighting, the features of the facial features other than the eyes, the biological features other than the facial features, the jewelry features, the makeup features, etc. can be weakened. In this way, when performing gaze point detection, more focus can be placed on the eye features and / or facial contour features, thereby improving the accuracy of detection.

[0013] Optionally, detecting whether the user's gaze point is within the display screen of the electronic device based on the second facial image includes: determining a left eye image and a right eye image based on the second facial image; processing the left eye image and the right eye image based on an open and closed eyes recognition model, and outputting an open and closed eyes result, wherein the open and closed eyes result is used to indicate whether the user is in an open eyes state; when the user is in an open eyes state, detecting whether the user's gaze point is within the display screen of the electronic device based on the second facial image, the left eye image and the right eye image.

[0014] In the above scheme, after obtaining the second face image, gaze recognition is not performed directly, but first the left and right eye images are used to identify whether the user is in an open-eye state. If so, the gaze recognition process is continued. Since the training difficulty of the open-closed eye recognition model is much lower than that of the gaze recognition model, its accuracy is very high, so accurate open-closed eye filtering in advance helps to improve the overall accuracy of gaze recognition; secondly, compared with the deeper neural network structure and more parameters of the face feature extraction model, the very shallow open-closed eye recognition model requires very little calculation. By reducing meaningless full-face branch operations, the computing power consumption can be further reduced, which helps to achieve low power consumption requirements.

[0015] Optionally, detecting whether the user's gaze point is located within the display screen of the electronic device based on the second facial image, the left eye image and the right eye image includes: extracting left eye features and right eye features from the left eye image and the right eye image respectively using an eye feature extraction model; extracting facial features from the second facial image using a facial feature extraction model; fusing the left eye features, the right eye features and the facial features to obtain fused data; processing the fused data using a gaze recognition model and outputting a gaze result, wherein the gaze result is used to indicate whether the user's gaze point is located within the display screen of the electronic device.

[0016] In the above scheme, the features of the human eye image and the features of the face image are extracted by different models respectively, which can improve the precision of feature extraction. Because different images have different characteristics, if the same model is used to extract the features of different types of images, it may affect the accuracy of subsequent gaze recognition.

[0017] Optionally, the electronic device also includes a second camera; the method also includes: when the credibility of the detection result of the facial feature detection service is less than a second threshold or the facial feature detection service fails to be executed, obtaining an average pixel value of the first facial image; when the average pixel value of the first facial image is lower than a third threshold, using the second camera to shoot to obtain a second image, the second image including a third facial image; using the second neural network model to process the third facial image and output a fourth facial image; performing a facial feature detection service according to the fourth facial image; wherein the feature intensity of the fifth feature in the fourth facial image is greater than or equal to the feature intensity of the sixth feature in the third facial image, the feature intensity of the seventh feature in the fourth facial image is less than the feature intensity of the eighth feature in the third facial image, the fifth feature corresponds to the sixth feature, the seventh feature corresponds to the eighth feature, the fifth feature and the sixth feature are related to the facial feature detection service, the seventh feature and the eighth feature are not related to the facial feature detection service, the first image is a color image, and the second image is an infrared (infrared radiation, IR) image. Therefore, in the above scheme, a color image (i.e., the first image) is first captured by the first camera to detect facial features. Since the power consumption caused by capturing color images is relatively low, and most electronic devices are used in scenes with bright light, the imaging quality of color images meets the requirements in most scenes. Therefore, by calling the first camera to detect facial features, low power consumption can be maintained while ensuring a good detection success rate.

[0018] If the facial feature detection service is performed based on the first image, but the credibility of the detection result is low (such as less than the second threshold) or the detection fails, there may be many reasons for this situation, such as it may be caused by the current shooting light environment, it may also be caused by the detected target itself, it may also be caused by some other unexpected errors, etc. At this time, the average pixel value of the first facial image can be obtained. If the average pixel value is lower than the third threshold, it means that the image quality of the first facial image is very poor, so the poor image quality is likely to be the main reason for the low credibility of the detection result or the failure of the detection. For example, in dark light, backlight, side light or reflective scenes, the facial features in the first facial image may not be clear enough, and the electronic device may not be able to detect facial features or human eye features from the first facial image. In this case, the electronic device can continue to call the second camera to collect infrared images (i.e., the second image) to perform facial feature detection services. Since the imaging quality of infrared images is almost unaffected by light conditions, even if the light conditions are relatively poor, a high image quality can be guaranteed, thereby improving the success rate and detection accuracy of facial feature detection.

[0019] If the facial feature detection service is performed based on the first image, and the credibility of the detection result is relatively high (such as greater than or equal to the second threshold), and the detection result indicates that the user's gaze point is outside the display screen of the electronic device, then the current detection process is terminated, or the current detection process is restarted, such as re-determining whether the screen off countdown is less than or equal to the first threshold. If so, the first camera is used to capture a color image to perform the facial feature detection task. The specific implementation scheme will not be repeated. In this case, there is no need to calculate the average pixel value of the first facial image, nor is it necessary to enable the second camera to capture the second image. In other words, if the credibility of the detection result is relatively high, the detection result is directly used, and the second camera is no longer used for re-detection to reduce unnecessary waste of power consumption.

[0020] In summary, the solution provided in the present application involves two cameras (i.e., a first camera and a second camera), wherein the first camera can capture color images and the second camera can capture infrared images. In a scene with relatively good lighting conditions, using the first camera to capture color images to detect facial features can maintain low power consumption and ensure a good detection success rate. In a scene with poor lighting conditions, using the second camera to capture infrared images can ensure a high detection success rate under various lighting conditions. However, since lighting conditions are not easy to quantify, if the camera is selected only by the ambient light intensity, facial features will still fail to be detected in many scenes, or even if facial features are detected, the credibility of the detection results is relatively low. For example, in scenes with backlight, side light, or glasses reflective light, the ambient light intensity may be relatively high, but the quality of the collected color image may be relatively poor. The solution provided in the embodiment of the present application can detect facial features based on the first image but the credibility of the detection result is lower than the second threshold. It is found through analysis that the cause of failure is likely to be the poor image quality of the first image. Only then is the second camera called to capture an infrared image. Because capturing infrared images will result in higher power consumption, the above-mentioned solution can reduce the number of times infrared images are captured, thereby reducing power consumption. At the same time, a higher detection success rate can be guaranteed, and at least the situation where facial feature detection fails due to image quality can be reduced.

[0021] It is understandable that, in the case where the first image is a color image, the subsequent image processing and detection process can be directly performed based on the first image, or based on the image obtained after the first image is converted. In other words, the first face image can be an image directly captured from the first image, or an image captured from an image converted from the first image. For example, after the first image (color image) is captured by the first camera, the first image can be converted into a grayscale image, and then the first face image can be captured from the grayscale image.

[0022] Optionally, in a possible implementation, the first image is a color image. Then, using the first camera to shoot to obtain the first image includes: obtaining the current ambient light intensity; when the ambient light intensity is greater than or equal to a fourth threshold, using the first camera to shoot to obtain the first image. Optionally, in another possible implementation, the first image is an infrared image. Then, using the first camera to shoot to obtain the first image includes: obtaining the current ambient light intensity; when the ambient light intensity is less than the fourth threshold, using the first camera to shoot to obtain the first image.

[0023] Therefore, in the above scheme, when the light intensity is relatively high (ambient light intensity is greater than or equal to the fourth threshold), facial features are detected by collecting color images. When the light intensity is relatively low (ambient light intensity is less than the fourth threshold), facial features can be detected directly by collecting infrared images. Because the ambient light intensity is relatively low, it means that this is a dark light scene. If a color image is taken at this time, the image quality is likely to be poor. Directly calling the second camera to detect facial features can save the process of collecting the first image, processing the first facial image to obtain the second facial image, and detecting facial features based on the second facial image, thereby saving resources, reducing latency, and improving user experience.

[0024] Optionally, before using the first camera to capture the first image, the method further includes: determining to detect facial features. That is, the electronic device enables the first camera to capture when determining to detect facial features. The user may control the electronic device to detect facial features, or the electronic device may automatically detect facial features based on preset rules, which is not limited in this application.

[0025] Optionally, the first camera and the second camera may be one camera or two different cameras.

[0026] Optionally, the first camera and the second camera may be front cameras of the electronic device.

[0027] Optionally, the first camera is used to acquire a two-dimensional image, and the second camera is used to acquire an image containing depth information.

[0028] Optionally, the first camera is a red green blue (RGB) camera, and the second camera is a time of flight (Tof) camera.

[0029] Optionally, the first camera is an always-on (AO) camera, wherein the AO camera is a low-power RGB camera, and capturing the first image with the AO camera can further reduce power consumption.

[0030] Optionally, before processing the first facial image based on the first adaptive equalization model, the method further includes: determining that the first image includes the first facial image.

[0031] In the above scheme, after the electronic device obtains the first image by using the first camera, it can detect whether the first image includes the first facial image. If so, the subsequent process can be continued. If not, the subsequent process can be stopped, or the first camera can be called again for shooting, or other frame images taken by the first camera can be detected to see whether they include facial images (the first image and other frame images here can be multiple frame images taken by the first camera at one time), thereby avoiding the situation where the subsequent facial feature detection fails because the first image does not include the facial image, omitting the detection process in this case, saving resources and reducing latency.

[0032] Optionally, before obtaining the first image by photographing with the first camera, the method further includes: displaying a first interface on the screen, the first interface being any one of the following multiple interfaces: an interface of a first application installed in the electronic device, a system desktop, a negative one screen, and a lock screen interface.

[0033] Optionally, the first neural network model is trained based on training data, the training data includes multiple sample images, the multiple sample images are obtained by augmenting the same original image with features unrelated to the facial feature detection service, and the original image includes a third facial image.

[0034] Optionally, the first neural network model is trained based on the adversarial loss function value, the second loss function value, and the third loss function value of the first loss function value; wherein the first loss function value is obtained by comparing multiple sample images with multiple reconstructed images, the multiple reconstructed images are generated according to multiple feature vectors, the multiple feature vectors are extracted from the training data, and the multiple feature vectors, the multiple sample images, and the multiple reconstructed images correspond one to one; the second loss function value is obtained by comparing the average vector distance between the multiple feature vectors with the target distance; the third loss function value is obtained by comparing the detection result obtained by performing the face feature detection service according to the multiple feature vectors with the target result.

[0035] In a second aspect, a training method for a neural network model is provided, the method comprising: obtaining training data, the training data comprising a plurality of sample images, the plurality of sample images being obtained by augmenting the same original image with features irrelevant to a facial feature detection service, the original image comprising a facial image; extracting a plurality of feature vectors from the training data; generating a plurality of reconstructed images based on the plurality of feature vectors, the plurality of feature vectors, the plurality of sample images, and the plurality of reconstructed images corresponding one to one; performing a facial feature detection service based on the plurality of feature vectors to obtain a detection result; respectively comparing the plurality of sample images with the corresponding reconstructed images to obtain a first loss function value; comparing the average vector distance between the plurality of feature vectors with the target distance to obtain a second loss function value; comparing the detection result with the target result to obtain a third loss function value; and adjusting the parameters of the neural network model based on the first loss function value, the second loss function value, and the third loss function value.

[0036] According to the above scheme, the parameters of the neural network model can be adjusted through self-supervised training and adversarial training, so that when processing images, the neural network model can include features related to the facial feature detection business and weaken features that are not related to the facial feature detection business, thereby improving the accuracy of facial feature detection.

[0037] Optionally, the parameters of the neural network model are adjusted according to the first loss function value, the second loss function value, and the third loss function value, including: adjusting the parameters of the neural network model with the purpose of reducing the adversarial loss function value, the second loss function value, and the third loss function value of the first loss function value.

[0038] Optionally, the adversarial loss function value of the first loss function value is equal to the difference between 1 and the first loss function value.

[0039] According to a third aspect, an electronic device is provided, which has a first camera and specifically includes: a shooting module, which is used to use the first camera to shoot and obtain a first image, and the first image includes a first face image; a processing module, which is used to process the first face image based on a first neural network model and output a second face image, wherein the feature intensity of a first feature in the second face image is greater than or equal to the feature intensity of a second feature in the first face image, and the feature intensity of a third feature in the second face image is less than the feature intensity of a fourth feature in the first face image; a detection module, which is used to perform a face feature detection service according to the second face image; wherein the first feature and the second feature are related to the face feature detection service, the third feature and the fourth feature are not related to the face feature detection service, the first feature corresponds to the second feature, and the third feature corresponds to the fourth feature.

[0040] In a fourth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the electronic device implements the steps of the facial feature detection method as described in any one of the first aspect or the second aspect above.

[0041] In a fifth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the facial feature detection method as described in any one of the first aspect or the second aspect are implemented.

[0042] In a sixth aspect, a computer program product is provided. When the computer program product is run on an electronic device, the electronic device executes the facial feature detection method described in any one of the first aspect or the second aspect.

[0043] In the seventh aspect, a chip system is provided, which includes a processor coupled to a memory, and the processor executes a computer program stored in the memory to implement the facial feature detection method described in any one of the first aspect or the second aspect.

[0044] The chip system may be a single chip or a chip module composed of multiple chips.

[0045] It can be understood that the beneficial effects of the third to seventh aspects mentioned above can be found in the relevant descriptions in the first or second aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram showing an application example of an automatic screen off rule provided in an embodiment of the present application is shown;

[0047] Figure 2 A schematic diagram of an application scenario provided by an embodiment of the present application is shown;

[0048] Figure 3 Another application scenario schematic diagram provided by an embodiment of the present application is shown;

[0049] Figure 4 A schematic diagram of another application scenario provided by an embodiment of the present application is shown;

[0050] Figure 5 A schematic block diagram of a method for detecting facial features provided by an embodiment of the present application is shown;

[0051] Figure 6 An exemplary image captured by a camera of an electronic device is shown;

[0052] Figure 7An exemplary process of identifying and capturing a facial image is shown;

[0053] Figure 8 An exemplary process of determining a face image and an eye image is shown;

[0054] Fig. 9 An example diagram showing a face image and a face image after adaptive equalization processing;

[0055] Fig.10 An exemplary flow chart of image acquisition and image processing is shown;

[0056] Fig.11 An exemplary process of gaze recognition is shown;

[0057] Fig.12 A schematic diagram of another application scenario provided by an embodiment of the present application is shown;

[0058] Fig.13 A schematic diagram of an electronic device provided in an embodiment of the present application is shown;

[0059] Fig.14 An exemplary framework and an exemplary process of an electronic device for gaze recognition are shown;

[0060] Fig.15 A schematic flow chart of a method for detecting facial features provided in an embodiment of the present application is shown;

[0061] Fig.16 A system architecture diagram provided by an embodiment of the present application is shown;

[0062] Fig.17 An exemplary block diagram of a neural network model training method provided in an embodiment of the present application is shown;

[0063] Fig.18 A schematic diagram showing a process of training a neural network model provided by an embodiment of the present application;

[0064] Fig.19 A block diagram of a software and hardware system of an electronic device provided in an embodiment of the present application is shown;

[0065] Fig. 20 A block diagram of a hardware system of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. In order to more clearly describe the embodiments of the present application, some terms or technologies involved in the embodiments of the present application are first briefly introduced.

[0067] With the rapid development of electronic information technology and the maturity of communication technology, contactless interaction methods have begun to be applied to various electronic devices. The electronic device described in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA) and other devices. The embodiment of the present application does not impose any restrictions on the specific type of electronic device. For convenience, the following description is taken as an example of an electronic device being a mobile phone.

[0068] As an example, in a contactless interaction mode, a mobile phone can use a camera to detect facial features to perform corresponding operations, such as unlocking or payment operations, eye tracking, and control of display brightness or on / off status by detecting facial features. The following is an example.

[0069] In many cases, users do not actively press the power button to switch the phone's display to the off state after using the phone. In this case, if the phone always keeps the screen on, it will cause a waste of power. To solve this problem, you can configure automatic screen off rules for the phone.

[0070] For example, Figure 1 An example of an automatic screen-off rule is shown: the mobile phone currently displays the application interface of a shopping application (application, App) (such as Figure 1 In (a)), if the user does not perform any operation on the phone within 30 seconds, that is, the phone does not receive any instructions from the user within 30 seconds, the phone automatically switches from the on screen state to the off screen state (such as Figure 1 (b) in the figure). It is understandable that the 30s here can be the screen off countdown timer set by the user.

[0071] Among them, the screen-off countdown described in the embodiment of the present application is used for the electronic device to automatically complete the screen-off operation. For example, when an electronic device in the screen-on state receives an operation instruction from the user, the electronic device automatically starts the screen-off countdown through the timer. If the electronic device receives an instruction from the user again before the screen-off countdown ends, the electronic device resets the screen-off countdown, wherein the instruction here can be a touch operation of the user at any position of the display screen, or a voice operation, etc.; if the electronic device does not receive an instruction from the user before the screen-off countdown ends, the electronic device switches the display screen to the screen-off state.

[0072] In the embodiment of the present application, the screen-on state refers to the state in which the display screen of the electronic device is lit. At this time, the display screen may display the lock screen interface (i.e., the interface to be unlocked), or may display the system desktop or the negative one screen, or may display the application interface of an application installed in the electronic device (e.g., Figure 1 The application interface of the shopping app shown in (a) in the figure) and so on. The screen-on state can also be called the screen-on state, etc.; the screen-off state refers to the state in which the display screen of the electronic device is not lit, that is, the display screen is off. The screen-off state can also be called the screen-off state or the black screen state, etc.

[0073] It is understandable that the fact that the display screen is in the screen-off state does not necessarily mean that the display screen does not display any content at all. For example, the display screen can still display time, date, battery level, application notifications, wallpapers, etc. in part of the display screen when the display screen is in the screen-off state (i.e., the screen-off display function of the corresponding electronic device).

[0074] It can also be understood that if the electronic device includes multiple display screens, when the main screen is in the screen-on state and the sub-screen is in the screen-off state (or the main screen is in the screen-off state and the sub-screen is in the screen-on state), it can be determined based on actual strategies whether the electronic device is in the screen-on state or the screen-off state at this time, and this application does not make specific limitations on this situation.

[0075] The electronic device can switch between the screen-on state and the screen-off state based on the user's instructions. In one example, the electronic device is configured with a power button, and the user can switch the screen on and off state by pressing the power button, that is, pressing the power button in the screen-on state can switch to the screen-off state, and pressing the power button in the screen-off state can switch to the screen-on state. Alternatively, the electronic device can automatically switch between the screen-on state and the screen-off state based on a preset rule.

[0076] Optionally, in one implementation, the mobile phone may also reduce the brightness first and then switch to the screen off state. For example, if the user does not perform any operation on the mobile phone within 25 seconds, the mobile phone automatically reduces the brightness of the display screen (such as Figure 1After reducing the brightness of the display, if the user performs any operation on the phone within 5 seconds, the phone restores the brightness of the display to Figure 1 If the user does not perform any operation on the phone within 5 seconds, the phone automatically switches to the screen-off state (such as Figure 1 (as shown in (b)).

[0077] Through the above solution, the mobile phone can automatically turn off the screen when it does not receive any touch operation from the user for a period of time, thereby saving resources.

[0078] However, in some scenarios, the user may still be using the phone even though he or she is not performing any operation. For example, when the user is reading an article or thinking about a design solution on an electronic device, the user may be looking at the display of the electronic device but not performing any touch operation. For another example, after the user opens a shopping app on the phone, he or she may enter a certain state. Figure 1 The purchase interface shown in (a) in Figure 1 As can be seen from (a) in the figure, there are still 3 minutes and 6 seconds before the product is snapped up. In order to avoid missing the product, the user may stare at the current display page without performing any operation. If the phone automatically enters the screen-off state at this time, it will affect the user experience.

[0079] Based on this, in the solution involved in this application, the mobile phone can detect the user's line of sight before entering the screen-off state, and control the on and off state of the mobile phone based on the user's line of sight. Figure 2 If the user does not perform any operation within 30 seconds, the mobile phone can use the front camera 210 to detect whether the user is looking at the display screen before entering the screen-off state. If the user is looking at the display screen at this time, that is, the user's gaze point is within the display screen, it can be considered that the user is using the mobile phone at this time, and the mobile phone will not automatically enter the screen-off state (that is, maintain the screen of the mobile phone), and reset the screen-off timer (that is, the screen-off timer starts again from the beginning), thereby reducing the situation where the mobile phone automatically turns off the screen when the user is using the mobile phone and improving the user experience.

[0080] It should be noted that the gaze point described in the embodiment of the present application refers to a point at which the user's line of sight is aligned with the target object during the visual perception process. If the user's gaze point is within the display area of ​​the display screen of the electronic device (hereinafter referred to as the user's gaze point being within the display screen), it can be considered that the user is looking at the display screen, otherwise it is considered that the user is not looking at the display screen. In the embodiment of the present application, "detecting whether the user's gaze point is within the display screen" can be referred to as "gaze point detection" or "gaze recognition".

[0081] See also Figure 3If the user does not perform any operation within 30 seconds, the mobile phone can use the front camera 310 to detect whether the user is looking at the display screen before entering the screen-off state. If the user is not looking at the display screen at this time (that is, the user's gaze point is outside the display screen), it can be considered that the user is not using the mobile phone at this time, and the mobile phone automatically enters the screen-off state according to the original logic, thereby saving power and reducing power consumption.

[0082] From the above, we can see that the above Figure 2 and Figure 3 The scheme shown is explained by taking the "watching the screen" scenario as an example. In this scenario, the electronic device detecting facial features means that the electronic device detects whether the user's gaze point is within the display screen of the electronic device. However, it can be understood that the embodiments of the present application are not limited to the above scenario. For example, Figure 4 Another application scenario of detecting facial features is shown.

[0083] like Figure 4 As shown in (a) of FIG. 1 , the electronic device enters the lock screen interface (or is called the unlocked interface). At this time, the mark 410 is used to indicate to the user that the electronic device is currently in the lock screen state. At this time, the electronic device performs face recognition through the front camera. If the recognition is successful, the electronic device is unlocked. The interface after unlocking is as follows Figure 4 As shown in (b), the mark 420 is used to indicate to the user that the electronic device is in an unlocked state. Therefore, in this scenario, detecting facial features means capturing a facial image through a camera and extracting facial features, and then comparing the extracted facial features with information in a local or cloud database to determine whether the facial features are matched successfully.

[0084] Optionally, in Figure 4 In the scenario shown, before performing face recognition, it is also possible to determine whether the user is looking at the display screen. If so, face recognition is performed. If not, face recognition is no longer performed, thereby reducing the probability of mis-locking. Figure 4 In the scenario shown, detecting facial features may also include determining whether the gaze point of the human eye is located within the display screen.

[0085] It is understandable that the above application scenarios are only examples, and the present application is not limited thereto. Other scenarios that require detection of facial features are also within the scope of protection of the present application, such as the staring screen lighting scenario, that is, the electronic device automatically lights up the screen after detecting that the user is staring at the display screen for a period of time, or the electronic device reduces the brightness to reduce power consumption (such as Figure 1(c) in the figure, if it detects that the user is looking at the display screen, the brightness of the display screen will be automatically restored; for example, in the eye tracking control scenario, the electronic device detects the position of the user's gaze point on the display screen to open the application or notification in the gaze area, or to turn pages, zoom in on pages, etc., or track the user's dynamic gaze point to achieve contactless control, such as interactive electronic games through eye tracking; for example, in the electronic payment scenario, after the user opens the payment interface, the password verification is completed through face recognition, etc. For convenience, the following Figure 2 The “watch the screen without turning it off” scenario shown in the figure is used as an example to illustrate.

[0086] However, facial feature detection often involves complex algorithms and has high requirements on the quality of the captured facial images, so detection failure or misdetection often occurs. Regardless of the above application scenarios, once facial feature detection fails, the subsequent solutions based on the facial feature detection results may not be implemented smoothly.

[0087] In view of this, an embodiment of the present application provides a facial feature detection method that can improve detection accuracy. Specifically, before performing facial feature detection based on a captured facial image, the captured facial image can be preprocessed: on the one hand, the features associated with the facial feature detection service in the facial image are retained or even enhanced to improve the detection accuracy; on the other hand, the features in the facial feature image that are not associated with the facial feature detection service are weakened to reduce interference in the facial feature detection process. Through this solution, the probability of detection failure and false detection can be reduced. The following is combined with Figure 5 The method 100 in the embodiment of the present application is described in detail.

[0088] S110: Use a first camera to capture a first image.

[0089] Exemplarily, the electronic device has a first camera, which is, for example, a front camera of the electronic device. Figure 2 The camera 210 in FIG. 1 is a camera 210. It is understandable that the first camera may also be an external camera of the electronic device, and this application does not limit this.

[0090] The electronic device can use the first camera to capture a first image. The first image can be one of the multiple frames of images captured by the first camera. It is understandable that after the electronic device uses the first camera to capture the first image, the first image can be stored in an internal data buffer without displaying the first image on the display screen. In other words, capturing the first image is an internal operation of the electronic device, and the user is usually not aware of the process.

[0091] This application does not limit the image type of the first image.

[0092] In one example, the first image is a color image. That is, the first camera can be used to capture a color image, or the first camera can be used to obtain a two-dimensional image. For example, the first camera is an RGB camera. Specifically, the first camera can be an AO camera in the RGB camera. Its related algorithms can be run on the smart sensor hub SensorHub, and the power consumption is low. Therefore, the power consumption can be further reduced by using the AO camera.

[0093] In another example, the first image is an infrared image. That is, the first camera can be used to capture infrared images, or in other words, the first camera can be used to obtain images containing depth information, for example, the first camera is a Tof camera.

[0094] Optionally, before S110, the electronic device may also obtain the current ambient light intensity. For example, when the first image is a color image, the electronic device may pre-determine whether the current ambient light intensity is greater than or equal to the fourth threshold value. If so, it means that the lighting conditions may be good, and the first camera may be used to shoot the color image. Since the power consumption of shooting color images is relatively low, and the imaging quality of color images can be guaranteed even when the lighting conditions are good, the subsequent facial feature detection service is performed by shooting color images, which can not only reduce power consumption, but also ensure the accuracy of facial feature detection to a certain extent. For another example, when the first image is an infrared image, the electronic device may pre-determine whether the current ambient light intensity is less than the fourth threshold value. If so, it means that the lighting conditions are not good. At this time, the first camera may be used to shoot the infrared image. Since the imaging quality of the infrared image is almost unaffected by the lighting conditions, the subsequent facial feature detection service is performed through the infrared image, which can improve the accuracy of facial feature detection.

[0095] For convenience, the following embodiments are described by taking the first image as a color image as an example.

[0096] Exemplarily, the first image includes a first face image. The face image described in the embodiment of the present application refers to an image containing part or all of a face. For example, assuming Figure 6 The image 610 in corresponds to the first image (for convenience, the image 610 is displayed in black and white), and the partial image 620 therein is the first face image.

[0097] The first face image is used to perform a face feature detection service, or in other words, the first face image is used to detect face features, for example, corresponding to Figure 2In the application scenario shown, the first face image is used to detect whether the user's gaze point is within the display screen. In this scenario, the electronic device in the screen-on state can use the first camera to capture the first image when the screen-off countdown is less than a preset value (such as 4s); or the electronic device obtains the current ambient light intensity when the screen-off countdown is less than a preset value, and then uses the first camera to capture the first image based on the ambient light intensity.

[0098] Optionally, after S110 and before S120, the method further includes: determining that the first image includes a first face image. For example, detecting the first image based on a face detection model and outputting a detection result, the detection result indicating that the first image includes the first face image.

[0099] That is, after the electronic device captures the first image with the first camera, it detects whether the first image includes a face image. If it does, it continues to execute the subsequent S120 operation; if it does not, it can re-execute S110, or continue to detect whether other images captured by the first camera include a face image, or stop the face feature detection process, which is not limited in this application.

[0100] It is understandable that the present application does not limit the specific method for detecting whether an image includes a face image. Figure 7 and Figure 8 A possible implementation method is described by way of example.

[0101] Figure 7 An exemplary process of identifying and capturing a face image is given. Optionally, the original image (i.e., the first image mentioned above, here is Figure 6 The image 610 in the figure is used as an example for explanation) to perform basic preprocessing, such as scaling processing, downsampling processing, etc.

[0102] The image obtained after the processing is then input into a face detection model, which is, for example, a neural network model. The face detection model processes the input image and outputs a detection result. If the input image includes a face image, the output detection result includes the face image, and optionally, an eye image (including a left eye image and a right eye image) can also be output. If the input image does not include a face image, the output detection result indicates that the image does not include a face image.

[0103] See also Figure 8The face detection model can use facial key point detection algorithms (such as deepblueface (Dbface), practical facial landmark detector (PFLD), face-landmark-factory, multi-task convolutional neural network (MTCNN), centerface and other algorithms, not limited here) to determine the facial key points in the input image: left eye A, right eye B, nose C, left lip corner D, right lip corner E, and determine the coordinate position of each key point (such as Figure 8 Optionally, if the face is tilted, such as key points A and B are not on a horizontal line, the face image can be corrected by a face correction algorithm, and the specific process is not limited in this application. It is understandable that if the face detection model does not detect the face key points from the input image (or does not detect all five key points), it is considered that there is no face image in the input image.

[0104] Furthermore, the face detection model outputs a detection result. The detection result includes a face image. For example, the face detection model can determine a rectangle with the nose C as the center, and the rectangle includes several other key points. The image intercepted by the rectangle is the face image (corresponding to the first face image mentioned above), such as Figure 8 Optionally, the detection result also includes a left eye image. For example, the face detection model determines a fixed-size rectangle with the left eye A as the center, and the image intercepted by the rectangle is the left eye image, as shown in (b). Figure 8 As shown in (c) in FIG. 1 ; Optionally, the detection result also includes a right eye image. For example, the face detection model determines a fixed-size rectangle as the center of the right eye B, and the image intercepted by the rectangle is the right eye image, such as Figure 8 As shown in (d) in .

[0105] S120, processing the first facial image based on the first neural network model, and outputting a second facial image.

[0106] Exemplarily, before executing the facial feature detection service, the first facial image may be pre-adaptively equalized to make targeted improvements to the first facial image for the facial feature detection service, so as to improve the accuracy of subsequent facial feature detection.

[0107] For example, the first facial image is processed based on the first neural network model, and the second facial image is output. That is to say, the second facial image is an image obtained by processing the first facial image using the first neural network model.

[0108] As an example, the first neural network model is a lightweight graph-to-graph conversion neural network model, and the first neural network model is used to perform adaptive equalization processing on the image. The adaptive equalization processing in this application refers to the targeted adjustment of the features in the image for the face feature detection service, including retaining or enhancing the features related to the face feature detection service, and weakening the features unrelated to the face feature detection service, so as to achieve the purpose of targeted improvement of image quality.

[0109] Therefore, the features in the second facial image are improved in a targeted manner compared to the features in the first facial image. For example, the feature strength of the first feature in the second facial image is greater than or equal to the feature strength of the second feature in the first facial image, and the feature strength of the third feature in the second facial image is less than the feature strength of the fourth feature in the first facial image. The first and second features are related to the facial feature detection business, the third and fourth features are not related to the facial feature detection business, the first feature corresponds to the second feature, and the third feature corresponds to the fourth feature. The first feature corresponds to the second feature, which means that the second feature is the feature of the first feature after being processed by the first neural network model; similarly, the third feature corresponds to the fourth feature, which means that the fourth feature is the feature of the third feature after being processed by the first neural network model. In other words, the first feature is the same as the second feature, or the first feature is a feature obtained by performing an enhancement process on the second feature, and the third feature is a feature obtained by performing a weakening process on the fourth feature.

[0110] It should be noted that, in the embodiment of the present application, the feature related to the facial feature detection service means that the feature can be used in the facial feature detection process, that is, when executing the facial feature detection service, it is necessary to detect facial features based on the feature. Correspondingly, the feature unrelated to the facial feature detection service means that the feature is not used in the facial feature detection process, that is, the feature is an interference feature in the facial feature detection process.

[0111] For example, when the facial feature detection service is gaze point detection, the first feature and the second feature may include eye features and / or facial contour features, and the third feature and the fourth feature may include one or more of the following: light and shadow features caused by lighting (such as highlights or shadows on the face), facial features other than eyes (such as nose, mouth, etc.), biological features other than facial features (such as freckles, moles on the face, etc.), jewelry features (such as earrings, nose studs, etc.), and makeup features (such as lipstick, eyeliner, etc.). Fig. 9 A specific example is shown. Assume Fig. 9(a) in FIG. 1 is a first face image, which includes a mole 910, an earring 920a and an earring 920b, and has uneven light. After the first face image is input into the first neural network model, the second face image is input. Assume that Fig. 9 As shown in (b) in FIG. 1 , it can be seen from the image that the light and shadow features are balanced, the features of the eye features and the facial contour features are preserved, and the mole 910, the earring 920a and the earring 920b are weakened (assuming that they are weakened to an invisible state). Alternatively, the second face image may also be as follows: Fig. 9 As shown in (c), in the second face image, the feature strengths of eye features and facial contour features are enhanced, and other features are weakened (assuming they are weakened to an invisible state). In this way, when performing subsequent facial feature detection services through the second face image, interference can be reduced and the detection accuracy, precision, and recall rate can be improved.

[0112] As can be seen from the above, the above scheme can be used to obtain face images and perform adaptive equalization processing on the obtained face images to retain or even strengthen the features related to the face feature detection business and weaken the features that are not related to the face feature detection business, that is, retain or strengthen the features to be used later and weaken other noise features. For example, in the gaze point detection scenario, the image feature changes caused by different lighting conditions are balanced to reduce the fluctuation of model performance with changes in lighting; other high-frequency facial details that are not related to gaze are suppressed, and the key features of the eyes are retained, thereby reducing the possibility of the model being interfered by other irrelevant high-frequency details (such as earrings, mouth expressions, and other details that are not related to gaze).

[0113] Combine the following Fig.10 The above-mentioned facial image acquisition process and image preprocessing process are generally described. First, the first camera is enabled, and the first image is acquired using the first camera. Then basic preprocessing is performed on the first image, such as image scaling, downsampling, pixel normalization, etc. Then, face detection and key point detection are performed on the processed image to intercept the facial image and binocular image (optional). Furthermore, the intercepted facial image can be input into the adaptive equalization network model for image adaptive equalization processing to improve the original image.

[0114] It is understandable that the present application does not limit the training method of the first neural network model. In one implementation, the first neural network model is trained based on training data, the training data includes multiple sample images, the multiple sample images are obtained by augmenting the same original image with features unrelated to the facial feature detection service, and the original image includes a third facial image. For example, the first neural network model is trained based on the adversarial loss function value of the first loss function value, the second loss function value, and the third loss function value; wherein the first loss function value is obtained by comparing the multiple sample images with multiple reconstructed images, the multiple reconstructed images are generated according to multiple feature vectors, the multiple feature vectors are extracted from the training data, and the multiple feature vectors, the multiple sample images, and the multiple reconstructed images correspond one to one; the second loss function value is obtained by comparing the average vector distance between the multiple feature vectors with the target distance; the third loss function value is obtained by comparing the detection result obtained by executing the facial feature detection service according to the multiple feature vectors with the target result. For the specific model training method, please refer to the specific description in the subsequent method 200, which will not be repeated here.

[0115] It is understandable that the embodiment of the present application is described by taking the adaptive equalization processing of the first face image as an example, but the present application is not limited to this. For example, the histogram equalization processing can also be performed on the first face image to improve the image quality of the first face image, and the present application does not limit this.

[0116] S130: Execute facial feature detection service according to the second facial image.

[0117] Exemplarily, after the first face image is processed by the first neural network model to obtain the second face image, the face feature detection service is performed according to the second face image. It can be understood that the execution of the face feature detection service described in the embodiment of the present application refers to the process of collecting face images through a camera and performing feature detection. The embodiment of the present application can be applied to various scenarios for detecting face features. For example, corresponding to Figure 2 In the application scenario shown, detecting facial features based on the second facial image means: detecting whether the user's gaze point is located within the display screen of the electronic device based on the second facial image. This application does not limit the specific detection method. As an example, the electronic device can determine the left eye image and the right eye image based on the second facial image. For specific implementation methods, please refer to Figure 7 and Figure 8 Then, whether the user's gaze point is located within the display screen of the electronic device is detected based on the second face image, the left eye image, and the right eye image.

[0118] For example, the left eye image and the right eye image may be processed based on the eye-opening and eye-closing recognition model, and an eye-opening and eye-closing result may be output, where the eye-opening and eye-closing result is used to indicate whether the user is in an eye-opening state.

[0119] In one example, the eye-opening and eye-closing recognition model is a shallow classification model composed of fewer neural network layers, so only a very low amount of computation is required to achieve accurate eye-opening and eye-closing judgment. It is understandable that the eye-opening and eye-closing recognition model can determine that the user is in an eye-opening state when both eyes are open, or when only one eye is open, and this application does not limit this.

[0120] When the user is in an open-eye state, it is detected whether the user's gaze point is located within the display screen of the electronic device based on the second face image, the left eye image, and the right eye image. When the user is in a closed-eye state, it is determined that the user's gaze point is not located within the display screen of the electronic device.

[0121] Combine the following Fig.11 The specific detection process is exemplified as follows: the left eye feature and the right eye feature are extracted from the left eye image and the right eye image respectively using a human eye feature extraction model, wherein the human eye feature extraction model is a neural network model for extracting eye image features to generate high-order representations of left and right eye images.

[0122] The left eye features and the right eye features are input into the open and closed eyes recognition model to identify whether the user is currently in an open-eye state. If the user is in a closed-eye state, it is determined that the user's gaze point is not located within the display screen of the electronic device. If the user is in an open-eye state, the face recognition branch is activated: the face feature extraction model is used to extract the face features from the second face image, and the face feature extraction model is a neural network model for extracting the features of the full face image to generate a high-order representation of the full face image. Then, the left eye features, the right eye features, and the face features are feature fused to obtain fused data. Finally, the gaze recognition model is used to process the fused data and output the gaze result, which is used to indicate whether the user's gaze point is located within the display screen of the electronic device, wherein the gaze recognition model is a shallow classification model composed of fewer neural network layers, which can achieve high-precision judgment of gaze and non-gaze.

[0123] Therefore, in the above scheme, gaze recognition is not performed directly, but first the left and right eye images are used to identify whether the user is in an open-eye state. If so, the gaze recognition process is continued. Since the training difficulty of the open-closed-eye recognition model is much lower than that of the gaze recognition model, its accuracy is very high, so accurate open-closed-eye filtering in advance helps to improve the overall accuracy of gaze recognition; secondly, compared with the deeper neural network structure and more parameters of the facial feature extraction model, the very shallow open-closed-eye recognition model requires very little calculation. By reducing meaningless full-face branch operations, the computing power consumption can be further reduced, which helps to achieve low power consumption requirements.

[0124] It is understandable that in another implementation method, it is also possible not to perform open or closed eye recognition, but to directly fuse the left eye features, right eye features, and facial features to obtain fused data, and then perform gaze recognition based on the fused data. This application does not limit this.

[0125] Optionally, after detecting whether the user's gaze point is within the display screen, the on / off state of the display screen of the electronic device can be controlled according to the detection result. The on / off state of the display screen described in the embodiment of the present application is used to indicate whether the display screen of the electronic device is currently in the on screen state or the off screen state. For example, corresponding to Figure 2 In the application scenario shown, the detection result can be fed back to the automatic screen off control program to control the on and off state of the display screen. Specifically, when the detection result indicates that the user's gaze point is within the display screen of the electronic device, the relevant control program can control the electronic device to maintain the screen on state and reset the screen off countdown.

[0126] In summary, the above-mentioned scheme provided in the embodiment of the present application can perform adaptive equalization processing on the captured facial image to retain or strengthen the features related to the facial feature detection service, and weaken the features that are not related to the facial feature detection service, thereby improving the detection accuracy.

[0127] However, if the first camera is a color camera, when the first camera is used to capture color images and perform facial feature detection, the detection effect may be affected by light and other issues. For example, in an extremely dark environment, or in a backlit scene with the back facing the light source, or in a scene with reflections from glasses, the facial features in the facial image captured by the first camera may not be clear, thereby affecting the detection effect. Fig.12 A specific example scenario is given. See Fig.12, the electronic device has a front camera 1210, which is a two-dimensional camera, that is, the front camera 1210 is a camera for shooting two-dimensional color images, such as an RGB camera. When the user lies on the bed and uses the mobile phone without turning on the light at night, if the front camera is called to shoot the face image, the display effect of the captured face image may be relatively poor, so it may not be possible to successfully detect the user's gaze point based on the face image. If the first camera is a Tof camera, when the first camera is used to collect infrared images and perform face feature detection services, the detection effect is generally better, because the infrared image is emitted by the infrared light transmitter. The modulated infrared light pulses are continuously hit on the surface of the object, and are received by the receiver after reflection. The time difference is calculated by the change of the phase, and then the depth information of the object is calculated in combination with the speed of light. Generally, the photo effect is not affected by the ambient light. Therefore, even in extremely dark or uneven light scenes, the infrared image captured by the electronic device can also have a clearer user face image. However, the relevant algorithms for collecting infrared images run at the hardware abstraction layer (HAL) layer, and the power consumption is relatively high.

[0128] In view of this, an embodiment of the present application provides an electronic device with two cameras, wherein one camera is used to obtain color images, such as an RGB camera, and the other camera is used to obtain infrared images. For example, the camera can be used to capture images containing depth information or a three-dimensional camera, such as a Tof camera. For example, Fig.13 The mobile phone in the embodiment has a front camera module including two cameras, wherein the camera 1310 is an RGB camera and the camera 1320 is a Tof camera. The two cameras can detect facial features under different ambient light conditions. For example, when the light intensity is higher than a preset value, the RGB camera is used to capture color images; when the light intensity is lower than a preset value, the Tof camera is used to capture infrared images. If the credibility of the detection result is lower than a threshold when the color image is used to detect the facial features, the Tof camera can be used to capture the infrared image.

[0129] The following is explained in conjunction with the above method 100: Assume that the first image described in method 100 is a color image, that is, the first camera is a camera for shooting color images. After determining the second face image based on the first image and performing the face feature detection service according to the second face image, when the credibility of the detection result of the face feature detection service is lower than the second threshold or the face feature detection service fails to be executed, the average pixel value of the first face image is obtained. When the average pixel value of the first face image is lower than the third threshold, the second image is obtained by shooting with the second camera, and the second image includes the third face image. When the average pixel value of the first face image is higher than the third threshold, the first camera can be used to re-capture the face image and re-perform the face feature detection service, or the face feature detection service can be directly determined to fail to be executed, or the above-mentioned low-credibility detection result can also be used, which can be configured based on actual needs.

[0130] Specifically, when the average pixel value of the first facial image is higher than the third threshold, the probability of detection failure caused by quality problems of the first facial image is less than a preset value. That is to say, if the average pixel value of the first facial image is higher than the third threshold, it means that the image quality of the first facial image is relatively good. Therefore, it is highly likely that facial feature detection will not fail due to quality problems. In other words, if the average pixel value of the first facial image is higher than the third threshold, but the facial feature detection service using the first facial image still fails, the reason for the failure is generally not due to the poor quality of the first facial image.

[0131] Accordingly, when the average pixel value of the first face image is lower than the third threshold, the probability of detection failure due to the quality problem of the first face image is greater than the preset value. In other words, if the average pixel value of the first face image is lower than the third threshold, it means that the image quality of the first face image is relatively poor, so there is a high probability that the facial feature detection will fail due to quality problems. In other words, if the average pixel value of the first face image is lower than the third threshold, and the facial feature detection using the first face image fails, the poor image quality is likely to be one of the reasons for the detection failure. Therefore, in this case, the second camera is called to retake the second image, which includes the third face image and is an infrared image, and the imaging quality can be unaffected by ambient light. Furthermore, a facial feature detection service can be performed based on the third facial image: the third facial image is processed by the second neural network model to output a fourth facial image; the facial feature detection service is performed based on the fourth facial image; wherein the feature strength of the fifth feature in the fourth facial image is greater than or equal to the feature strength of the sixth feature in the third facial image, the feature strength of the seventh feature in the fourth facial image is less than the feature strength of the eighth feature in the third facial image, the fifth feature corresponds to the sixth feature, the seventh feature corresponds to the eighth feature, the fifth feature and the sixth feature are related to the facial feature detection service, the seventh feature and the eighth feature are not related to the facial feature detection service, the first image is a color image, and the second image is an infrared image.

[0132] It is understandable that the execution process of the above-mentioned facial feature detection service is similar to the execution process of S120 and S130, and for the sake of brevity, it will not be repeated here. However, it should be noted that the second neural network model is a model trained in the same way as the first neural network model, but the training data of the second neural network is an infrared image set, and the training data of the first neural network is a color image set. The specific implementation method can refer to the description of the subsequent method 200 part, and is not specifically limited here.

[0133] It can also be understood that the average pixel value of the above-mentioned first face image can be calculated in advance by a face detection model or other models. In this case, the electronic device can directly obtain the pre-generated average pixel value of the first face image after the credibility of the detection result is lower than the second threshold or the facial feature detection service fails to execute, thereby reducing the delay; alternatively, the electronic device can also calculate the average pixel value of the first face image after the credibility of the detection result is lower than the second threshold or the facial feature detection service fails to execute, and the present application does not limit this.

[0134] It can also be understood that the embodiment of the present application only uses the average pixel value of the first face image to characterize the image quality of the first face image as an example for illustration, but the present application is not limited to this. That is to say, other parameters used to characterize the image quality can also be calculated to decide whether to start the second camera, such as calculating the parameters used to characterize the image quality based on the brightness, contrast, clarity and other parameters of the first face image. The specific method is not limited in this application.

[0135] In summary, in the facial feature detection method provided by the embodiment of the present application, if the facial feature detection fails according to the first facial image in the first image captured by the first camera, and the reason for the failure may be that the average pixel value of the first facial image is lower than the third threshold value, the second camera is used to capture the second image, and the facial feature is detected according to the third facial image in the second image. That is to say, in the scheme of the present application, after the facial feature detection fails through the first facial image, it will not be directly judged that the detection fails, but it is necessary to first judge that the average pixel value of the first facial image is less than or equal to the third threshold value. If so, it means that the detection failure is likely to be caused by the image quality problem of the first image. For example, in a backlit scene, the imaging quality of the color image will be relatively poor. At this time, switching to the second camera to capture the infrared image can improve the detection success rate, because the imaging quality of the infrared image is almost unaffected by the light problem. Therefore, in the above scheme, the first camera can be used first to capture images and perform facial feature detection, thereby minimizing the additional power consumption. If the detection fails and the reason for the failure may be an image quality problem, the second camera is used to capture images for facial feature detection, thereby improving the detection success rate.

[0136] In addition, after the first face image is acquired by the first camera, the first face image is preprocessed using the first neural network model before the face feature detection service is executed. Since the preprocessing process can make targeted improvements to the image quality for the face feature detection service, the quality of the color image under non-ideal lighting is enhanced to a certain extent (or, the performance of the color camera under non-ideal lighting is enhanced), which is equivalent to reducing the shooting of infrared images (or, reducing the calls to the infrared camera), thereby indirectly reducing unnecessary power consumption.

[0137] The embodiments of the present application can be applied to various scenarios of detecting facial features. After successfully executing the facial feature detection service, subsequent operations can be performed based on the detection results.

[0138] For example, corresponding to Figure 4 In the scenario shown, performing the facial feature detection service refers to the process of face recognition. If the detection is successful, the electronic device will automatically change from the unlocked state (such as Figure 4 (a) in the figure) to the unlocked state (as shown in Figure 4 , as shown in (b) in the figure).

[0139] For example, corresponding to Figure 2 In the scenario shown, executing the facial feature detection service refers to detecting whether the user's gaze point is located within the display screen of the electronic device. If the detection is successful, the electronic device controls the on and off state of the display screen of the electronic device according to the detection result. For example, if the user's gaze point is within the display screen of the electronic device, the screen of the electronic device is maintained in the on state, and the screen-off countdown is reset; if the user's gaze point is outside the display screen of the electronic device, the screen-off countdown continues, and after the countdown expires, it automatically switches from the on state to the off state. In order to more clearly describe the application method of the solution provided in the embodiment of the present application in an actual scenario, an electronic device is introduced below in conjunction with this scenario. The electronic device includes a module corresponding to each of the above-mentioned method embodiments. The module can be software, hardware, or a combination of software and hardware. Fig.14 As shown, the electronic device includes a system excitation module, an RGB image acquisition and preprocessing module, an RGB image gaze recognition module, an IR image acquisition and preprocessing module, and an IR image gaze recognition module. Among them, the system excitation module is used to execute the screen off countdown, and execute the decision to enable the Tof camera; the RGB image acquisition and preprocessing module is used to enable the RGB camera to shoot RGB images, and perform facial key point detection on the captured RGB images, and intercept the face and binocular images and image adaptive equalization processing; the RGB image acquisition and preprocessing module is used to perform eye opening and closing and gaze detection, and gaze recognition decision based on the face image. The IR image acquisition and preprocessing module is used to enable the IR camera to shoot IR images, and perform facial key point detection on the captured IR images, and intercept the face and binocular images and image adaptive equalization processing; the IR image acquisition and preprocessing module is used to perform eye opening and closing and gaze detection, and gaze recognition decision based on the face image. The following is combined with Fig.15 The method flow chart shown is an exemplary description of the execution process of each module in the above-mentioned gaze recognition scenario.

[0140] S1. Determine whether the screen off countdown is less than t.

[0141] Exemplarily, the electronic device can start the screen-off countdown after the user performs an operation. When the screen-off countdown is less than t, the system excitation module activates the system, that is, triggers the execution of step S2.

[0142] It is understandable that t here can be a pre-configured threshold value for stimulating the entire feature of watching the screen without turning off. It is also understandable that the t value and other subsequent threshold values ​​can be determined by sufficient experiments of specific electronic devices in different scenarios. This solution does not impose specific restrictions on this and will not be repeated in the following.

[0143] S2. Obtain the ambient light intensity.

[0144] S3. Determine whether the ambient light smoothing value is less than p.

[0145] Exemplarily, when the screen-off countdown is less than t, the system excitation module triggers the electronic device to use the ambient light sensor to obtain the current ambient light intensity. For convenience, the embodiment of the present application uses the ambient light smoothing value (ambient light sliding average) to characterize the ambient light intensity. Further, it is determined whether the ambient light smoothness is less than p, where the p value corresponds to a light intensity value reflecting dark light collected by the ambient light sensor, which means that when the ambient light intensity is less than p, the RGB camera can hardly collect images that can be used for subsequent algorithm analysis. Therefore, when the ambient light smoothing value is less than p, the Tof camera is triggered to obtain infrared IR images (i.e., triggering steps S4 to S12); when the ambient light smoothing value is greater than or equal to p, the RGB camera is triggered to obtain RGB images (i.e., triggering steps S14 to S22). The following will first be explained using steps S4 to S12 as an example.

[0146] S4. Enable Tof camera.

[0147] S5. Acquire an IR image.

[0148] Exemplarily, if the ambient light smoothing value is less than p, it indicates that the current scene is dark light, and the Tof related process is directly entered. Specifically, the IR image acquisition and preprocessing module enables the Tof camera and uses the Tof camera to shoot the IR image.

[0149] S6. IR face detection.

[0150] S7: Determine whether a face is detected.

[0151] Exemplarily, after acquiring the IR image, face detection can be performed based on the IR image. For example, the IR image acquisition and preprocessing module inputs the IR image into an IR face detection model to detect whether the IR image includes a face image. The IR face detection model is a neural network model trained using an IR face data set. If no face is detected, the process is restarted. If a face is detected, step S8 is continued.

[0152] S8, capturing the face image + the binocular image.

[0153] For example, after the IR image acquisition and preprocessing module detects a face from the IR image, it captures a face image and a binocular image from the IR image. For example, the specific implementation method can refer to Figure 7 and Figure 8 The corresponding methods will not be described in detail here.

[0154] S9, IR image adaptive equalization.

[0155] Exemplarily, the IR image acquisition and preprocessing module performs adaptive equalization processing on the captured face image (optionally, it can also perform adaptive equalization processing on the captured binocular image). The specific implementation method can be referred to the description of step S120 in method 100, which will not be repeated here.

[0156] S10. IR gaze recognition.

[0157] Exemplarily, after the IR image acquisition and preprocessing module performs adaptive equalization processing on the face image, the processed image is output to the IR image gaze recognition module, and the IR image gaze recognition module performs gaze recognition based on the processed image. The specific process can be referred to Fig.11 The corresponding methods will not be described in detail here.

[0158] S11, determining whether the recognition credibility is lower than a credibility threshold.

[0159] Exemplarily, after completing the gaze recognition, the electronic device can determine whether the recognition credibility is lower than the credibility threshold. The credibility in the embodiment of the present application is a value used to measure the recognition credibility, and the specific calculation method is not limited in this application. As an example, the recognition credibility can be calculated based on the current light intensity, the clarity of the IR image, the brightness of the IR image, the contrast of the IR image, etc.

[0160] If the recognition reliability is lower than the reliability threshold, the current process is reset. If the recognition reliability is greater than or equal to the reliability threshold, S12 is further executed.

[0161] It can be understood that when this process is combined with the above-mentioned method 100, the credibility threshold may correspond to the second threshold in method 100.

[0162] S12. Determine whether the human eye is looking at the screen.

[0163] Exemplarily, the IR image gaze recognition module can determine whether the human eye is gazing at the screen (i.e., whether the user's gaze point is located within the display screen) according to the gaze recognition result. If not, the current process is reset. If yes, S13 is continued.

[0164] S13. Keep the screen on; reset the screen off countdown; turn off the camera.

[0165] For example, if the user is currently looking at the screen, the electronic device keeps the screen on and resets the screen off countdown, and turns off the IR camera.

[0166] It can be understood that S14~S22 are similar to S4~S12. The main difference is that in S14~S22, the RGB camera is enabled to collect RGB images, and face detection, image capture, adaptive equalization processing, etc. are performed based on the RGB images. The specific process will not be repeated here.

[0167] In addition, when the RGB image is used for gaze recognition, if it is determined in S21 that the recognition reliability is lower than the reliability threshold, S23 is continued to be executed.

[0168] S23: Determine whether the average pixel intensity of the face image is lower than a pixel threshold.

[0169] Exemplarily, after the RGB image gaze recognition module completes gaze recognition, if the recognition credibility is lower than the credibility threshold, the average pixel intensity of the face image is calculated. If the average pixel intensity of the face image is lower than the pixel threshold, it means that the image quality of the captured RGB image may be relatively poor. At this time, the Tof camera can be enabled to re-shoot the IR face image and perform gaze recognition, that is, execute steps S4 to S12, which will not be repeated here.

[0170] It can be understood that when this process is combined with the above-mentioned method 100, the pixel threshold may correspond to the third threshold in the method 100.

[0171] In summary, based on Fig.15 The process shown can determine the camera used to collect face images based on factors such as ambient light intensity and the credibility of gaze recognition, which can minimize power consumption and ensure high recognition accuracy. At the same time, adaptive equalization processing of face images before gaze recognition can further improve the accuracy of gaze recognition.

[0172] The embodiments of the present application also provide a training system for a neural network model, which can be used to train any model involved in the above embodiments of the present application, and subsequently to train an adaptive equalization network model (such as the first neural network model or the second neural network model in method 100).

[0173] It should be noted that training a model in machine learning means learning (determining) the ideal values ​​of all weights and biases through labeled samples. What a machine learning algorithm does during training is to examine multiple samples and try to find a model that minimizes loss; the goal is to minimize loss.

[0174] In the general process of general model training, (1) Model (prediction function): takes one or more features as input and returns a prediction (y') as output. For simplification, consider a model that takes a feature x as input and returns a prediction, as follows (where b is bias and w is weight): y' = b + w1x1. (2) Calculate loss: Calculate the loss under the parameters (bias, weight) through the loss function. (3) Calculate parameter updates: Detect the value of the loss function and generate new values ​​for parameters such as bias and weight to minimize the loss.

[0175] The neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial model during the training process, so that the reconstruction error loss of the model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0176] Fig.16 A schematic diagram of a system architecture 1600 is shown. Fig.16 In the method for training a neural network according to an embodiment of the present application, the training data may include multiple sample images, and the multiple sample images are obtained by augmenting the same original image with features that are not related to the facial feature detection service, and the original image includes a facial image. However, it should be noted that the training data used for training different models are different. For example, when the first image is a color image, when training the first neural network model described in method 100, the sample images in the training data are color images; when the second image is an infrared image, when training the second neural network model described in method 100, the sample images in the training data are infrared images.

[0177] After acquiring the training data, the data acquisition device 1660 stores the training data in the database 1630 , and the training device 1620 obtains the target model / rule 1601 through training based on the training data maintained in the database 1630 .

[0178] The specific manner in which the training device 1620 obtains the target model / rule 1601 based on the training data will be described in detail in the method 200 below, which will not be described in detail here. The target model / rule 1601 can be used to perform adaptive equalization processing on the face image, that is, to enhance the features related to the face feature detection business and weaken the features unrelated to the face feature detection business. That is, the image to be processed is input into the target model / rule 1601, and a face image that is targeted for the face feature detection business can be obtained.

[0179] It should be noted that, in actual applications, the training data maintained in the database 1630 may not all come from the data acquisition device 1660, but may also be received from other devices. It should also be noted that the training device 1620 may not train the target model / rule 1601 entirely based on the training data maintained by the database 1630, but may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0180] The target model / rule 1601 obtained by training the training device 1620 can be applied to different systems or devices, such as Fig.16 The execution device 1610 shown in the figure may be, for example, the electronic device described in method 100, or may also be a server or a cloud. Fig.16 In the embodiment of the present application, the execution device 1610 is configured with an input / output (I / O) interface 1612 for data interaction with an external device. A user may input data to the I / O interface 1612 through a client device 1640. The input data may include a facial image to be processed, such as the first facial image or the third facial image in method 100.

[0181] When the execution device 1610 preprocesses the input data, or when the computing module 1611 of the execution device 1610 performs calculations and other related processing, the execution device 1610 can call the data, code, etc. in the data storage system 1650 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 1650.

[0182] Finally, the I / O interface 1612 returns the processing result, such as the second face image obtained after processing the first face image, to the client device 1640, thereby providing it to the user. It can be understood that the client device 1640 here can be the same device as the execution device 1610, such as both are electronic devices, and the I / O interface 1612 may not be required in this case.

[0183] It is worth noting that the training device 1620 can generate corresponding target models / rules 1601 based on different training data for different goals or different tasks, and the corresponding target models / rules 1601 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0184] It is worth noting that Fig.16 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Fig.16 In the embodiment, the data storage system 1650 is an external memory relative to the execution device 1610. In other cases, the data storage system 1650 can also be placed in the execution device 1610.

[0185] Combine the following Fig.17 The method 200 in the embodiment of the present application is used as an example to illustrate the neural network model training method provided in the embodiment of the present application. The method 200 can be applied to Fig.16 In the illustrated system architecture 1600, the method 200 may be used to train the first neural network model or the second neural network model in the method 100. The method 200 is described in detail below in conjunction with specific steps.

[0186] S210, obtaining training data.

[0187] For example, first, training data is obtained, which is used to train model parameters. Fig.18 , training data can be obtained by augmenting the original image. For example, the same original image can be augmented with features unrelated to the facial feature detection service to obtain multiple sample images, which are the training data. The original image includes a facial image. When the facial detection service is gaze point recognition, the augmented features can be, for example, various lighting, irrelevant accessories, makeup, facial defects, high-frequency signal interference, etc.

[0188] S220, extracting multiple feature vectors from the training data.

[0189] For example, after obtaining training data by performing data augmentation on the original image, the training data can be input into the neural network structure to extract feature vectors. Fig.18 , multiple feature vectors can be extracted from multiple sample images in the training data through the encoder in the neural network structure, and these multiple feature vectors correspond one-to-one to these multiple sample images.

[0190] S230, generating multiple reconstructed images according to the multiple feature vectors.

[0191] Exemplarily, after obtaining multiple feature vectors, multiple reconstructed images are generated according to the multiple feature vectors. Fig.18 , multiple feature vectors are input into the decoder in the neural network structure to regenerate multiple face images, i.e., multiple reconstructed images, which correspond one-to-one to the multiple feature vectors. Therefore, the multiple reconstructed images correspond one-to-one to the multiple sample images. It should be noted that in the process of generating the reconstructed image, the reconstructed image should be close to the sample image corresponding to the reconstructed image.

[0192] S240, performing facial feature detection according to the multiple feature vectors to obtain a detection result.

[0193] Exemplarily, multiple feature vectors may be input into a facial feature detection model to perform facial feature detection services. The facial feature detection model may be, for example, a gaze recognition model. In this case, the detection result is used to indicate whether the user's gaze point is within the display screen of the electronic device. The specific implementation process may refer to the aforementioned embodiment, which will not be described in detail here.

[0194] S250, respectively calculate a first loss function value, a second loss function value, and a third loss function value.

[0195] Before introducing the solution corresponding to this step, let's first introduce the loss function: In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually a process of optimization before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible. Generally, the smaller the loss, the higher the training quality of the deep neural network, and the larger the loss, the lower the training quality of the deep neural network. Similarly, the smaller the loss fluctuation, the more stable the training; the larger the loss fluctuation, the more unstable the training.

[0196] The following is an exemplary description of the calculation methods of the three loss function values ​​in the embodiments of the present application.

[0197] Exemplarily, a plurality of sample images are compared with the corresponding reconstructed images to obtain a first loss function value (corresponding to Fig.18 The first loss function value is used to characterize the difference between the sample image (i.e., the input image of the encoder) and the reconstructed image corresponding to the sample image (i.e., the output image of the decoder). The smaller the first loss function value is, the more similar the sample image is to the reconstructed image.

[0198] Exemplarily, the average vector distance between multiple feature vectors is compared with the target distance to obtain the second loss function value (corresponding to Fig.18 The second loss function value is used to characterize the difference between different feature vectors. The smaller the second loss function value is, the smaller the vector distance between sample images from the same original image is.

[0199] Exemplarily, the detection result is compared with the target result to obtain the third loss function value (corresponding to Fig.18 The third loss function value is used to characterize the difference between the actual detection result and the target detection result. The smaller the third loss function value is, the closer the detection result is to the actual result, and the feature vector can better characterize the features related to the face feature detection service.

[0200] S260, adjusting the parameters of the neural network model according to the first loss function value, the second loss function value, and the third loss function value.

[0201] As an example, the parameters of the neural network model are adjusted with the purpose of reducing the adversarial loss function value, the second loss function value, and the third loss function value of the first loss function value.

[0202] Among them, the adversarial loss function refers to a loss function in generative adversarial networks (GAN). GAN is a machine learning architecture that mainly includes two types of networks, a generator G (generator) and a discriminator D (discriminator). G is responsible for generating images, that is, after inputting a random code (random code) z, it will output a fake image G (z) automatically generated by the neural network. D is responsible for accepting the image output by G as input, and then judging whether the image is true or false. The real output is 1, and the fake output is 0. For example, GAN can be simply regarded as a game process between two networks. D trains a binary classification neural network with data from real and fake images. G can fabricate a "fake image" based on a string of random numbers, and then G uses the fabricated fake image to deceive D. D is responsible for distinguishing whether it is a real image or a fake image and giving a score.

[0203] As an example, the adversarial loss function value of the first loss function value is equal to the difference between 1 and the first loss function value. That is, the larger the first loss function value is, the smaller the adversarial loss function value of the first loss function value is.

[0204] Thus, the first loss function value is used to train the parameters of the decoder, while the other three loss function values ​​are used to train the encoder. The first loss function value is used for the feature vector to retain all the information of the input image to achieve reconstruction; the third loss function value is used to ensure that the features related to the face feature detection business are well represented, and the combination of the second loss function value and the adversarial loss function value of the first loss function value forms adversarial learning with the first loss function value, which prevents the retention of all features and tends to remove features that are not related to the face feature detection business.

[0205] Finally, after the above self-supervised training and adversarial training, the encoder and decoder constitute an adaptive equalization network model. The image reconstructed by this model can retain the features related to the facial feature detection business.

[0206] Corresponding to the facial feature detection method provided in the above method embodiments, the present application embodiment also provides a corresponding facial feature detection system, which is deployed in an electronic device and is used to execute the method provided in the above method embodiments. Therefore, the contents not described in detail can be referred to the above method embodiments, and for the sake of brevity, they will not be repeated here.

[0207] Fig.19 The present invention provides a system corresponding to an electronic device provided in an embodiment of the present invention. The system may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture, etc. The present invention takes the Android system of a layered architecture as an example to exemplify the software system of the electronic device.

[0208] See also Fig.19 The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided from top to bottom into the application layer (also called the application layer), the application framework layer (also called the application framework layer), the hardware abstraction layer (HAL), and the driver layer. In addition to the system, it also includes a hardware layer.

[0209] The application layer may include multiple applications, such as a dialing application, a gallery application, etc. In the embodiment of the present application, the application layer also includes a face feature detection toolkit (software development kit, SDK), such as a gaze recognition SDK. The system of the electronic device and the third application installed on the electronic device can detect face features by calling the face feature detection SDK, such as identifying the position of the user's gaze point.

[0210] The framework layer provides an application programming interface (API) and a programming framework for the application program of the application layer. The framework layer includes some predefined functions. In an embodiment of the present application, the framework layer may include a camera service interface, a facial feature detection service interface, and an image pixel analysis service interface. The camera service interface is used to provide an application programming interface and a programming framework for using a camera. The facial feature detection service interface provides an application programming interface and a programming framework for using a facial feature detection model (such as a gaze recognition model) and an adaptive equalization model (such as the first neural network model in method 100). The image pixel analysis service interface provides an application programming interface and a programming framework for an algorithm for calculating the average pixel intensity of a facial image. It can be understood that the facial feature detection service interface and the image pixel analysis service interface can be two interfaces or combined into one interface, which is not limited in the present application. For the convenience of description, the embodiment of the present application is described as an example in which the facial feature detection service interface and the image pixel analysis service interface are two different interfaces.

[0211] The hardware abstraction layer is an interface layer located between the framework layer and the driver layer, which provides a virtual hardware platform for the operating system. In an embodiment of the present application, the hardware abstraction layer may include a camera hardware abstraction layer and a facial feature detection process. The camera hardware abstraction layer may provide virtual hardware for one or more camera devices. Here, the electronic device in the embodiment of the present application includes two camera devices as an example for illustration, such as camera device 1 (such as an RGB camera) and camera device 2 (such as a Tof camera). The calculation process of detecting facial features through a facial feature detection model is executed in the facial feature detection process, such as the calculation process of identifying the position of the user's gaze point through a gaze recognition module is executed in the facial feature detection model.

[0212] The driver layer is a layer between hardware and software. The driver layer includes drivers for various hardware. The driver layer may include a camera device driver. The camera device driver is used to drive the camera sensor to collect images and drive the image signal processor to pre-process the image.

[0213] The hardware layer includes a sensor and optionally a data buffer. The sensor includes a first camera (such as an RGB camera) and a second camera (such as a Tof camera). The functions of the first camera and the second camera can be found in the description of the above method embodiment, which will not be repeated here. Optionally, the sensor may also include a light sensor, which is used to detect the ambient light intensity.

[0214] Optionally, the data collected by the camera can be stored in a data buffer. When the upper-layer process or reference obtains the image data collected by the camera, it can obtain it from the data buffer.

[0215] In combination with the above system structure, the interactive process of the face feature detection method in the embodiment of the present application is briefly introduced below. For the contents not described in detail, please refer to the above method embodiment.

[0216] The electronic device determines the facial features detected (such as Figure 2 In the scenario shown, when the screen off countdown is less than a preset value, it is determined to detect facial features, and the facial feature detection service is called through the facial feature detection SDK.

[0217] On the one hand, the facial feature detection service can call the camera service of the framework layer, and collect and obtain image frames containing the user's facial image through the camera service. Specifically, the camera service can send an instruction to start the first camera by calling the camera device 1 (first camera) in the camera hardware abstraction layer. The camera hardware abstraction layer sends the instruction to the camera device driver of the driver layer. The camera device driver can start the first camera according to the above instruction. The instruction sent by the camera device 1 to the camera device driver can be used to start the first camera. After the first camera is turned on, it collects light signals and generates a color image of the electrical signal through the image signal processor.

[0218] On the other hand, the facial feature detection service can create a facial feature detection process and initialize facial feature detection models (such as facial image acquisition model, gaze recognition model, and adaptive equalization model).

[0219] The color image generated by the image signal processor can be stored in the data buffer. After the facial feature detection process is created and initialized, the color image data stored in the data buffer can be input into the facial feature detection process. In the facial feature detection process, the input image can be processed using an adaptive equalization model, and then the processed image can be detected using a facial feature detection model (such as a gaze recognition model) to perform facial feature detection (such as detecting whether the user's gaze point is within the display screen of the electronic device). The detection results can then be returned to the application layer facial feature detection SDK via the camera service and the facial feature detection service.

[0220] On the other hand, if the credibility of the detection result of the facial feature detection is lower than the preset value, the image pixel analysis service can obtain the color image from the data buffer and determine whether the average pixel intensity of the color image is lower than the preset value. If so, the image pixel analysis service calls the camera service of the framework layer through the facial feature detection service, and re-collects and obtains the image frame containing the user's facial image through the camera service. Specifically, the camera service can send an instruction to start the second camera by calling the camera device 2 (such as the Tof camera) in the camera hardware abstraction layer. The camera hardware abstraction layer sends the instruction to the camera device driver of the driver layer. The camera device driver can start the second camera based on the above instruction. The instruction sent by the camera device 2 to the camera device driver can be used to start the second camera. After the second camera is turned on, it collects light signals and generates infrared images of electrical signals through the image signal processor.

[0221] The infrared image generated by the image signal processor can be stored in the data buffer. After the facial feature detection process is created and initialized, the infrared image data stored in the data buffer can be input into the facial feature detection process. In the facial feature detection process, the infrared image can be processed in sequence using an adaptive equalization model and a facial feature detection model (such as a gaze recognition model) to determine whether the user's gaze point is within the display screen. The recognition result can then be returned to the application layer facial feature detection SDK via the camera service and the facial feature detection service.

[0222] It should be noted that the embodiments of the present application are only illustrated by taking the Android system as an example. In other operating systems (such as the IOS system, etc.), the solutions of the present application can also be implemented as long as the functions implemented by each functional module are similar to those of the embodiments of the present application.

[0223] Corresponding to the methods provided in the above-mentioned method embodiments, the embodiments of the present application also provide a hardware architecture of a corresponding electronic device.

[0224] For example, Fig. 20 A detailed architecture diagram of an electronic device 1000 to which the present application is applicable is shown.

[0225] like Fig. 20 As shown, the electronic device 1000 may include a processor 1010, an internal memory 1021, one or more cameras 1093 (these multiple display screens may be represented by 1 to N, where N is a positive integer greater than 1), one or more display screens 1094 (these multiple display screens may be represented by 1 to N), and a sensor module 1080, wherein the sensor module includes an ambient light sensor 1080L.

[0226] Optionally, the electronic device 1000 may also include an external memory interface 1020, a universal serial bus (USB) interface 1030, a charging management module 1040, a power management module 1041, a battery 1042, an audio module 1070, a speaker 1070A, a receiver 1070B, a microphone 1070C, an earphone interface 1070D, a sensor module 1080, a button 1090, a motor 1091, an indicator 1092, and a subscriber identification module (SIM) card interface 1095, etc. In addition to the ambient light sensor, the sensor module 1080 may also include a pressure sensor 1080A, a gyroscope sensor 1080B, an air pressure sensor 1080C, a magnetic sensor 1080D, an acceleration sensor 1080E, a distance sensor 1080F, a proximity light sensor 1080G, a fingerprint sensor 1080H, a temperature sensor 1080J, a touch sensor 1080K, a bone sensor 1080M, and the like.

[0227] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 1000. In other embodiments of the present application, the electronic device 1000 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0228] Processor 1010 may include one or more processing units. For example, processor 1010 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc.

[0229] The processor 1010 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 1010 is a cache memory. The memory may store instructions or data that the processor 1010 has just used or cyclically used. If the processor 1010 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 1010, and thus improves the efficiency of the system.

[0230] The internal memory 1021 can be used to store computer executable program codes, which include instructions. The processor 1010 executes various functional applications and data processing of the electronic device 1000 by running the instructions stored in the internal memory 1021. In an example, the internal memory 1021 stores a computer program, and when the processor calls the computer program to run, the electronic device 1000 can implement the steps of any method embodiment in the above-mentioned various embodiments.

[0231] The electronic device 1000 implements the display function through a GPU, a display screen 1094, and an application processor.

[0232] Camera 1093 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In an embodiment of the present application, the electronic device 1000 may include two cameras, one of which can be used to capture color images, such as an RGB camera, and the other camera can be used to capture infrared images, such as a Tof camera. It will be understood that the electronic device 1000 may also include more than two cameras.

[0233] The display screen 1094 is used to display images, videos, etc. The display screen 1094 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 1000 may include 1 or N display screens 1094, where N is a positive integer greater than 1.

[0234] The ambient light sensor 1080L is used to sense the brightness of the ambient light. The electronic device 1000 can adaptively adjust the brightness of the display screen 1094 according to the perceived brightness of the ambient light. The ambient light sensor 1080L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 1080L can also cooperate with the proximity light sensor 1080G to detect whether the electronic device 1000 is in a pocket to prevent accidental touches.

[0235] In the embodiment of the present application, the electronic device being in the screen-on state means that the display panel of the display screen 1094 is in the lighted state, and the electronic device being in the screen-off state means that the display panel of the display screen 1094 is in the unlighted state. When the electronic device is in the screen-on state, the display screen 1094 can display any interface such as the desktop, the negative one screen, the application interface of a certain application, the lock screen interface, etc.

[0236] Optionally, in one implementation, the electronic device 1000 may implement a shooting function through an ISP, a camera 1093, a video codec, a GPU, and an application processor to acquire an image including a face, so as to detect facial features based on the image.

[0237] by Figure 2 Taking the application scenario shown as an example, in the embodiment of the present application, when the screen off countdown is less than or equal to 4s, the processor 1010 calls the ambient light sensor 1080L to obtain the current ambient light intensity. When the ambient light intensity is greater than or equal to the second threshold, the RGB camera in the camera 1093 is called to shoot a color image containing a face, and the facial features are detected based on the color image, such as detecting whether the user's gaze point is located within the display screen 1094. If the reliability of the detection result is low, the processor 1010 obtains the average pixel intensity of the image. If the average pixel intensity is less than or equal to the preset value, the Tof camera in the camera 1093 is called again to re-shoot the infrared image containing the face, and the facial features are detected based on the infrared image.

[0238] It is understandable that if the integrated units in the above-mentioned device embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0239] An embodiment of the present application provides a computer program product. When the computer program product runs on the electronic device 1000, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0240] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0241] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0242] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

[0243] The terms "first", "second", "third", "fourth" and other various terminology and numbers (if any) in the specification and claims of this application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or quantity. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0244] The terms "including" and "having" and any variations thereof mean "including but not limited to" unless specifically emphasized otherwise. For example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed but may include other steps or units not explicitly listed or inherent to such process, method, product or apparatus.

[0245] In the various embodiments of the present application, if there is no special explanation and logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships. The specific operating methods in the method embodiments of the present application can also be applied to device embodiments or system embodiments.

[0246] In the present application, "pre-configuration" may include pre-definition, for example, protocol definition. Among them, "pre-definition" may be implemented by pre-saving corresponding codes, tables or other methods that can be used to indicate relevant information in a device (for example, including each network element), and the present application does not limit the specific implementation method.

[0247] In the schematic diagrams of the drawings in the specification of this application, the dashed lines, arrows or boxes represent optional steps or optional modules, and the dotted lines or boxes represent the contents of the annotations.

Claims

1. A facial feature detection method, applied to an electronic device, wherein the electronic device has a first camera, characterized in that: The method comprises: Using the first camera to capture a first image, the first image includes a first face image; Processing the first facial image based on a first neural network model and outputting a second facial image; Performing facial feature detection according to the second facial image; Among them, the feature intensity of the first feature in the second facial image is greater than or equal to the feature intensity of the second feature in the first facial image, the feature intensity of the third feature in the second facial image is less than the feature intensity of the fourth feature in the first facial image, the first feature corresponds to the second feature, the third feature corresponds to the fourth feature, the first feature and the second feature are related to the facial feature detection service, and the third feature and the fourth feature are not related to the facial feature detection service.

2. The method according to claim 1, characterized in that The performing of the facial feature detection service according to the second facial image includes: Detecting whether the user's gaze point is located within the display screen of the electronic device according to the second facial image; The method further comprises: According to the detection result of the facial feature detection service, the on / off state of the display screen of the electronic device is controlled.

3. The method according to claim 2, characterized in that The electronic device is currently in a screen-on state, and the first camera is used to capture a first image, including: When the screen-off countdown is less than or equal to a first threshold, using the first camera to capture the first image; The step of controlling the on / off state of the display screen of the electronic device according to the detection result of the facial feature detection service includes: When the detection result indicates that the user's gaze point is located within the display screen of the electronic device, the display screen of the electronic device is maintained in a lit state and the screen-off countdown is reset.

4. The method according to claim 2 or 3, characterized in that: The first feature and the second feature include eye features and / or facial contour features, and the third feature and the fourth feature include one or more of the following: light and shadow features caused by lighting, features of facial features other than eyes, biological features other than facial features, jewelry features, and makeup features.

5. The method according to claim 2 or 3, characterized in that: The detecting, according to the second facial image, whether the user's gaze point is located within the display screen of the electronic device comprises: determining a left-eye image and a right-eye image according to the second face image; Processing the left eye image and the right eye image based on an eye-opening and eye-closing recognition model, and outputting an eye-opening and eye-closing result, wherein the eye-opening and eye-closing result is used to indicate whether the user is in an eye-opening state; When the user is in an eye-open state, it is detected whether the user's gaze point is located within the display screen of the electronic device based on the second facial image, the left eye image, and the right eye image.

6. The method according to claim 5, characterized in that The detecting whether the user's gaze point is located within the display screen of the electronic device according to the second face image, the left eye image, and the right eye image comprises: Extracting left-eye features and right-eye features from the left-eye image and the right-eye image respectively using a human eye feature extraction model; extracting facial features from the second facial image using a facial feature extraction model; Fusing the left eye feature, the right eye feature, and the face feature to obtain fused data; The fused data is processed using a gaze recognition model, and a gaze result is output, where the gaze result is used to indicate whether the gaze point of the user is located within the display screen of the electronic device.

7. The method according to any one of claims 1 to 3, characterized in that The electronic device also includes a second camera; When the credibility of the detection result of the facial feature detection service is lower than a second threshold or the facial feature detection service fails to be executed, obtaining an average pixel value of the first facial image; When the average pixel value of the first facial image is lower than a third threshold, using the second camera to obtain a second image, where the second image includes a third facial image; Processing the third facial image using a second neural network model to output a fourth facial image; Execute the facial feature detection service according to the fourth facial image; Among them, the feature intensity of the fifth feature in the fourth facial image is greater than or equal to the feature intensity of the sixth feature in the third facial image, the feature intensity of the seventh feature in the fourth facial image is less than the feature intensity of the eighth feature in the third facial image, the fifth feature corresponds to the sixth feature, the seventh feature corresponds to the eighth feature, the fifth feature and the sixth feature are related to the facial feature detection service, the seventh feature and the eighth feature are not related to the facial feature detection service, the first image is a color image, and the second image is an infrared image.

8. The method according to claim 7, characterized in that The step of capturing a first image using the first camera includes: Get the current ambient light intensity; When the ambient light intensity is greater than or equal to a fourth threshold, the first camera is used to capture the first image.

9. The method according to claim 7 or 8, characterized in that: The first camera is used to acquire a two-dimensional image, and the second camera is used to acquire an image containing depth information.

10. The method according to claim 9, characterized in that The first camera is a red, green, and blue (RGB) camera, and the second camera is a time-of-flight (Tof) camera.

11. The method according to any one of claims 1 to 3, characterized in that Before processing the first facial image based on the first neural network model, the method further includes: It is determined that the first image includes the first face image.

12. The method according to any one of claims 1 to 3, characterized in that The first neural network model is trained based on training data, the training data includes multiple sample images, the multiple sample images are obtained by augmenting the same original image with features unrelated to the facial feature detection business, and the original image includes a fifth facial image.

13. The method according to claim 12, characterized in that The first neural network model is trained based on the adversarial loss function value, the second loss function value, and the third loss function value of the first loss function value; The first loss function value is obtained by comparing the multiple sample images with multiple reconstructed images, the multiple reconstructed images are generated according to multiple feature vectors, the multiple feature vectors are extracted from the training data, and the multiple feature vectors, the multiple sample images, and the multiple reconstructed images correspond one to one; The second loss function value is obtained by comparing the average vector distance between the plurality of feature vectors with the target distance; The third loss function value is obtained by comparing the detection result obtained by performing the facial feature detection service according to the multiple feature vectors with the target result.

14. A neural network model training method, characterized in that: include: Acquire training data, the training data comprising a plurality of sample images, the plurality of sample images being obtained by augmenting the same original image with features irrelevant to the facial feature detection service, the original image comprising a facial image; Extracting a plurality of feature vectors from the training data; Generating a plurality of reconstructed images according to the plurality of feature vectors, wherein the plurality of feature vectors, the plurality of sample images, and the plurality of reconstructed images correspond to each other in a one-to-one manner; Execute the facial feature detection service according to the multiple feature vectors to obtain a detection result; Comparing the plurality of sample images with the corresponding plurality of reconstructed images respectively to obtain a first loss function value; Comparing the vector distance between the plurality of feature vectors with the target distance to obtain a second loss function value; Comparing the detection result with the target result to obtain a third loss function value; Adjust the parameters of the neural network model according to the first loss function value, the second loss function value, and the third loss function value.

15. The method according to claim 14, characterized in that The adjusting the parameters of the neural network model according to the first loss function value, the second loss function value, and the third loss function value includes: Adjust the parameters of the neural network model with the purpose of reducing the adversarial loss function value of the first loss function value, the second loss function value, and the third loss function value.

16. The method according to claim 15, characterized in that The adversarial loss function value of the first loss function value is equal to the difference between 1 and the first loss function value.

17. An electronic device, characterized in that: The structure of the electronic device includes a processor and a memory; The memory is used to store a program that supports the electronic device to perform the method provided in any one of claims 1 to 13, and to store data involved in implementing the method provided in any one of claims 1 to 13; or, the memory is used to store a program that supports the electronic device to perform the method provided in any one of claims 14 to 16, and to store data involved in implementing the method provided in any one of claims 14 to 16; The processor is configured to execute the program stored in the memory.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 13, or to execute the method according to any one of claims 14 to 16.

Images