Unlocking method and device of electronic equipment

Abstract

The invention discloses an unlocking method and device of electronic equipment. The unlocking method comprises the following steps: acquiring a shot image shot by a camera of the electronic equipment; detecting the shot image to obtain the prediction confidence and the biological feature category of each biological feature image on the shot image; under the condition that at least two biological feature categories exist on the shot images, determining the biological feature image with the highest prediction confidence coefficient on the shot images as a target image; and unlocking the electronic equipment based on the target image. According to the method, after the shot image is acquired, the shot image is automatically detected, and the prediction confidence of each biological feature image on the shot image is obtained; according to the embodiment of the invention, the biological feature image with the highest prediction confidence coefficient on the shot image is determined as the target image, so that the biological features in the target image are most easily recognized relative to other biological feature images in the shot image, unlocking is carried out based on the target image which is most easily recognized on the shot image, and the convenience of equipment unlocking can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and device for unlocking an electronic device. Background Art

[0002] Due to the rapid development of mobile devices in recent years, people spend more and more time on their mobile phones. Electronic device unlocking includes technologies that use biometric features such as fingerprint unlocking and face unlocking. However, in the existing technology, most device unlocking is based on a fixed unlocking method. When there are multiple biometric features available for unlocking, it cannot automatically match the best unlocking method according to the current scene, resulting in low unlocking convenience.

[0003] That is, the unlocking convenience of the device in the prior art is low. Summary of the invention

[0004] The embodiments of the present application provide a method and device for unlocking an electronic device, which can improve the convenience of unlocking the device.

[0005] In a first aspect, the present application provides a method for unlocking an electronic device, comprising:

[0006] Acquire an image captured by a camera of the electronic device;

[0007] Detecting the captured image to obtain prediction confidence and biometric category of each biometric image on the captured image;

[0008] In the case where there are at least two biometric feature categories on the captured image, determining the biometric feature image with the highest prediction confidence on the captured image as the target image;

[0009] The electronic device is unlocked based on the target image.

[0010] In a second aspect, the present application provides an unlocking device for an electronic device, comprising:

[0011] An acquisition module, used to acquire an image captured by a camera of the electronic device;

[0012] A detection module, used to detect the captured image and obtain the prediction confidence and biometric category of each biometric image on the captured image;

[0013] a determination module, configured to determine, when there are at least two biometric feature categories on the captured image, a biometric feature image with the highest prediction confidence on the captured image as a target image;

[0014] An unlocking module is used to unlock the electronic device based on the target image.

[0015] In a third aspect, the electronic device provided in the present application includes a memory and a processor, the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the unlocking method of the electronic device provided in the present application.

[0016] In a fourth aspect, the computer-readable storage medium provided in the present application stores a plurality of instructions, which are suitable for loading by a processor to implement the steps in the unlocking method of the electronic device provided in the present application.

[0017] In a fifth aspect, the computer program product provided in the present application includes a computer program or instructions, which, when executed by a processor, implement the steps in the unlocking method of an electronic device provided in the present application.

[0018] In this application, compared with the related art, a captured image taken by a camera of an electronic device is obtained; the captured image is detected to obtain the predicted confidence and biometric category of each biometric image on the captured image; when there are at least two biometric categories on the captured image, the biometric image with the highest predicted confidence on the captured image is determined as the target image; and the electronic device is unlocked based on the target image. After acquiring the captured image, the present application automatically detects the captured image to obtain the predicted confidence and biometric category of each biometric image on the captured image; when there are at least two biometric categories on the captured image, the biometric image with the highest predicted confidence on the captured image is determined as the target image. Therefore, the biometrics in the target image are the easiest to be identified compared to other biometric images in the captured image. Unlocking based on the target image that is the easiest to identify on the captured image can improve the convenience of unlocking the device. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a schematic diagram of a scenario of an unlocking system for an electronic device provided in an embodiment of the present application;

[0021] Figure 2 It is a flowchart of an embodiment of a method for unlocking an electronic device provided in an embodiment of the present application;

[0022] Figure 3 It is a schematic diagram of TPH-yolov5 in one embodiment of the unlocking method of the electronic device provided in the embodiment of the present application;

[0023] Figure 4 It is a schematic diagram of a process of obtaining a preset biometric detection model in an embodiment of a method for unlocking an electronic device provided in an embodiment of the present application;

[0024] Figure 5 is a flowchart of another embodiment of the unlocking method of the electronic device provided in the embodiment of the present application;

[0025] Figure 6 It is a structural schematic diagram of an embodiment of an unlocking device for an electronic device provided in an embodiment of the present application;

[0026] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] It should be noted that the principles of the present application are illustrated by implementing them in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application and should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0028] In the following description of the present application, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0029] In the following description of the present application, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0031] In order to improve the efficiency of unlocking an electronic device, the embodiments of the present application provide an unlocking method of an electronic device, an unlocking device of an electronic device, an electronic device, a computer-readable storage medium, and a computer program product. The unlocking method of an electronic device can be executed by the unlocking device of the electronic device, or by an electronic device in which the unlocking device of the electronic device is integrated.

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0033] Please refer to Figure 1 , the present application also provides an unlocking system for an electronic device, such as Figure 1 As shown, the unlocking system of the electronic device includes an electronic device 100, and the electronic device 100 is integrated with the unlocking device of the electronic device provided in the present application.

[0034] Among them, the electronic device 100 can be any device equipped with a processor and has processing capabilities, such as mobile electronic devices with processors such as smart phones, tablet computers, PDAs, laptops, smart speakers, or fixed electronic devices with processors such as desktop computers, televisions, servers, industrial equipment, etc.

[0035] In addition, the unlocking system of the electronic device may further include a memory 200 for storing original data, intermediate data and result data.

[0036] In the embodiment of the present application, the memory may be a cloud memory. Cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as the storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of storage devices of various types (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0037] At present, the storage method of the storage system is: create a logical volume, and when creating a logical volume, allocate physical storage space for each logical volume. The physical storage space may be composed of disks of a storage device or several storage devices. The client stores data on a logical volume, that is, stores the data on the file system. The file system divides the data into many parts, each of which is an object. The object contains not only data but also additional information such as data identification (ID, ID entity). The file system writes each object into the physical storage space of the logical volume, and the file system records the storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.

[0038] The process of the storage system allocating physical storage space to a logical volume is as follows: based on the estimated capacity of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of independent redundant disk arrays (RAID, Redundant Array of Independent Disks), the physical storage space is pre-divided into stripes. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0039] It should be noted that Figure 1 The scenario diagram of the unlocking system of the electronic device shown is merely an example. The unlocking system and scenario of the electronic device described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the unlocking system of electronic devices and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is equally applicable to similar technical problems.

[0040] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0041] Please refer to Figure 2 , Figure 2 FIG. 1 is a flow chart of an embodiment of a method for unlocking an electronic device provided in an embodiment of the present application. Figure 2 As shown, the process of the unlocking method of the electronic device provided by this application is as follows:

[0042] 201. Obtain an image captured by a camera of an electronic device.

[0043] In a specific embodiment, when the unlock command input by the user is obtained, the captured image captured by the camera of the electronic device is obtained. The user can also input the unlock command by key, voice, etc. When the electronic device obtains the unlock command input by the user, the electronic device calls the camera to capture the captured image.

[0044] In another specific embodiment, obtaining a captured image captured by a camera of an electronic device includes: detecting the object distance between an object close to the electronic device and the electronic device; when the object distance falls within a preset distance range, controlling the camera of the electronic device to capture the captured image. The preset distance range may be set according to specific circumstances, for example, the preset range is 5 cm to 10 cm. The object may be a human body. Specifically, detecting the object distance between a human body close to the electronic device and the electronic device; when the object distance falls within the preset distance range, controlling the camera of the electronic device to capture the captured image. Of course, the object may also be an identifier set by the user.

[0045] In another specific embodiment, obtaining an image captured by a camera of an electronic device includes: detecting the object distance between an object close to the electronic device and the electronic device; when the object distance falls within a preset distance range, turning on the screen and controlling the camera of the electronic device to capture the image.

[0046] Specifically, the distance sensor of the electronic device is used to detect the object distance between the object near the electronic device and the electronic device; when the object distance is within a preset distance range, the camera of the electronic device is controlled to capture an image. Furthermore, in order to accurately measure the distance, multiple distance values ​​between the object near the electronic device and the electronic device measured by the distance sensor are obtained, and the average value of the multiple distance values ​​is determined as the object distance.

[0047] Distance sensor, also called displacement sensor, is a kind of sensor, which is used to sense the distance between it and an object to complete a preset function, and has been widely used. Distance sensor can be divided into optical distance sensor, infrared distance sensor, ultrasonic distance sensor and other types according to its working principle.

[0048] 202. Detect the captured image and obtain the prediction confidence and biometric category of each biometric image on the captured image.

[0049] In an embodiment of the present application, a captured image is detected to obtain a predicted confidence and a biometric category of each biometric image on the captured image, including: detecting the captured image based on a preset biometric detection model to obtain a predicted confidence and a biometric category of each biometric detection frame on the captured image; determining the image within each biometric detection frame as a biometric image, and obtaining a predicted confidence and a biometric category of each biometric image on each captured image.

[0050] The biometric feature category is one of a preset feature category set, and the preset feature category set includes at least one of a palm feature category, a fingerprint feature category, a face feature category, and an iris feature category.

[0051] The prediction confidence of the biometric detection frame indicates the probability that the biometric in the biometric detection frame belongs to the biometric category of the biometric detection frame. For example, there are two biometric detection frames on the captured image, one of which has a biometric category of palm feature category and a prediction confidence of palm feature category biometric detection frame is 0.8; the other biometric category of the biometric detection frame is face feature category and a prediction confidence of face feature category biometric category is 0.7.

[0052] In a specific embodiment, the captured image is detected based on a preset biometric detection model to obtain the prediction confidence and biometric category of each biometric detection frame on the captured image, including:

[0053] (1) Obtain a preset training set and a preset target detection model.

[0054] The preset training set includes sample images and various biometric feature annotation boxes annotated on the sample images. The biometric feature annotation boxes contain biometric features, and the biometric feature annotation boxes are divided into various biometric feature categories according to the biometric features therein.

[0055] Among them, the preset target detection model is TPH-yolov5, and the structure diagram of TPH-yolov5 is as follows Figure 3 Of course, in other embodiments, the preset target detection model can be yolov5, yolov4, etc., which can be set according to specific circumstances.

[0056] Figure 3 In the figure, training images refers to the training image set for training TPH-yolov5, testing images refers to the test image set for testing TPH-yolov5, and Data Augmentation refers to data augmentation. The main purpose of data augmentation is to expand the data set so that the model has higher robustness to images obtained in different environments. Specifically, Data Augmentation methods include Photometric, geometric, MixUp, and Mosaic.

[0057] Photometric mainly adjusts the hue, saturation and value of the image. When processing geometric, it mainly adds random scaling, cropping, translation, shearing and rotation. MixUp randomly selects 2 samples from the training image for random weighted summation, and the label of the sample also corresponds to the weighted summation. Unlike the usual occlusion work of using a zero-pixel mask to occlude the image, CutMix uses the area of ​​another image to cover the occluded area. Mosaic is an improved version of CutMix. Stitching 4 images greatly enriches the background of the detected object.

[0058] TPH-yolov5 includes 4 TPHs (Transformer Prediction Heads), which are used to detect tiny, small, medium and large objects respectively.

[0059] Ms-testing refers to multi-scale testing during model inference. The scale (adjusting image size) method and horizontal flip (horizontal flip) method are used in multi-scale testing. The implementation details of ms-testing are the following three steps. First, scale the test image to 1.3 times. Then reduce the image to 1 times, 0.83 times, and 0.67 times respectively. Then flip the image horizontally. Finally, we input six images of different scales into TPH-YOLOv5 and use the fusion test prediction through Model Ensemble.

[0060] The model ensemble method can be NMS (non-maximum suppression) or WBF (weighted box fusion).

[0061] Self-trained classifier is a self-trained classifier. Self-trained classifier builds a training set by cropping the true bounding box and adjusting the size of each image. Then ResNet18 is selected as the classifier network.

[0062] Ground Truth represents the real detection box, including the Localization (position) and Classification (category) of the real detection box. Prediction represents the predicted detection box, including the Localization (position) and Classification (category) of the predicted detection box.

[0063] (2) A preset target detection model is trained based on a preset training set to obtain a preset biometric detection model.

[0064] Specifically, the sample image is input into the preset target detection model to obtain various biometric prediction boxes on the sample image, and the target loss value is calculated based on the various biometric prediction boxes and various biometric annotation boxes on the sample image to determine whether the target loss value is lower than the preset loss value. If the target loss value is lower than the preset loss value, the preset target detection model at this time is determined as the preset biometric detection model; if the target loss value is not lower than the preset loss value, the weight of the preset target detection model is updated, and the sample image is re-input into the updated preset target detection model to obtain an updated target loss value. After multiple iterations of updating until the target loss value is lower than the preset loss value, the preset target detection model at this time is determined as the preset biometric detection model.

[0065] In a specific embodiment, the target loss value is calculated based on each biometric prediction box and each biometric annotation box on the sample image according to a preset loss function. Among them, the preset loss function is obtained by weighting softmaxloss and focal loss function. Softmaxloss is a kind of loss function, and softmaxloss is a loss function composed of softmax and cross-entropy loss. Focal loss is based on binary cross entropy. It is a dynamically scaled cross entropy loss. Through a dynamic scaling factor, it can dynamically reduce the weight of easy-to-distinguish samples during training, thereby quickly focusing on those difficult-to-distinguish samples. During the training process, we used softmaxloss and focal loss for joint training. After repeated verification and training, the network designed in this application has improved the accuracy by more than 5% and the speed by more than 500% compared with the conventional network.

[0066] It should be noted that artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0067] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes machine learning (ML) technology, among which deep learning (DL) is a new research direction in machine learning. It is introduced into machine learning to make it closer to the original goal, namely artificial intelligence. At present, deep learning is mainly used in the fields of machine vision and natural speech processing.

[0068] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images, and sound. Using deep learning technology and corresponding sample sets, network models that can achieve different functions can be trained.

[0069] 203. When there are at least two biometric feature categories on the captured image, determine the biometric feature image with the highest prediction confidence on the captured image as the target image.

[0070] In the embodiment of the present application, the image in the biometric detection frame with the highest prediction confidence on the captured image is cropped out to obtain the target image. For example, there are two biometric detection frames on the captured image, one of which has a biometric category of palm feature category, and the prediction confidence of the biometric detection frame of palm feature category is 0.8; the other biometric category of biometric detection frame is face feature category, and the prediction confidence of the biometric category of face feature category is 0.7. The image in the biometric detection frame of palm feature category is determined as the target image for subsequent processing.

[0071] 204. Unlock the electronic device based on the target image.

[0072] In the embodiment of the present application, after the target image is acquired, the electronic device can be unlocked based on the target image.

[0073] In a specific embodiment, unlocking the electronic device based on the target image includes: obtaining a preset image library, matching the target image with each image in the preset image library, and if there is an image matching the target image in the preset image library, adjusting the electronic device to an unlocked state; if there is no image matching the target image in the preset image library, adjusting the electronic device to a locked state. The images in the preset image library are pre-collected and stored by the electronic device.

[0074] In another specific embodiment, the biometric feature category of the biometric feature detection frame corresponding to the target image is a palm feature, and the electronic device is unlocked based on the target image, including: extracting palm print features from the target image to obtain target palm print features; determining whether there is a palm print feature matching the target palm print feature in a preset palm print feature library; if there is a palm print feature matching the target palm print feature in the preset palm print feature library, adjusting the electronic device to an unlocked state; if there is no palm print feature matching the target palm print feature in the preset palm print feature library, adjusting the electronic device to a locked state.

[0075] In another specific embodiment, the biometric feature category of the biometric feature detection frame corresponding to the target image is a palm feature category, and the electronic device is unlocked based on the target image, including: extracting palm vein features from the target image to obtain target palm vein features; determining whether there is a palm vein feature matching the target palm vein feature in the preset palm vein feature library; if there is a palm vein feature matching the target palm vein feature in the preset palm vein feature library, the electronic device is adjusted to an unlocked state; if there is no palm vein feature matching the target palm vein feature in the preset palm vein feature library, the electronic device is adjusted to a locked state. Palm vein, as the name implies, is the vein in the palm. Palm vein recognition is a type of vein recognition, which belongs to biometric recognition. The palm vein recognition system first obtains a personal palm vein distribution map through a vein recognition device, extracts feature values ​​from the palm vein distribution map according to a special comparison algorithm, obtains images of finger, palm, and back of hand veins through a near-infrared camera, stores the digital image of the vein in a computer system, and stores the feature values. During vein matching, a vein map is taken in real time to extract feature values. Advanced filtering, image binarization, and refinement methods are used to extract features from the digital image and compared with the vein feature values ​​stored in the host. A complex matching algorithm is used to match the vein features, thereby identifying and confirming the individual's identity.

[0076] Furthermore, the unlocking method of the electronic device also includes: obtaining a liveness detection result obtained by performing liveness detection on a palm object in a target image, wherein the liveness detection result is that the palm object in the target image is alive or the palm object in the target image is not alive; when a palm vein feature matching the target palm vein feature exists in a preset palm vein feature library and the palm object in the target image is alive, adjusting the electronic device to an unlocked state.

[0077] Liveness detection can use detection algorithms such as stereoscopic liveness detection, sub-surface detection, or infrared spectrometer detection. Stereoscopic liveness detection uses depth images to determine the 3D nature of the face, which can defend against 2D attacks such as mobile phone, computer and other display screens and printed photos. Sub-surface detection uses the difference in sub-surface scattering to determine the skin of the face, which can defend against non-human face material prostheses. Infrared spectrometer detection is based on infrared cameras and floodlight images in dark light environments.

[0078] Specifically, the electronic device may be configured with a depth lens, so as to obtain a palm depth image corresponding to the target image using the depth lens. After obtaining the palm depth image, the electronic device may obtain the depth value of each pixel point according to the palm depth image. If the depth values ​​are the same, it means that the palm object in the target image, that is, the presented user's palm, is actually a flat image, which means that the palm object in the target image is not alive. If there is a difference, it means that the palm object in the target image is alive.

[0079] Furthermore, the unlocking method of the electronic device also includes: determining whether the target image is an abnormal image, and when the target image is an abnormal image, reacquiring the image taken by the camera of the electronic device; when the target image is not an abnormal image, unlocking the electronic device based on the target image.

[0080] In a specific embodiment, the average brightness of each pixel on the target image is obtained, and if the average brightness is greater than a first preset brightness value and less than a second preset brightness value, the target image is determined not to be an abnormal image; if the average brightness is not greater than the first preset brightness value or not less than the second preset brightness value, the target image is determined to be an abnormal image. The first preset brightness value and the second preset brightness value can be set according to specific circumstances.

[0081] In another specific embodiment, the clarity of the target image is obtained. If the clarity of the target image is greater than the preset clarity, it is determined that the target image is not an abnormal image; if the clarity of the target image is not greater than the preset clarity, it is determined that the target image is an abnormal image. Specifically, the clarity of the target image is calculated using the Tenengrad gradient method, the Laplacian gradient method or other methods. Among them, the Tenengrad gradient method uses the Sobel operator to calculate the gradients in the horizontal and vertical directions respectively. The higher the gradient value in the same scene, the clearer the image. The following is a specific implementation. The indicator measured is the average grayscale value of the image after being processed by the Sobel operator. The larger the value, the clearer the image. The Laplacian gradient method only needs to replace the Sobel operator in the Tenengrad gradient method with the Laplacian operator.

[0082] In another specific embodiment, the target image is segmented to obtain the area of ​​the biometric segmentation region in the target image, and the occupied area ratio of the biometric segmentation region in the target image to the target image is calculated. If the occupied area ratio is greater than a preset ratio, it is determined that the target image is not an abnormal image; if the occupied area ratio is not greater than the preset ratio, it is determined that the target image is an abnormal image.

[0083] In another specific embodiment, the biometric deflection angle in the target image is detected based on a preset deflection angle model to obtain the biometric deflection angle. If the biometric deflection angle is not greater than the preset angle, it is determined that the target image is not an abnormal image; if the biometric deflection angle is greater than the preset angle, it is determined that the target image is an abnormal image.

[0084] Further, the target image is segmented to obtain the palm segmentation area, and the palm area ratio of the palm segmentation area on the target image is calculated. If the palm area ratio belongs to the preset ratio range, the target image is determined not to be an abnormal image; if the palm posture does not belong to the preset ratio range, the target image is determined to be an abnormal image. Among them, the preset ratio range can be set according to the specific situation and is not limited here.

[0085] Furthermore, the palm posture of the target image is estimated using a preset palm posture estimation model to obtain the palm posture. If the palm posture belongs to the preset posture set, the target image is determined not to be an abnormal image; if the palm posture belongs to the preset posture set, the target image is determined to be an abnormal image. For example, the preset posture set includes the posture of the palm extending toward the camera, and the palm posture that does not belong to the preset posture set may be a fist posture, a heart posture, a bowing posture, a farewell posture, a prayer posture, etc. Among them, the preset palm posture estimation model may be a Deeppose model or other models, which are not limited here. Palm posture estimation refers to accurately locating the position of the palm key point detection from the video or image, so as to infer the corresponding palm posture based on these positional relationships.

[0086] Furthermore, the target image is detected for occlusion and the occlusion rate is obtained. If the occlusion rate is higher than the preset value, the target image is determined to be an abnormal image; if the occlusion rate is not higher than the preset value, the target image is determined not to be an abnormal image. The occlusion rate is the ratio of the area of ​​the occluded area to the area of ​​the target image. Occlusion detection generally occurs when the camera is blocked by foreign objects, resulting in the entire scene or a part of the scene being invisible. The blocked part is generally black, resulting in image abnormality.

[0087] Specifically, the target image is detected for occlusion and the occlusion rate is obtained, including: binarizing the target image, with the darker part as the foreground and the other part as the background. Connected area detection is performed on the foreground to obtain the maximum connected area, and the maximum connected area is determined as the occluded area. The ratio of the area of ​​the occluded area to the area of ​​the target image is determined as the occlusion rate.

[0088] Further, in order to reduce the size of the preset biometric detection model while maintaining the performance of the preset biometric detection model, refer to Figure 4 In the embodiment of the present application, a preset target detection model is trained based on a preset training set to obtain a preset biometric detection model, including:

[0089] 301. Add a quantization operator to the preset target detection model to obtain a first target detection model.

[0090] In a specific embodiment, the quantization operator can be a fake operator, which is responsible for quantizing and then dequantizing the parameters and inputs of the operator, and then recording the scale. For example, the fake operator is an FQ (fake-quan) operator, and the accuracy of the preset target detection model is FP32, so the input and weight are also FP32. The FQ operator will convert the FP32 precision input and weight into INT8 and then back to FP32, remembering the scale information during the conversion process. FP32 means 32-bit floating point, single precision, occupies 4 bytes, a total of 32 bits. INT8 means 8-bit fixed-point integer, eight-bit integer, occupies 1 byte.

[0091] Specifically, the FQ (fake-quan) operator is a QDQ module, which contains the quantization ratio of the layer or the activation value. QDQ is the Q (quantization) operation and the DQ (dequantization) operation, which are usually used as simulated quantization operations in the network. The QDQ module will participate in the training when training the model, and is responsible for quantizing the input FP32 tensor to INT8, and then dequantizing the INT8 tensor to FP32. The accuracy used for training in the actual network is still FP32, but this quantization operator can learn the scale information of quantization and dequantization during training, so that the model weights and quantization parameters can better adapt to the quantization process during training, and the accuracy after quantization is relatively higher.

[0092] 302. Train the first target detection model based on a preset training set to obtain a second target detection model.

[0093] Specifically, the sample image is input into the first target detection model to obtain various biometric prediction boxes on the sample image, and the target loss value is calculated based on the various biometric prediction boxes and various biometric annotation boxes on the sample image to determine whether the target loss value is lower than the preset loss value. If the target loss value is lower than the preset loss value, the first target detection model at this time is determined as the second target detection model; if the target loss value is not lower than the preset loss value, the weight of the first target detection model is updated, and the sample image is re-input into the updated first target detection model to obtain an updated target loss value. After multiple iterations of updating until the target loss value is lower than the preset loss value, the first target detection model at this time is determined as the second target detection model.

[0094] 303. Determine a preset biometric detection model based on the second target detection model.

[0095] In a specific embodiment, the second target detection model is determined as a preset biometric detection model.

[0096] In order to further reduce the size of the preset biometric detection model while maintaining the performance of the preset biometric detection model, in another specific embodiment, determining the preset biometric detection model based on the second target detection model includes:

[0097] (1) Pruning the second target detection model based on multiple different model pruning strategies to obtain multiple pruned target detection models.

[0098] Among them, the model pruning strategy can be head pruning, weight pruning, block pruning, layer pruning or a combination thereof. For example, if you want to maximize the accuracy, you can choose weight pruning and block pruning, which have less impact on performance. If you want to maximize model acceleration, you can sacrifice some model performance and choose layer pruning and head pruning, which have more obvious acceleration effects.

[0099] Head pruning is a pruning method that uses attention heads as pruning units. This method prunes some attention heads in the second object detection model. Each attention head has an independent set of parameters, so the number of parameters can be reduced by pruning the attention heads.

[0100] Weight Pruning performs global pruning on the second object detection model based on the size of the weights. Similar to the convolutional model, connections with smaller weights are considered unimportant and can be pruned.

[0101] Block Pruning divides the second object detection model into multiple blocks and then prunes the entire block. A block can be a multi-head attention or a feedforward network. After pruning, the unpruned blocks will continue to be used, but the number of input and output channels also needs to be adjusted accordingly.

[0102] Layer Pruning is a pruning method that uses the weight of the entire layer as the pruning unit. It sets importance indicators for different layers of the second object detection model, and then prunes the unimportant layers as a whole. This is a coarse-grained pruning method. Although it can bring great acceleration effects, it also has a greater impact on performance.

[0103] (2) Calculate the model evaluation indicators of each pruned target detection model on the preset training set.

[0104] In the embodiment of the present application, the model evaluation index may include indicators such as prediction speed and prediction accuracy, etc., which can be selected according to the specific situation.

[0105] (3) The pruned target detection model whose model evaluation indicators meet the preset conditions is determined as the preset biometric detection model.

[0106] In the embodiment of the present application, the preset condition may be that the prediction speed is the maximum and the prediction accuracy is the highest. The pruned target detection model with the maximum prediction speed and the highest prediction accuracy is determined as the preset biometric detection model.

[0107] This application prunes and optimizes the network so that the network model can run quickly on the mobile phone with very low MACs (Multiply-Accumulate Operations), which can fully meet the real-time detection requirements on the mobile phone. In the process of designing the algorithm, this application further optimizes for various scenarios. After testing, it can achieve higher detection accuracy in the recognition of various scenarios. This application uses training quantization technology in the process of training the target detection network, which can greatly reduce the model size while maintaining the model accuracy, and greatly improve the model reasoning speed.

[0108] Please refer to Figure 5 , Figure 5 FIG. 1 is a flow chart of another embodiment of the unlocking method of the electronic device provided in the embodiment of the present application. Figure 5 As shown, the process of the unlocking method of the electronic device provided by this application is as follows:

[0109] 401. Obtain a preset training set and a preset target detection model.

[0110] The preset training set includes sample images and various biometric annotation boxes annotated on the sample images. The biometric annotation boxes contain biometric features, and the biometric annotation boxes are divided into various biometric categories according to the biometric features therein. The preset target detection model is TPH-yolov5.

[0111] 402. Add a quantization operator to the preset target detection model to obtain a first target detection model.

[0112] The quantization operator can be a fake operator, which is responsible for quantizing and dequantizing the parameters and inputs of the operator, and then recording the scale. For example, if the fake operator is an FQ (fake-quan) operator, the accuracy of the preset target detection model is FP32, and the input and weight are therefore also FP32. The FQ operator will convert the input and weight of FP32 accuracy to INT8 and then back to FP32, and remember the scale information during the conversion process.

[0113] 403. Train the first target detection model based on a preset training set to obtain a second target detection model.

[0114] Specifically, the sample image is input into the first target detection model to obtain various biometric prediction boxes on the sample image, and the target loss value is calculated based on the various biometric prediction boxes and various biometric annotation boxes on the sample image to determine whether the target loss value is lower than the preset loss value. If the target loss value is lower than the preset loss value, the first target detection model at this time is determined as the second target detection model; if the target loss value is not lower than the preset loss value, the weight of the first target detection model is updated, and the sample image is re-input into the updated first target detection model to obtain an updated target loss value. After multiple iterations of updating until the target loss value is lower than the preset loss value, the first target detection model at this time is determined as the second target detection model.

[0115] 404. Prune the second object detection model based on a plurality of different model pruning strategies to obtain a plurality of pruned object detection models.

[0116] The model pruning strategy may be head pruning, weight pruning, block pruning, layer pruning, or a combination thereof. The model pruning strategy is used to delete some network layers in the second object detection model and reduce the weights in some network layers.

[0117] 405. Calculate the model evaluation index of each pruned target detection model on the preset training set.

[0118] In the embodiment of the present application, the model evaluation index may include indicators such as prediction speed and prediction accuracy, etc., which can be selected according to the specific situation.

[0119] 406. Determine the pruned target detection model whose model evaluation index meets the preset conditions as the preset biometric detection model.

[0120] In the embodiment of the present application, the preset condition may be that the prediction speed is the maximum and the prediction accuracy is the highest. The pruned target detection model with the maximum prediction speed and the highest prediction accuracy is determined as the preset biometric detection model.

[0121] 407 : Determine whether the object close to the electronic device and the object distance between the electronic device is within a preset range.

[0122] The preset distance range may be set according to specific circumstances, for example, the preset range is 5 cm to 10 cm.

[0123] 408. Obtain an image captured by a camera of the electronic device.

[0124] In the embodiment of the present application, if the object distance between the object close to the electronic device and the electronic device is within a preset range, an image captured by a camera of the electronic device is acquired.

[0125] 409. Detect the captured image based on a preset biometric detection model to obtain prediction confidence and biometric category of each biometric detection frame on the captured image.

[0126] For example, there are two biometric detection frames on the captured image, the biometric category of one of the biometric detection frames is the palm feature category, and the prediction confidence of the biometric detection frame of the palm feature category is 0.8; the biometric category of the other biometric detection frame is the face feature category, and the prediction confidence of the biometric category of the face feature category is 0.7.

[0127] 410. When there are at least two biometric feature categories on the captured image, determine the image within the biometric feature detection frame with the highest prediction confidence on the captured image as the target image.

[0128] For example, there are two biometric detection frames on the captured image, one of which has a biometric category of palm feature category and a prediction confidence of 0.8; the other has a biometric category of face feature category and a prediction confidence of 0.7. The image in the biometric detection frame of the palm feature category is determined as the target image.

[0129] 411. When the biometric feature category in the target image is a palm feature category, determine whether the target image is an abnormal image.

[0130] In a specific embodiment, the average brightness value of each pixel on the target image is obtained. If the average brightness value is not greater than a first preset brightness value or not less than a second preset brightness value, the target image is determined to be an abnormal image; if the average brightness value is greater than the first preset brightness value and less than the second preset brightness value, the clarity of the target image is obtained. If the clarity of the target image is not greater than the preset clarity, the target image is determined to be an abnormal image; if the clarity of the target image is greater than the preset clarity, the target image is segmented to obtain the area of ​​the biometric feature segmentation region in the target image, and the occupied area ratio of the biometric feature segmentation region in the target image to the target image is calculated. If the occupied area ratio is not greater than a preset ratio, the target image is determined to be an abnormal image; if the occupied area ratio is greater than the preset ratio, the biometric feature deflection angle in the target image is detected based on a preset deflection angle model to obtain the biometric feature deflection angle. If the biometric feature deflection angle is not greater than the preset angle, it is determined that the target image is not an abnormal image; if the biometric feature deflection angle is greater than the preset angle, it is determined that the target image is an abnormal image.

[0131] 412. Determine whether the palm object in the target image is alive based on the liveness detection result.

[0132] Specifically, when it is determined that the target image is not an abnormal image, a liveness detection result obtained by performing liveness detection on the palm object in the target image is obtained, and whether the palm object in the target image is alive is determined based on the liveness detection result. The liveness detection result is that the palm object in the target image is alive or the palm object in the target image is not alive. When it is determined that the target image is an abnormal image, return to execute 408: obtain the captured image captured by the camera of the electronic device.

[0133] Specifically, a detection algorithm such as stereoscopic liveness detection, sub-surface detection, or infrared spectrometer detection may be used to perform liveness detection on the palm object in the target image to obtain a liveness detection result.

[0134] In an embodiment of the present application, if the palm object in the target image is judged to be alive based on the liveness detection result, it is determined whether there is a palm print feature matching the target palm print feature in the preset palm print feature library; if the palm object in the target image is judged not to be alive based on the liveness detection result, the electronic device is kept in a locked state.

[0135] 413. Determine whether there is a palm print feature matching the target palm print feature in the preset palm print feature library.

[0136] In an embodiment of the present application, palm print feature extraction is performed on a target image to obtain a target palm print feature. If a palm print feature matching the target palm print feature exists in a preset palm print feature library, palm vein feature extraction is performed on the target image to obtain a target palm vein feature, and it is determined whether a palm vein feature matching the target palm vein feature exists in the preset palm vein feature library. If a palm print feature matching the target palm vein feature does not exist in the preset palm print feature library, the electronic device is kept in a locked state.

[0137] 414. Determine whether there is a palm vein feature matching the target palm vein feature in the preset palm vein feature library.

[0138] In an embodiment of the present application, palm vein features are extracted from a target image to obtain a target palm vein feature, and it is determined whether there is a palm vein feature matching the target palm vein feature in a preset palm vein feature library. If there is a palm vein feature matching the target palm vein feature in the preset palm vein feature library, the electronic device is adjusted to an unlocked state; if there is no palm vein feature matching the target palm vein feature in the preset palm vein feature library, the electronic device is kept in a locked state.

[0139] 415. Adjust the electronic device to unlock state.

[0140] In the embodiment of the present application, if there is a palm vein feature matching the target palm vein feature in the preset palm vein feature library, the electronic device is adjusted to an unlocked state.

[0141] 416. Keep electronic devices locked.

[0142] In the embodiment of the present application, if there is no palm vein feature matching the target palm vein feature in the preset palm vein feature library, the electronic device is kept in a locked state.

[0143] 417. When the biometric feature category in the target image is a facial feature category, start face unlocking.

[0144] 418. When the biometric feature category in the target image is a fingerprint feature category, fingerprint unlocking is initiated.

[0145] Of course, in other embodiments, when the biometric feature category in the target image is another biometric feature category, the corresponding unlocking method is activated.

[0146] It should be noted that the order of steps 412-418 can be changed according to the specific situation, and this application does not limit this. For example, the palm print feature of the target image is extracted to obtain the target palm print feature. It is determined whether there is a palm print feature matching the target palm print feature in the preset palm print feature library. If there is no palm print feature matching the target palm print feature in the preset palm print feature library, the electronic device is kept in a locked state; if there is a palm print feature matching the target palm print feature in the preset palm print feature library, the liveness detection result obtained by performing liveness detection on the palm object in the target image is obtained. Based on the liveness detection result, it is determined whether the palm object in the target image is alive. If the palm object in the target image is judged not to be alive based on the liveness detection result, the electronic device is kept in a locked state; if the palm object in the target image is judged to be alive based on the liveness detection result, palm vein features are extracted from the target image to obtain target palm vein features, and it is judged whether there is a palm vein feature matching the target palm vein feature in a preset palm vein feature library; if there is a palm vein feature matching the target palm vein feature in the preset palm vein feature library, the electronic device is adjusted to an unlocked state; if there is no palm vein feature matching the target palm vein feature in the preset palm vein feature library, the electronic device is kept in a locked state.

[0147] In order to better implement the unlocking method of the electronic device provided in the embodiment of the present application, the embodiment of the present application also provides an unlocking device of the electronic device based on the unlocking method of the electronic device. The meanings of the terms are the same as those in the unlocking method of the electronic device. For specific implementation details, please refer to the description in the above method embodiment.

[0148] Please refer to Figure 6 , Figure 66 is a schematic diagram of the structure of an unlocking device for an electronic device provided in an embodiment of the present application. The unlocking device for an electronic device may include an acquisition module 601, a detection module 602, a determination module 603, and an unlocking module 604, wherein:

[0149] An acquisition module 601 is used to acquire an image captured by a camera of the electronic device;

[0150] A detection module 602, configured to detect the captured image based on a preset biometric detection model, and obtain a prediction confidence and a biometric category of each biometric detection frame on the captured image;

[0151] A determination module 603 is used to determine the image in the biometric feature detection frame with the highest prediction confidence on the captured image as the target image;

[0152] The unlocking module 604 is configured to unlock the electronic device based on the target image.

[0153] The specific implementation of each of the above modules can be found in the previous embodiments and will not be described in detail here.

[0154] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the processor is used to execute the steps in the unlocking method of the electronic device provided in this embodiment by calling a computer program stored in the memory.

[0155] Please refer to Figure 7 , Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0156] The electronic device may include components such as a processor 101 with one or more processing cores, a memory 102 with one or more computer-readable storage media, a power supply 103, and an input unit 104. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0157] The processor 101 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 102, and calling data stored in the memory 102. Optionally, the processor 101 may include one or more processing cores; optionally, the processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 101.

[0158] The memory 102 can be used to store software programs and modules. The processor 101 executes various functional applications and data processing by running the software programs and modules stored in the memory 102. The memory 102 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 102 may also include a memory controller to provide the processor 101 with access to the memory 102.

[0159] The electronic device also includes a power supply 103 for supplying power to each component. Optionally, the power supply 103 can be logically connected to the processor 101 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 103 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0160] The electronic device may further include an input unit 104, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0161] Although not shown, the electronic device may also include a display unit, an image acquisition component, etc., which will not be described in detail here. Specifically in this embodiment, the processor 101 in the electronic device will load the executable code corresponding to one or more computer programs into the memory 102 according to the following instructions, and the processor 101 will execute the steps in the unlocking method of the electronic device provided in this application, such as:

[0162] Acquire an image captured by a camera of an electronic device; detect the captured image to obtain prediction confidences and biometric categories of each biometric image on the captured image; when there are at least two biometric categories on the captured image, determine the biometric image with the highest prediction confidence on the captured image as a target image; and unlock the electronic device based on the target image.

[0163] It should be noted that the unlocking method of the electronic device provided in the embodiment of the present application and the electronic device in the above embodiment belong to the same concept, and its specific implementation process is detailed in the above related embodiments and will not be repeated here.

[0164] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program stored therein is executed on a processor of an electronic device provided in an embodiment of the present application, the processor of the electronic device executes the steps in the unlocking method of the electronic device provided in the present application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0165] The present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes various optional implementations of the unlocking method of the electronic device.

[0166] The above is a detailed introduction to the unlocking method and device of an electronic device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0167] It should be noted that when the above embodiments of the present application are applied to specific products or technologies, the relevant data of the user is involved, and the user's permission or consent is required, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A method for unlocking an electronic device, characterized in that: include: Acquire an image captured by a camera of the electronic device; Detecting the captured image to obtain prediction confidence and biometric category of each biometric image on the captured image; In the case where there are at least two biometric feature categories on the captured image, determining the biometric feature image with the highest prediction confidence on the captured image as the target image; The electronic device is unlocked based on the target image.

2. The method for unlocking an electronic device according to claim 1, characterized in that: The detecting the captured image to obtain the prediction confidence and biometric category of each biometric image on the captured image includes: Detecting the captured image based on a preset biometric detection model to obtain a prediction confidence and a biometric category of each biometric detection frame on the captured image; The images within each of the biometric feature detection frames are determined as each of the biometric feature images, and the prediction confidence and biometric feature category of each of the biometric feature images on each of the captured images are obtained.

3. The method for unlocking an electronic device according to claim 2, characterized in that: Before the step of detecting the captured image based on a preset biometric detection model to obtain the prediction confidence and biometric category of each biometric detection frame on the captured image, the step includes: Obtaining a preset training set and a preset target detection model, wherein the preset training set includes a sample image and various biometric feature annotation boxes annotated on the sample image; The preset target detection model is trained based on the preset training set to obtain the preset biometric detection model.

4. The method for unlocking an electronic device according to claim 3, characterized in that: The step of training the preset target detection model based on the preset training set to obtain the preset biometric detection model includes: Adding a quantization operator to the preset target detection model to obtain a first target detection model; Training the first target detection model based on the preset training set to obtain a second target detection model; The preset biometric detection model is determined based on the second target detection model.

5. The method for unlocking an electronic device according to claim 4, characterized in that: The determining the preset biometric detection model based on the second target detection model includes: Pruning the second object detection model based on a plurality of different model pruning strategies respectively to obtain a plurality of pruned object detection models; Calculate the model evaluation index of each of the pruned target detection models on the preset training set respectively; The pruned target detection model whose model evaluation index meets the preset conditions is determined as the preset biometric detection model.

6. The method for unlocking an electronic device according to claim 1, characterized in that: The biometric feature category of the target image is one of a preset feature category set, and the preset feature category set includes at least one of a palm feature category, a fingerprint feature category, a face feature category, and an iris feature category.

7. The method for unlocking an electronic device according to claim 6, characterized in that: The biometric feature category of the biometric feature detection frame corresponding to the target image is a palm feature category, and the unlocking process of the electronic device based on the target image includes: Extracting palmprint features from the target image to obtain target palmprint features; Determining whether there is a palm print feature matching the target palm print feature in a preset palm print feature library; When there is a palm print feature matching the target palm print feature in the preset palm print feature library, the electronic device is adjusted to an unlocked state.

8. The method for unlocking an electronic device according to claim 7, characterized in that: The unlocking method of the electronic device comprises: Acquire a liveness detection result obtained by performing liveness detection on the palm object in the target image, wherein the liveness detection result is that the palm object in the target image is alive or that the palm object in the target image is not alive; Determining whether the palm object in the target image is a living object based on the living body detection result; In the case that a palm print feature matching the target palm print feature exists in the preset palm print feature library and the palm object in the target image is alive, the electronic device is adjusted to an unlocked state.

9. The method for unlocking an electronic device according to claim 1, characterized in that: The step of obtaining the image captured by the camera of the electronic device includes: detecting an object distance between an object close to the electronic device and the electronic device; When the object distance is within a preset distance range, the camera of the electronic device is controlled to capture the captured image.

10. An unlocking device for an electronic device, characterized in that: include: An acquisition module, used to acquire an image captured by a camera of the electronic device; A detection module, used to detect the captured image and obtain the prediction confidence and biometric category of each biometric image on the captured image; a determination module, configured to determine, when there are at least two biometric feature categories on the captured image, a biometric feature image with the highest prediction confidence on the captured image as a target image; An unlocking module is used to unlock the electronic device based on the target image.

11. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the steps in the unlocking method of the electronic device according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the unlocking method of an electronic device according to any one of claims 1 to 9.

13. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps in the method for unlocking an electronic device as described in any one of claims 1 to 9 are implemented.

Images