Fingerprint recognition method and device

Abstract

A fingerprint recognition method and device, the method being applied to the field of fingerprint recognition technology. The method comprises: receiving a first operation from a user; in response to the first operation, obtaining a first fingerprint, and performing fingerprint recognition based on the first fingerprint; if fingerprint recognition fails, repairing the first fingerprint using a first repair model to obtain a second fingerprint; the first repair model is obtained by training a model based on fingerprint image data, and a first loss function is determined during the model training process, the first loss function being determined based on directional field information, the directional field information being obtained by filtering the repaired fingerprint image obtained during the training process; performing fingerprint recognition based on the second fingerprint can effectively repair the fingerprint image, improve the success rate of fingerprint recognition, and greatly enhance the user's fingerprint recognition experience.

Description

Technical Field

[0001] The present application relates to the field of biometrics, and in particular, to a fingerprint recognition method and device. Background Art

[0002] With the widespread adoption of smart devices, fingerprint recognition technology has rapidly developed in the terminal sector. However, in fingerprint applications, poor quality fingerprints often occur. This can affect the success rate of fingerprint recognition. Existing technologies use repair techniques to obtain high-quality fingerprints. However, these methods have limitations and are unable to effectively improve fingerprint recognition rates, impacting the user experience. Therefore, improving fingerprint recognition rates has become a pressing issue. Summary of the Invention

[0003] In view of this, the present application provides a fingerprint recognition method, device, computer-readable storage medium and computer program product, which can effectively repair fingerprint images, improve the success rate of fingerprint recognition, and greatly enhance the user's fingerprint recognition experience.

[0004] In a first aspect, a fingerprint recognition method is provided, comprising:

[0005] receiving a first operation of a user, where the first operation is used to trigger fingerprint recognition;

[0006] In response to the first operation, obtaining a first fingerprint, and performing fingerprint recognition based on the first fingerprint;

[0007] When fingerprint recognition fails, repairing the first fingerprint using the first repair model to obtain a second fingerprint;

[0008] The first restoration model is obtained by performing model training based on fingerprint image data, and a first loss function is determined during the model training process, the first loss function is determined based on directional field information, and the directional field information is obtained by filtering the restored fingerprint image obtained during the training process;

[0009] The fingerprint image data pair includes data of a third fingerprint and a fourth fingerprint having a corresponding relationship, and the quality of the fourth fingerprint is better than that of the third fingerprint;

[0010] Fingerprint recognition is performed based on the second fingerprint.

[0011] The above method can be executed by a terminal device or a chip in the terminal device. Based on the above solution, when fingerprint recognition is unsuccessful, the first fingerprint is repaired by the first repair model to obtain a second fingerprint, that is, a repaired fingerprint. The repaired fingerprint image has higher quality. Using the repaired fingerprint for recognition can improve the success rate of fingerprint recognition and help improve the user's fingerprint recognition experience. Among them, the first repair model is a training model obtained by considering the directional field information. By considering the directional field information, the accuracy of the training model can be improved, and the convergence speed can be accelerated, so that the fingerprint image repaired by the first repair model will be smoother, which greatly improves the success rate of fingerprint recognition.

[0012] In a possible implementation, the direction field information is obtained by filtering the restored fingerprint image data, including:

[0013] Normalizing the repaired fingerprint image to obtain a normalized fingerprint image;

[0014] Performing image segmentation on the normalized fingerprint image to obtain a plurality of image blocks;

[0015] determining, for each image block of the plurality of image blocks, a direction and a frequency of the image block;

[0016] Spatial filtering is performed based on the direction and frequency of the image block to obtain the direction field information.

[0017] It can be understood that spatial filtering refers to image processing using a spatial template (such as a spatial filter).

[0018] In a possible implementation, the normalized fingerprint image satisfies the following formula:

[0019]

[0020] Among them, N1 represents the normalized result of the pixel point whose mean is greater than the prior mean, M0 represents the preset prior mean, V0 represents the prior standard deviation, and g1(x,y) represents the coordinates of the pixel point whose mean is greater than the prior mean M0. represents the mean of the restored fingerprint image, and S represents the standard deviation of the restored fingerprint image;

[0021] For pixels smaller than the prior value M, the normalized result is expressed as:

[0022]

[0023] Among them, N2 represents the normalized result of the pixel point whose mean is less than the prior mean, M0 represents the preset prior mean, V0 represents the prior standard deviation, and g2(x,y) represents the coordinates of the pixel point whose mean is less than the prior mean M0. represents the mean of the restored fingerprint image, and S represents the standard deviation of the restored fingerprint image.

[0024] In one possible implementation, the first loss function is determined based on a first output result, wherein the first output result includes direction field information of the repaired fingerprint image, and the repaired fingerprint image refers to a repaired fingerprint image obtained when the third fingerprint is input into the training network of the first repair model.

[0025] Optionally, the first loss function satisfies the following formula:

[0026]

[0027] Among them, L 方向场信息 Represents the loss function of the direction field information, is the actual value, θ i is the predicted value, and n represents the number of angles.

[0028] In one possible implementation, a second loss function is also determined during the training process of the first repair model. The second loss function is determined based on the second output result and the fourth fingerprint. The second output result refers to: the result of the repaired fingerprint image obtained when the third fingerprint is input into the training network of the first repair model during the model training process.

[0029] The embodiment of the present application does not specifically limit the filtering method for obtaining the directional field information.

[0030] In a possible implementation, the directional field information is obtained through Gabor filtering.

[0031] Therefore, the directional field information of the repaired fingerprint image can be obtained by using Gabor filtering.

[0032] In a possible implementation, the directional field information includes a feature map of the directional field. Here, the feature map of the directional field is used in the training network of the first restoration model, so that the training process converges faster.

[0033] In a possible implementation, the fingerprint image data pair is obtained through a two-way generative adversarial network degradation model, wherein the two-way generative adversarial network degradation model is trained based on training data.

[0034] Compared with manually selecting fingerprint image data pairs for model training, generating the above fingerprint image data pairs through a two-way generative adversarial network model helps reduce the cost of collecting training data.

[0035] Optionally, if the fingerprint image data pairs used to train the model are generated by a data generation model, the generation loss function can also be used as a loss function in the total loss function.

[0036] Secondly, a model training method is provided, including:

[0037] Acquiring a fingerprint image data pair, the fingerprint image data pair including a third fingerprint and a fourth fingerprint, the fourth fingerprint being superior to the third fingerprint;

[0038] During the model training process, the third fingerprint is input into the training network of the first restoration model to obtain a first output result and a second output result. The first output result includes the directional field information of the restored fingerprint image, and the restored fingerprint image refers to the restored fingerprint image obtained when the third fingerprint is input into the training network of the first restoration model. The second output result refers to the restored fingerprint image obtained when the third fingerprint is input into the training network of the first restoration model during the model training process;

[0039] Determining a first loss function based on the first output result, and determining a second loss function based on the second output result and a fourth fingerprint;

[0040] The first repair model is obtained by training the first loss function and the second loss function.

[0041] The above method can be performed by a model training device. Based on the above scheme, model training is performed using fingerprint image data, and a loss function is determined based on the first and second output results, ultimately resulting in a highly accurate first restoration model. By considering directional field information, the trained fingerprint restoration model achieves higher accuracy and faster convergence. This results in smoother fingerprint images restored using the first restoration model, significantly improving the success rate of fingerprint recognition.

[0042] The relevant descriptions about the first repair model have been mentioned in the first aspect, and these descriptions are also applicable to the second aspect. For the sake of brevity, they will not be repeated here.

[0043] It can be understood that the above-mentioned first restoration model can be applied to the fingerprint recognition method of the first aspect.

[0044] In a third aspect, a fingerprint recognition device is provided, comprising a unit for executing any of the methods described in the first aspect. The device may be a terminal (or terminal device) or a chip within the terminal (or terminal device). The device includes an input unit, a display unit, and a processing unit.

[0045] When the device is a terminal, the processing unit may be a processor, the input unit may be a communication interface, and the display unit may be a graphics processing module and a screen; the terminal may also include a memory for storing computer program code, and when the processor executes the computer program code stored in the memory, the terminal executes any one of the methods in the first aspect.

[0046] When the device is a chip in a terminal, the processing unit may be a logic processing unit inside the chip, the input unit may be an output interface, a pin or a circuit, etc., and the display unit may be a graphics processing unit inside the chip; the chip may also include a memory, which may be a memory inside the chip (for example, a register, a cache, etc.) or a memory located outside the chip (for example, a read-only memory, a random access memory, etc.); the memory is used to store computer program code, and when the processor executes the computer program code stored in the memory, the chip executes any one of the methods of the first aspect.

[0047] In a fourth aspect, a model training device is provided, which is used to execute the method in the second aspect or any possible implementation of the second aspect. Specifically, the device includes a unit for executing the method in the second aspect or any possible implementation of the second aspect.

[0048] In a fifth aspect, a model training apparatus is provided. The apparatus includes a processor, a memory, and a communication interface. The processor is connected to the memory and the communication interface. The memory is used to store instructions, the processor is used to execute the instructions, and the communication interface is used to communicate with other network elements under the control of the processor. When the processor executes the instructions stored in the memory, the execution causes the processor to perform the method of the second aspect or any possible implementation of the second aspect.

[0049] In a sixth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program code. When the computer program code is executed by a fingerprint recognition device, the device executes any one of the methods in the first aspect.

[0050] In the seventh aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program code. When the computer program code is executed by a device for model training, the device executes any one of the methods in the second aspect.

[0051] In an eighth aspect, a computer program product is provided, comprising: a computer program code, which, when executed by a fingerprint recognition device, causes the device to execute any one of the methods in the first aspect.

[0052] In a ninth aspect, a computer program product is provided, comprising: a computer program code, which, when executed by a device trained by a model, causes the device to execute any one of the methods in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is an example diagram of an application scenario of an embodiment of the present application;

[0054] Figure 2 is a schematic diagram of a hardware system of an electronic device applicable to the present application;

[0055] Figure 3 is a schematic diagram of a software system for an electronic device applicable to the present application;

[0056] Figure 4 is a schematic flow chart of a fingerprint recognition method according to an embodiment of the present application;

[0057] Figure 5 is an example diagram of the direction angle;

[0058] Figure 6 is a schematic flow chart of the training process of the first repair model in an embodiment of the present application;

[0059] Figure 7 This is an example of the comparison of the effects of fingerprint restoration images;

[0060] Figure 8 is a schematic diagram of an application scenario of an embodiment of the present application;

[0061] Figure 9 It is a schematic block diagram of a fingerprint recognition device according to an embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0063] The fingerprint recognition method provided in the embodiments of the present application can be applied to electronic devices with fingerprint recognition functions. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a wearable device, a multimedia player, an e-book reader, a personal computer, a personal digital assistant (PDA), a netbook, an augmented reality (AR) device, a virtual reality (VR) device, or other electronic device. The present application does not limit the specific form of the electronic device.

[0064] As an example and not a limitation, when the electronic device is a wearable device, the wearable device can be a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the human body or integrated into the user's clothes or accessories, and can collect the user's biometric data. Wearable devices are not just hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. In one implementation, wearable smart devices include devices that are full-featured, large in size, and can achieve full or partial functions without relying on smartphones, such as smart watches or smart glasses. In another implementation, wearable smart devices can be devices that only focus on a certain type of application function and need to be used in conjunction with other devices (such as smartphones), such as smart bracelets and smart jewelry with unlocked touch screens.

[0065] The embodiments of this application do not specifically limit the application scenarios of fingerprint recognition, and are applicable to any scenario involving fingerprint recognition, such as when a user uses a fingerprint to unlock, pay, or authenticate an identity.

[0066] It is understood that the embodiments of the present application are applicable to all scenarios involving fingerprint image processing, such as ultrasonic fingerprint recognition, optical fingerprint recognition, under-screen fingerprint recognition, and behind-screen fingerprint recognition (i.e., the fingerprint recognition device is set on the back of the electronic device).

[0067] Generally speaking, due to one or more of the following factors: wetness, pressure, damage, sweat, and the cleanliness of the fingerprint sensor, the user's fingerprint quality may be poor during fingerprint recognition. When the collected fingerprint quality is poor, fingerprint recognition will fail, affecting the user's fingerprint recognition experience.

[0068] Figure 1 This is a schematic diagram of an application scenario of an embodiment of the present application. Take the electronic device as an example, the mobile phone uses an under-screen fingerprint unlocking, such as Figure 1 As shown in (1), the user presses the fingerprint unlocking area 10 on the screen with his finger to try to unlock the phone with his fingerprint. After the user presses the fingerprint unlocking area 10, the phone will match the collected fingerprint with the user's pre-stored fingerprint. If the match is successful, the phone screen is unlocked successfully.

[0069] It should be understood that Figure 1 The fingerprint unlocking area 10 shown in (1) is only an exemplary description, and the embodiments of the present application are not limited thereto. In fact, the fingerprint unlocking area 10 can be located in other areas of the screen, such as the screen area near the power button.

[0070] It should also be understood that Figure 1The fingerprint unlocking shown in (1) is described by taking the fingerprint unlocking under the screen as an example, and the embodiments of the present application are not limited to this. For example, the embodiments of the present application are also applicable to the fingerprint unlocking on the back of the mobile phone.

[0071] If the user fails to match the fingerprint after multiple attempts, the user will be prompted that the fingerprint matching failed. A possible scenario is, for example, Figure 1 As shown in (2), the mobile phone displays a prompt box 11 to the user: "Fingerprint matching failed, please keep your finger and sensor clean."

[0072] It should be understood that the content in the prompt box 11 is only an example, and the embodiments of the present application are not limited thereto. For example, the content in the prompt box 11 can also be "The fingerprint verification has failed the maximum number of times, please use the password to unlock", or "Fingerprint matching failed".

[0073] If the user's fingerprint is not matched successfully after multiple attempts, the user can be prompted to enter the password to unlock. Figure 1 As shown in (3), the mobile phone displays the interface for the user to enter the password. Of course, Figure 1 The interface shown in (3) is just one possible scenario, and the embodiments of the present application are not limited thereto.

[0074] It should be understood that Figure 1 The scenario in the figure is only a schematic illustration of an application scenario of the present application, which does not constitute a limitation on the embodiments of the present application, and the present application is not limited to this.

[0075] In order to improve the user's fingerprint recognition experience, the embodiment of the present application proposes a fingerprint recognition method, which repairs the fingerprint that failed to be recognized through a first repair model, and uses the repaired fingerprint for fingerprint recognition to improve the success rate of fingerprint recognition. In the embodiment of the present application, the first repair model takes into account the directional field information of the fingerprint image, which can better evaluate the repaired fingerprint image, with higher accuracy, improve the image quality of the fingerprint after repair, and greatly improve the success rate of fingerprint recognition. In addition, considering the directional field information of the fingerprint image during the training process can also make the training network converge faster.

[0076] The following first combines Figure 2 and Figure 3 Describe the hardware system and software architecture applicable to the embodiments of the present application.

[0077] Figure 2 A hardware system of an electronic device suitable for the present application is shown.

[0078] The electronic device 100 can be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, etc. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device 100.

[0079] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0080] It should be noted that Figure 2 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include Figure 2 More or fewer components than those shown, or the electronic device 100 may include Figure 2 Combinations of some of the components shown, or alternatively, the electronic device 100 may include Figure 2 Subcomponents of some of the components shown. For example, Figure 2 The illustrated proximity light sensor 180G may be optional. Figure 2 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0081] The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). The different processing units may be independent devices or integrated devices.

[0082] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0083] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0084] Figure 2 The connection relationship between the modules shown is only for illustrative purposes and does not limit the connection relationship between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of the multiple connection modes in the above embodiments.

[0085] Electronic device 100 can implement display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0086] The display screen 194 can be used to display images or videos. The display screen 194 includes a display panel. The display panel can use a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini light-emitting diode (Mini LED), a micro light-emitting diode (Micro LED), a micro OLED, or a quantum dot light emitting diode (QLED). In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0087] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0088] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can perform algorithmic optimization on image noise, brightness, and color. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0089] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard red, green, blue (RGB), YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0090] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0091] The electronic device 100 can implement audio functions, such as music playback and recording, through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0092] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, including resistive, inductive, and capacitive pressure sensors. A capacitive pressure sensor may comprise at least two parallel plates made of conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes, and electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch location based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch location but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, an instruction to view short messages is executed; when a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, an instruction to create a new short message is executed.

[0093] The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The electronic device 100 emits infrared light outward through the LED. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When reflected light is detected, the electronic device 100 can determine that there is an object nearby. When no reflected light is detected, the electronic device 100 can determine that there is no object nearby. The electronic device 100 can use the proximity light sensor 180G to detect whether the user is holding the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used for automatic unlocking and automatic screen lock in leather case mode or pocket mode. It should be understood that Figure 2 The proximity light sensor 180G described in the foregoing may be an optional component. In some scenarios, an ultrasonic sensor may be used to replace the proximity light sensor 180G to detect proximity light.

[0094] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement functions such as unlocking, accessing application locks, taking photos, and answering calls.

[0095] The touch sensor 180K is also referred to as a touch-sensitive device. The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also referred to as a touch screen. The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor 180K can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100 and at a different location from the display screen 194.

[0096] The buttons 190 include a power button and a volume button. The buttons 190 can be mechanical buttons or touch buttons. The electronic device 100 can receive button input signals and implement functions related to the case input signals.

[0097] Motor 191 can generate vibration. Motor 191 can be used for incoming call reminders or for touch feedback. Motor 191 can produce different vibration feedback effects for touch operations acting on different applications. Motor 191 can also produce different vibration feedback effects for touch operations acting on different areas of the display screen 194. Different application scenarios (for example, time reminders, receiving messages, alarm clocks, and games) can correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0098] The hardware system of electronic device 100 is described in detail above. The following describes the software system of electronic device 100. The software system can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. In the embodiment of the present application, the layered architecture is used as an example to exemplify the software system of electronic device 100.

[0099] like Figure 3 As shown, a software system using a layered architecture is divided into several layers, each with clear roles and divisions of labor. Layers communicate with each other through software interfaces. In some embodiments, the software system can be divided into five layers: from top to bottom, the application layer, the application framework layer, the Android runtime and system libraries, the kernel layer, and the trusted execution environment (TEE) layer.

[0100] The application layer may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.

[0101] The application framework layer provides an application programming interface (API) and a programming framework for applications in the application layer. The application framework layer may include some predefined functions.

[0102] For example, the application framework layer includes the window manager, content provider, view system, telephony manager, resource manager, and notification manager.

[0103] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, and take screenshots.

[0104] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, and phone books.

[0105] The view system includes visual controls, such as controls for displaying text and images. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon can include a view for displaying text and a view for displaying images.

[0106] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (connected or hung up).

[0107] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, and video files.

[0108] The Notification Manager allows applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the Notification Manager is used to notify downloads and message reminders. The Notification Manager can also manage notifications that appear in the status bar at the top of the system in the form of icons or scrolling text, such as notifications from applications running in the background. The Notification Manager can also manage notifications that appear on the screen in the form of dialog windows, such as prompting text messages in the status bar, emitting alert sounds, vibrating electronic devices, and flashing indicator lights.

[0109] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

[0110] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0111] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine performs functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0112] The system library can include multiple functional modules, such as a surface manager, a media library, a 3D graphics processing library (such as the open graphics library for embedded systems (OpenGL ES)) and a 2D graphics engine (such as the skia graphics library (SGL)).

[0113] The surface manager is used to manage the display subsystem and provide the fusion of 2D layers and 3D layers for multiple applications.

[0114] The media library supports playback and recording of multiple audio and video formats, as well as still image files. It supports a variety of audio and video codecs, such as MPEG4, H.264, Moving Picture Experts Group Audio Layer III (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR), Joint Photographic Experts Group (JPG), and Portable Network Graphics (PNG).

[0115] The 3D graphics processing library can be used to implement 3D graphics drawing, image rendering, compositing and layer processing.

[0116] A 2D graphics engine is a drawing engine for 2D drawings.

[0117] The kernel layer is the layer between hardware and software. The kernel layer can include driver modules such as display driver, camera driver, audio driver, and sensor driver.

[0118] The TEE layer provides security services to the Android system. It is used to execute various biometric algorithms. The TEE layer is typically used to run key operations: (1) mobile payments: fingerprint verification, PIN code entry, etc.; (2) secure storage of confidential data: private keys, certificates, etc.; and (3) content: DRM (digital rights management).

[0119] In some possible embodiments, the TEE layer includes a fingerprint recognition algorithm module and a fingerprint repair module. Optionally, the fingerprint repair module can be independently set in the TEE layer (for example, Figure 3 ), and may also be located in the fingerprint recognition algorithm module, which is not specifically limited in the present embodiment. In the present embodiment, the fingerprint repair module is used to repair fingerprints that fail to be recognized.

[0120] In some possible examples, the fingerprint repair module is used to perform the fingerprint recognition method of the embodiment of the present application. Optionally, the fingerprint repair module uses the first repair model to repair the fingerprint that failed to be recognized.

[0121] It should be understood that the above is based on Figure 2 The structural diagram of the electronic equipment is given as an example. Figure 3 The software architecture of the embodiment of the present application is illustrated by way of example, but the embodiment of the present application is not limited thereto.

[0122] The following combination Figures 4 to 7 The fingerprint recognition method according to the embodiment of the present application is described. It can be understood that the fingerprint recognition method shown below can be used in an electronic device (such as, Figure 2 The electronic device shown is implemented in FIG.

[0123] Figure 4 FIG is a schematic flow chart of a fingerprint identification method according to an embodiment of the present application. Figure 4 As shown, the method includes the following steps:

[0124] Step 310: Receive a first operation from the user, where the first operation is used to trigger fingerprint recognition.

[0125] The embodiment of the present application does not limit the specific form of the first operation.

[0126] For example, the first operation may be an operation in which the user presses the fingerprint unlocking area with a finger to start fingerprint recognition and prepare for subsequent fingerprint unlocking.

[0127] For example, the first operation may be an operation in which a user places a finger on a fingerprint unlocking area to trigger the fingerprint recognition function. Step 320: In response to the first operation, a first fingerprint is acquired, and fingerprint recognition is performed based on the first fingerprint.

[0128] Specifically, acquiring the first fingerprint refers to collecting data of the first fingerprint. The embodiment of the present application does not limit the specific form of the fingerprint data. For example, the data of the first fingerprint includes a fingerprint image.

[0129] As a possible implementation manner, obtaining the first fingerprint includes: collecting the fingerprint by a fingerprint collection module.

[0130] Taking the fingerprint unlocking scenario as an example, the touch screen built into the electronic device is equipped with an under-screen fingerprint recognition module. The fingerprint recognition area of ​​the under-screen fingerprint recognition module can be set in a designated area of ​​the touch screen or cover the entire touch screen. This is not specifically determined. If the under-screen fingerprint recognition module is set in a designated area of ​​the touch screen, the fingerprint unlocking process will only be triggered when the user presses the designated area. If the under-screen fingerprint recognition module covers the entire touch screen, then when the electronic device is locked, the fingerprint unlocking process will be triggered by the user pressing any area of ​​the touch screen.

[0131] Fingerprint recognition based on the first fingerprint here can be achieved using current fingerprint recognition technology. For example, information of at least one standard fingerprint (e.g., fingerprint feature data, fingerprint image, etc.) can be pre-entered in the electronic device. The electronic device uses the standard fingerprint to match the first fingerprint. If the fingerprint match is successful, the next related processing can be performed. The specific processing steps may depend on the application scenario, such as payment, unlocking, etc. If fingerprint recognition fails, it is necessary to use the first repair model of the embodiment of the present application to repair the fingerprint.

[0132] Step 330: When fingerprint recognition fails, the first fingerprint is repaired using a first repair model to obtain a second fingerprint. The first repair model is obtained by training a fingerprint image data pair, and a first loss function is determined during the model training process. The first loss function is determined based on directional field information obtained by filtering the repaired fingerprint image obtained during the training process. The fingerprint image data pair includes data of a third fingerprint and a fourth fingerprint having a corresponding relationship, and the quality of the fourth fingerprint is better than that of the third fingerprint.

[0133] “Inpainting” can be understood as the process of converting the image of a first fingerprint into the image of a second fingerprint.

[0134] The first restoration model is used to restore fingerprints that failed recognition (or low-quality fingerprints) to obtain a high-quality fingerprint (e.g., a second fingerprint). In other words, the first fingerprint image is input into the first restoration model, which then outputs the second fingerprint image. The fingerprint image restored by the first restoration model is of higher quality than the original fingerprint.

[0135] It should be understood that the embodiment of the present application does not specifically limit the name of the first restoration model. The first restoration model can also be called a fingerprint restoration model, a fingerprint training model, a fingerprint feature restoration model, etc.

[0136] It should also be understood that the embodiment of the present application does not specifically limit the type of the first repair model. For example, the first repair model can be a machine learning model, a deep learning model, a neural network model, a prediction model, etc.

[0137] In the embodiment of the present application, the training process of the first restoration model considers the directional field information of the fingerprint image. Direction field information is key information for determining fingerprint quality. The directional field information is used to describe the directional field of a fingerprint. Generally speaking, the directional field of a fingerprint is an inherent property of a fingerprint image. The directional field of a fingerprint reflects the center of the fingerprint image field, the circular grain trend field, the oblique angle straight grain trend field, the horizontal grain trend field, and the mixed trend field.

[0138] The embodiment of the present application does not limit the specific form of the direction field information. The direction field information can be an angle (such as the angle of the local ridge direction), a feature map of the direction field, or other forms used to represent the direction field.

[0139] Optionally, the direction field information includes a direction angle. Figure 5 Taking any pixel point (x, y) in as an example, the direction angle of (x, y) can be defined as the angle between the tangent direction of the ridge line (or valley line) where the pixel point is located and the horizontal axis.

[0140] The direction field information is obtained by filtering the restored fingerprint image. The present embodiment does not specifically limit the filtering method. For example, Gaussian kernel filtering, Gabor filtering, scale-invariant feature transform (SIFT) filtering, etc.

[0141] Optionally, the direction field information is obtained by performing Gabor filtering on the repaired fingerprint image.

[0142] Optionally, as a possible implementation manner, the direction field information is obtained through the following steps:

[0143] Step 1: normalize the repaired fingerprint image to obtain a normalized fingerprint image.

[0144] The embodiment of the present application does not specifically limit the specific process of normalization processing.

[0145] Exemplarily, the normalization process may include the following steps:

[0146] First, take the preset prior mean (which can be recorded as M0) and prior standard deviation (which can be recorded as V0);

[0147] Then, the mean of the repaired fingerprint image is calculated (which can be recorded as ) and standard deviation (which can be recorded as S);

[0148] Obtain the value of each pixel greater than the prior mean M0 and the pixel less than the prior value M0 in the repaired fingerprint image;

[0149] For pixels whose mean is greater than the prior mean M0, the normalized result is expressed as:

[0150]

[0151] Among them, N1 represents the normalization result, M0 represents the preset prior mean, V0 represents the prior standard deviation, and g1(x,y) represents the coordinates of the pixel points greater than the prior mean M0. represents the mean of the restored fingerprint image, and S represents the standard deviation of the restored fingerprint image.

[0152] For pixels whose mean is less than the prior mean M0, the normalized result is expressed as:

[0153]

[0154] Among them, N2 represents the normalized result of the pixel point whose mean is less than the prior mean, M0 represents the preset prior mean, V0 represents the prior standard deviation, and g2(x,y) represents the coordinates of the pixel point whose mean is less than the prior mean M0. represents the mean of the restored fingerprint image, and S represents the standard deviation of the restored fingerprint image.

[0155] It can be understood that the situation of “equal to the prior mean M0” can be divided into the situation of “less than the prior value M0”, or the situation of “greater than the prior value M0”, and there is no specific limitation on this.

[0156] Step 2: dividing the normalized fingerprint image into blocks to obtain multiple image blocks.

[0157] Optionally, the image segmentation method may include a segmentation method with overlapping blocks or a segmentation method without overlapping blocks. The embodiment of the present application does not specifically limit the image segmentation method.

[0158] For example, the block division is described in an overlapping manner. Assume that the size of the fingerprint image g(x,y) is H*W, where H represents the height of the fingerprint image g(x,y), W represents the width of the fingerprint image g(x,y), and (x,y) represents the coordinates of the fingerprint image g(x,y). The fingerprint image g(x,y) is divided into image blocks of size m*n. The adjacent image blocks overlap with each other, and the width of the overlapping area is num pixels. The fingerprint image block g is obtained. ij (x,y), where i Represents the fingerprint image block g ij (x,y) is located in the i-th row of the fingerprint image g(x,y), and j represents the fingerprint image block g ij (x,y) is located in the jth row of the fingerprint image g(x,y). For example, for a fingerprint image of size 499*499, if it is divided into overlapping blocks, 71*71 image blocks can be obtained, and the size of each image block is 8*8.

[0159] Step 3: Determine the direction and frequency of each image block.

[0160] Optionally, the direction and frequency of each image block may be calculated by means of fast Fourier transform.

[0161] For example, the direction and frequency of each image block can be calculated in the MATLAB code by calling a pre-packaged fast Fourier transform function (eg, fftshift).

[0162] Step 4: performing spatial domain filtering based on the direction and frequency of the image block to obtain direction field information of the restored fingerprint image.

[0163] In the field of digital image processing, spatial filtering is defined as image processing using a spatial template. The spatial template itself can be called a spatial filter. For example, a spatial filter can be used to filter an image.

[0164] For example, if Gabor filtering is used, the spatial domain filter is a Gabor filter function.

[0165] As a possible example, in the MATLAB code, the gabor filter function can be called to obtain the direction field information, such as θ, by inputting the direction and frequency of each image block.

[0166] As a possible example, the directional field information can also be calculated using the relevant formula of the Gabor filter. For example, the mathematical expression of the two-dimensional Gabor filter function is as follows:

[0167] Plural expression:

[0168]

[0169] The real part is represented as follows:

[0170]

[0171] The imaginary part is represented as follows:

[0172]

[0173] Among them, x′=x cosθ+y sinθ, y′=-x sinθ+y cosθ;

[0174] λ represents the wavelength (the value can be specified in pixels), θ represents the direction angle, σ represents the standard deviation of the Gaussian factor of the Gabor function, γ represents the spatial aspect ratio, and ψ represents the phase offset.

[0175] It should be understood that for the specific explanation of the two-dimensional Gabor filter function, reference may be made to the existing description, as in the embodiments of the present application.

[0176] It should also be understood that the above four steps are merely exemplary descriptions, and the embodiments of the present application are not limited thereto. In fact, those skilled in the art may also use other traditional algorithms to calculate the direction field information.

[0177] Optionally, the first restoration model is obtained by learning and training a fingerprint image data pair. The fingerprint image data pair includes data of a third fingerprint and a fourth fingerprint having a corresponding relationship or mapping relationship, and the quality of the fourth fingerprint is better than the quality of the third fingerprint. In other words, the fingerprint image data pair refers to data of a high-quality fingerprint image and a low-quality fingerprint. Among them, the third fingerprint and the fourth fingerprint are a data pair (or paired fingerprint data) in the fingerprint image data pair. For example, the third fingerprint is a low-quality fingerprint, and the fourth fingerprint is a high-quality fingerprint corresponding to the third fingerprint. It should be understood that the fingerprint image data pair includes multiple pairs of fingerprint image data, and the above description only takes one pair as an example.

[0178] Step 340: Perform fingerprint recognition based on the second fingerprint.

[0179] The second fingerprint is the restored fingerprint image. In one possible implementation, fingerprint recognition is performed using the second fingerprint, and the fingerprint recognition is successful. It should be understood that the process of fingerprint recognition using a fingerprint can refer to the existing technology and will not be repeated here.

[0180] In an embodiment of the present application, when fingerprint recognition is unsuccessful, the first fingerprint is repaired using a first repair model to obtain a second fingerprint, i.e., a repaired fingerprint. The repaired fingerprint image has higher quality. Using the repaired fingerprint for recognition can improve the success rate of fingerprint recognition and help improve the user's fingerprint recognition experience. Compared to a solution that does not consider the directional field information of the fingerprint image, the first repair model of the embodiment of the present application is a training model that considers the directional field information of the fingerprint image. By considering the directional field information of the fingerprint image, the repaired fingerprint image can be better evaluated. This makes the fingerprint image repaired by the first repair model smoother, improves the quality of the fingerprint repaired image, and greatly improves the success rate of fingerprint recognition.

[0181] The first restoration model is obtained by combining deep learning with the traditional direction field algorithm. The following describes the training process of the first restoration model. Figure 6 , Figure 6 FIG is a schematic flow chart of the training process of the first repair model according to an embodiment of the present application. Figure 6 Shown, including:

[0182] Step 401: Acquire a fingerprint image data pair, wherein the fingerprint image data pair includes a third fingerprint and a fourth fingerprint, and the fourth fingerprint is superior to the third fingerprint.

[0183] The fingerprint image data pair can be understood as data for model training. In other words, the fingerprint image data pair can be used for model training. The description of the fingerprint image data pair can be referred to above and will not be repeated here.

[0184] The above fingerprint image data pairs can be generated by a data generation model.

[0185] As one possible implementation, the fingerprint image data pairs are generated using a generative adversarial network (GAN). Compared to manually selecting fingerprint image data pairs for model training, generating the fingerprint image data pairs using a two-way GAN model helps reduce the cost of collecting training data.

[0186] Optionally, the above-mentioned two-way generative adversarial network model can be obtained through training. Simply put, GAN generally includes a generative network and a discriminative network. The generative network is used to generate (fake) data. The discriminative network is used to identify the data generated by the generator. Every time the generative network generates a batch of data, it will be sent to the discriminative network for training. The trained discriminative network will identify whether it is true or false, and at the same time, the results will be fed back to the generator. The feedback results will be used to assist in retraining the generative network to make the generative network generate more realistic data, and then the data will be given to the discriminative network for training and identification. This cycle will be repeated until the generated data discrimination network can no longer distinguish between true and false, that is, there is no effective information fed back to the generative network for training, and then the training will stop.

[0187] Optionally, the present application utilizes a two-way generative adversarial network model (TwoWayGAN) to generate the above fingerprint image data pairs. Through TwoWayGAN, multiple fingerprint image data pairs can be generated.

[0188] Exemplarily, the TwoWayGAN model can be trained by artificially selecting high-quality (good) fingerprint data and low-quality (bad) fingerprint data. High-quality fingerprint data and low-quality fingerprint data can be considered as two sets (which can be represented as set A and set B respectively). TwoWayGAN consists of two generators for domain conversion (which can be represented as GA and GB respectively) and two discriminators (which can be represented as DA and DB respectively). For set A, the estimation result of set B can be first generated by GB, and then the estimation result of set B is input into GA to obtain the estimation result of set A; for set B, the estimation result of set A can be generated by GA, and then the estimation result of set A is input into GB to obtain the estimation result of set B. The two generators GA and GB can be used to generate paired data. In an embodiment of the present application, paired high-quality fingerprint data and low-quality fingerprint data can be obtained based on TwoWayGAN. In one possible implementation, after the high-quality fingerprint data is input into the TwoWayGAN model, corresponding low-quality fingerprint data can be generated, thereby obtaining a fingerprint image data pair. Another possible implementation method is to input low-quality fingerprint data into the TwoWayGAN model to generate corresponding high-quality fingerprint data, thereby obtaining a fingerprint image data pair.

[0189] Step 402: During the model training process, the third fingerprint is input into the training network of the first restoration model to obtain a first output result and a second output result. The first output result includes the directional field information of the restored fingerprint image. The restored fingerprint image refers to the restored fingerprint image obtained when the third fingerprint is input into the training network of the first restoration model. The second output result refers to the restored fingerprint image obtained when the third fingerprint is input into the training network of the first restoration model during the model training process.

[0190] In an embodiment of the present application, the training network of the first repair model may be a neural network. Exemplarily, the neural network may include: an input layer, a downsampling layer, an upsampling layer, and an output layer.

[0191] The input layer takes in an image and performs convolution preprocessing on it. The downsampling layer downsamples the fingerprint image to extract its features. The upsampling layer restores the image's dimensions (or, in other words, its resolution). The output layer performs post-processing (e.g., convolution) and outputs the training result (or, in other words, the restored fingerprint image).

[0192] Taking the third fingerprint (or the third fingerprint image) as an example to input the training network of the first repair model to obtain the first output result, the following is explained: first, the image of the third fingerprint is input through the input layer, and the image of the third fingerprint is preprocessed by convolution to obtain a preprocessed feature map; then, the preprocessed feature map is downsampled by the downsampling layer to obtain a downsampled feature map; then, the downsampled feature map is upsampled (such as deconvolution operation) by the upsampling layer to obtain an upsampled feature map (i.e., the repaired fingerprint image) and a feature map of the direction field of the upsampled feature map (i.e., the direction field feature map of the repaired fingerprint image), wherein the upsampled feature map is consistent with the image resolution of the third fingerprint; finally, the repaired fingerprint image (corresponding to the second output result) and the feature map of the direction field of the repaired fingerprint image (corresponding to the first output result) are output through the output layer.

[0193] It should be noted that in the training network of the first restoration model, the directional field information can be specifically expressed as a directional field feature map. The directional field feature map can be understood as a fingerprint directional map constructed based on the characteristics of the directional field of the fingerprint image. The fingerprint directional map can reflect the direction of the fingerprint ridges.

[0194] It should be understood that the description here is only based on the image of the third fingerprint as input, but the embodiments of the present application are not limited thereto. In fact, during the model training process, other fingerprints in the fingerprint image data pair can be used for training.

[0195] Those skilled in the art will understand that convolution preprocessing refers to performing a series of operations on each pixel in an image using a convolution kernel. Simply put, downsampling can be understood as the process of scaling an image, while upsampling can be understood as the process of magnifying an image.

[0196] In the embodiment of the present application, a deep learning algorithm is combined with a traditional algorithm to obtain a multi-task network when designing a training network. The multi-task network is then used to train the first restoration model. Specifically, the directional field information of the fingerprint image is determined using a traditional algorithm, and this directional field information is used as one of the learning branches during the deep learning process. Using the directional field information as a branch of the multi-task network not only improves the quality of fingerprint restoration but also accelerates the convergence of the training network, thereby increasing the training speed of the model.

[0197] Step 403: Determine a first loss function based on the first output result, and determine a second loss function based on the second output result and the fourth fingerprint. In other words, the second loss function is determined based on the second output result and the fourth fingerprint.

[0198] The first loss function is a loss function related to direction field information.

[0199] For example, taking the direction field information including the direction angle as an example, the first loss function is the root mean square error (RMSE) of the direction angle. For example, the first loss function satisfies the following formula:

[0200]

[0201] Among them, L 方向场信息 Represents the loss function of the direction field information, is the actual value, θ i is the predicted value, and n represents the number of angles.

[0202] For example, when calculating the direction of a sample based on the sample gradient information, the direction angle can be expressed as

[0203]

[0204] Among them, V X and V y It is determined based on the horizontal gradient and vertical gradient of the image block, and the embodiment of the present application does not specifically limit this.

[0205] As an example, the directional field angles θ of multiple image blocks are used as training samples. Assuming the directional angle range is [0-180°], samples can be taken at different angle granularities, such as 15° or 25°. For example, assuming sampling at intervals of 15°, the sample intervals are divided into [0-15°), [15°-30°), [30°-45°), … [165°-180°] based on the interval in which θ falls. Each interval is labeled with the corresponding number 1, 2, …, 12 to facilitate directional field prediction.

[0206] The second loss function is the root mean square error (RMSE) between the restored fingerprint image and the fourth fingerprint.

[0207] Exemplarily, the second loss function satisfies the following formula:

[0208]

[0209] Among them, F s (x, y) represents the restored fingerprint image (such as the second output result mentioned above), I t (x,y) represents the better quality image in the fingerprint image pair (such as the fourth fingerprint mentioned above), m represents the number of pixels, and (x,y) represents the pixel coordinates.

[0210] Optionally, as a possible implementation method, a total loss function can be obtained through the first loss function and the second loss function. The total loss function is defined as:

[0211] L=w1*L 方向场信息 +w2*RMSE

[0212] Among them, L represents the total loss function, L 方向场信息 The loss function represents the directional field information, RMSE represents the root mean square error of the pixel point, and the coefficients before each loss function (including: w1 and w2) represent the weight values ​​of each loss function. The weight values ​​can be based on prior values ​​and are not specifically limited in this embodiment of the application.

[0213] Whether to terminate the training process is determined by judging the output value of the total loss function. Optionally, the model training is terminated when the output value of the total loss function is stable and relatively small.

[0214] "Stable" means that the output value no longer changes, or in other words, the change is relatively small. In some possible embodiments, the output value of the total loss function can be considered stable when the change value is less than a certain change threshold.

[0215] The magnitude of the output value of the total loss function may also be determined by a threshold. In some possible embodiments, the training ends when the output value of the total loss function is less than a certain threshold.

[0216] Optionally, as a possible implementation, if the fingerprint image data pairs used to train the model are generated by a data generation model, the generation loss function can also be used as a loss function in the total loss function.

[0217] As an example, the generation loss function is used to measure the loss of fingerprint image data pairs generated based on the data generation model.

[0218] For example, when the data generation model is a GAN model, the generation loss function refers to the GAN loss function. The GAN loss function is used to measure the discrimination loss of paired images generated by the GAN model. Minimizing the GAN loss function can make the generated fingerprint image more realistic.

[0219] One possible implementation method is that the GAN loss function satisfies the following formula:

[0220] L GAN =-∑log D(I t ,F w (I s ))

[0221] Among them, L GAN Represents the GAN loss function, D represents the discriminator in the TwoWayGAN model, Fw (I s ) represents the second output result, I t An image representing the fourth fingerprint.

[0222] Optionally, a total loss function can be obtained based on the first loss function, the second loss function, and the GAN loss function. The description of the total loss function can be similar to the previous description.

[0223] Step 404: Obtain the first repair model by training the first loss function and the second loss function.

[0224] That is to say, the direction field information is taken into account during the model training process, so that the accuracy of the first restoration model obtained is higher, thereby making the quality of the fingerprint image subsequently restored by the first restoration model better.

[0225] Based on the above process, those skilled in the art can understand the training process of the first repair model in the embodiment of the present application.

[0226] As can be seen from the above, the fingerprint repair model of the embodiment of the present application takes into account the directional field information of the fingerprint image during the training process. Compared with the traditional algorithm for repairing fingerprint models, the trained fingerprint repair model of the embodiment of the present application has higher accuracy and faster convergence speed. In addition, through simulation comparison, it can be seen that the false rejection rate (FRR) of the trained fingerprint repair model of the embodiment of the present application is lower. FRR is one of the most important indicators to characterize the performance of biometric recognition systems.

[0227] It should be understood that the training process of the above-mentioned first repair model can be carried out on platforms such as servers, server clusters or cloud servers, can be executed in electronic devices, or can be run on other platforms, such as various simulation software (such as MATLAB) platforms, and the embodiments of this application are not limited to this.

[0228] As a possible implementation, if the training process of the first repair model is carried out on the platform, then after the training is completed, the trained first repair model can be preset in the electronic device. In this way, when performing fingerprint repair, the electronic device can directly call the first repair model to perform fingerprint repair.

[0229] Figure 7 An example diagram showing the comparison of the effects of fingerprint restoration images. Figure 7 (1) shows an example of a fingerprint image before fingerprint restoration. Figure 7 After the fingerprint image shown in (1) is repaired using the first repair model of the embodiment of the present application, the following is obtained: Figure 7 The fingerprint image shown in (2). Figure 7After the fingerprint image shown in (3) is repaired using the first repair model of the embodiment of the present application, the Figure 7 The fingerprint image shown in (4) is shown in Figure 7 As can be seen in the figure, the restored fingerprint image is smoother and clearer than the original one, and the noise is also removed. Using the restored fingerprint image for fingerprint recognition will greatly improve the success rate of fingerprint recognition.

[0230] Figure 8 Schematic diagram of an application scenario of the embodiment of the present application. Figure 8 As shown in (1), the user clicks the screen to unlock, the fingerprint unlock fails, and the interface displays as follows Figure 8 As shown in (2), a prompt window 11 is displayed to the user: "Fingerprint matching failed, please keep your finger and sensor clean." After adopting the fingerprint recognition method of the embodiment of the present application, the fingerprint collected by the mobile phone can be repaired by the first repair model. Figure 8 As shown in (3), when the user clicks the screen to unlock, the phone can repair the fingerprint that failed to be recognized through the first repair model. Unlocking with the repaired fingerprint can be successful, and the interface is displayed as follows Figure 8 As shown in (4), the interface shows the successful fingerprint unlocking interface.

[0231] It should be understood that Figure 8 The application scenarios are only for facilitating understanding by those skilled in the art and are not intended to limit the embodiments of the present application to the specific scenarios illustrated.

[0232] Combined with the above Figures 1 to 8 , describes in detail the fingerprint recognition method provided by the embodiment of the present application. Figure 9 It should be understood that the fingerprint recognition device of the present invention can perform the various fingerprint recognition methods of the aforementioned embodiments of the present invention, that is, the specific working processes of the following various products can refer to the corresponding processes in the aforementioned method embodiments.

[0233] Figure 9 is a schematic block diagram of a fingerprint recognition device 900 according to an embodiment of the present application.

[0234] It should be understood that the apparatus 900 may perform Figures 4 to 8 The fingerprint recognition method shown in FIG. 9 is a fingerprint recognition apparatus 900 including an input unit 910 and a processing unit 920. In a possible example, the apparatus 900 may be a terminal device.

[0235] In one example, the input unit 910 is used to receive a first operation of a user, where the first operation is used to trigger fingerprint recognition;

[0236] The processing unit 920 is configured to, in response to the first operation, call a fingerprint collection module to obtain a first fingerprint, and perform fingerprint recognition based on the first fingerprint;

[0237] The processing unit 920 is further configured to, when fingerprint recognition fails, repair the first fingerprint using the first repair model to obtain a second fingerprint;

[0238] The first restoration model is obtained by performing model training based on fingerprint image data, and a first loss function is determined during the model training process, the first loss function is determined based on directional field information, and the directional field information is obtained by filtering the restored fingerprint image obtained during the training process;

[0239] The fingerprint image data pair includes data of a third fingerprint and a fourth fingerprint having a corresponding relationship, and the quality of the fourth fingerprint is better than that of the third fingerprint;

[0240] The processing unit 920 is further configured to perform fingerprint recognition based on the second fingerprint.

[0241] Optionally, as a possible implementation manner, the direction field information is obtained by filtering the data of the restored fingerprint image, including:

[0242] Normalizing the repaired fingerprint image to obtain a normalized fingerprint image;

[0243] Performing image segmentation on the normalized fingerprint image to obtain a plurality of image blocks;

[0244] determining, for each image block of the plurality of image blocks, a direction and a frequency of the image block;

[0245] Spatial filtering is performed based on the direction and frequency of the image block to obtain the direction field information.

[0246] Optionally, as a possible implementation, the normalized fingerprint image satisfies the following formula:

[0247]

[0248] Among them, N1 represents the normalized result of the pixel point whose mean is greater than the prior mean, M0 represents the preset prior mean, V0 represents the prior standard deviation, and g1(x,y) represents the coordinates of the pixel point whose mean is greater than the prior mean M0. represents the mean of the restored fingerprint image, and S represents the standard deviation of the restored fingerprint image;

[0249] For pixels smaller than the prior value M, the normalized result is expressed as:

[0250]

[0251] Among them, N2 represents the normalized result of the pixel point whose mean is less than the prior mean, M0 represents the preset prior mean, V0 represents the prior standard deviation, and g2(x,y) represents the coordinates of the pixel point whose mean is less than the prior mean M0. represents the mean of the restored fingerprint image, and S represents the standard deviation of the restored fingerprint image.

[0252] Optionally, as a possible implementation method, the first loss function is determined based on a first output result, wherein the first output result includes the direction field information of the repaired fingerprint image, and the repaired fingerprint image refers to the repaired fingerprint image obtained when the third fingerprint is input into the training network of the first repair model.

[0253] Optionally, as a possible implementation method, a second loss function is also determined during the training process of the first repair model. The second loss function is determined based on the second output result and the fourth fingerprint. The second output result refers to: the result of the repaired fingerprint image obtained when the third fingerprint is input into the training network of the first repair model during the model training process.

[0254] Optionally, as a possible implementation manner, the directional field information is obtained through Gabor filtering.

[0255] Optionally, as a possible implementation manner, the fingerprint image data pair is obtained through a two-way generative adversarial network degradation model, wherein the two-way generative adversarial network degradation model is trained based on training data.

[0256] It should be understood that the above-mentioned device 900 is embodied in the form of a functional unit. The term "unit" here can be implemented in the form of software and / or hardware, and this embodiment of the application does not specifically limit this.

[0257] For example, a "unit" may be a software program, a hardware circuit, or a combination of the two to implement the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group processor, etc.) that executes one or more software or firmware programs and a memory, an integrated logic circuit, and / or other suitable devices that can provide the above functions. In a simple embodiment, those skilled in the art can imagine that the device 900 can be used. Figure 2 The form shown.

[0258] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0259] The present application also provides a computer program product, which, when executed by a processor, implements the method described in any method embodiment of the present application.

[0260] The computer program product can be stored in a memory and finally converted into an executable target file that can be executed by a processor through preprocessing, compilation, assembly and linking.

[0261] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, implements the method described in any method embodiment of the present application. The computer program can be a high-level language program or an executable target program.

[0262] The computer-readable storage medium may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0263] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0264] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0265] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0266] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0267] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0268] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory ROM, a random access memory RAM, a magnetic disk, or an optical disk.

[0269] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0270] Additionally, the terms "system" and "network" are often used interchangeably in this document. The term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship. For example, A / B can mean either A or B.

[0271] The terms (or numbers) "first", "second", ... etc. that appear in the embodiments of the present application are only used for descriptive purposes, that is, they are only used to distinguish different objects, such as different "fingerprints", etc., and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first", "second", ... etc. may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, "at least one (item)" refers to one or more. "Multiple" means two or more. "At least one of the following (item)" or similar expressions refers to any combination of these items, including any combination of a single (item) or multiple (items).

[0272] For example, the meaning of expressions similar to "the item includes at least one of the following: A, B, and C" in the embodiments of the present application, unless otherwise specified, generally means that the item can be any one of the following: A; B; C; A and B; A and C; B and C; A, B and C; A and A; A, A and A; A, A and B; A, A and C, A, B and B; A, C and C; B and B, B, B and B, B, B and C, C and C; C, C and C, and other combinations of A, B and C. The above examples use A, B, and C as an example to illustrate the optional items of the item. When the expression is "the item includes at least one of the following: A, B, ..., and X", that is, when the expression has more elements, the items that can be applied to the item can also be obtained according to the above rules.

[0273] In short, the above description is only a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included in the scope of protection of this application.

Claims

1. A fingerprint recognition method, characterized in that: include: receiving a first operation of a user, where the first operation is used to trigger fingerprint recognition; In response to the first operation, obtaining a first fingerprint, and performing fingerprint recognition based on the first fingerprint; When fingerprint recognition fails, repairing the first fingerprint using the first repair model to obtain a second fingerprint; The first restoration model is obtained by performing model training based on fingerprint image data, and a first loss function is determined during the model training process, the first loss function is determined based on directional field information of the restored fingerprint image, and the directional field information is obtained by filtering the restored fingerprint image obtained during the training process; The fingerprint image data pair includes data of a third fingerprint and a fourth fingerprint having a corresponding relationship, and the quality of the fourth fingerprint is better than that of the third fingerprint; The first restoration model includes an input layer, a downsampling layer, an upsampling layer, and an output layer; the input layer is used to perform convolution preprocessing on the input image of the third fingerprint to obtain a preprocessed feature map; the downsampling layer is used to perform downsampling processing on the preprocessed feature map to obtain a downsampled feature map; the upsampling layer is used to perform upsampling processing on the downsampled feature map to obtain the restored fingerprint image and the direction field information of the restored fingerprint image; Fingerprint recognition is performed based on the second fingerprint.

2. The method according to claim 1, characterized in that The direction field information is obtained by filtering the data of the repaired fingerprint image, including: Normalizing the repaired fingerprint image to obtain a normalized fingerprint image; Performing image segmentation on the normalized fingerprint image to obtain a plurality of image blocks; determining, for each image block of the plurality of image blocks, a direction and a frequency of the image block; Spatial filtering is performed based on the direction and frequency of the image block to obtain the direction field information.

3. The method according to claim 2, characterized in that The normalized fingerprint image satisfies the following formula: ； in, Indicates that the mean of the pixels is greater than the normalized result of the prior mean. represents the preset prior mean, represents the prior standard deviation, Indicates that it is greater than the prior mean The coordinates of the pixel point, represents the mean of the restored fingerprint image, Represents the standard deviation of the restored fingerprint image; For pixels smaller than the prior value M, the normalized result is expressed as: in, Indicates that the mean of the pixels is less than the normalized result of the prior mean. represents the preset prior mean, represents the prior standard deviation, Indicates that it is less than the prior mean The coordinates of the pixel point, represents the mean of the restored fingerprint image, Represents the standard deviation of the restored fingerprint image.

4. The method according to any one of claims 1 to 3, characterized in that The first loss function is determined based on a first output result, wherein the first output result includes direction field information of the repaired fingerprint image, and the repaired fingerprint image refers to a repaired fingerprint image obtained when the third fingerprint is input into the training network of the first repair model.

5. The method according to any one of claims 1 to 3, characterized in that A second loss function is also determined during the training process of the first repair model. The second loss function is determined based on the second output result and the fourth fingerprint. The second output result refers to: the result of the repaired fingerprint image obtained when the third fingerprint is input into the training network of the first repair model during the model training process.

6. The method according to any one of claims 1 to 3, characterized in that The direction field information is obtained through Gabor filtering.

7. The method according to any one of claims 1 to 3, characterized in that The fingerprint image data pair is obtained through a two-way generative adversarial network degradation model, wherein the two-way generative adversarial network degradation model is trained based on training data.

8. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the processor and the memory are coupled, and the memory is used to store a computer program. When the computer program is executed by the processor, the electronic device executes the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

10. A chip, characterized in that: The method comprises a processor, and when the processor executes instructions, the processor performs the method according to any one of claims 1 to 7.

Images