Fingerprint image processing method and device, electronic equipment and storage medium

Abstract

The application provides a fingerprint image processing method and device, electronic equipment and storage medium. The fingerprint image processing method comprises: acquiring a plurality of fingerprint sub-images respectively collected by a plurality of pixel units of an optical fingerprint sensor at a current time, and performing fusion processing on the plurality of fingerprint sub-images to obtain a fusion image; determining a first image region corresponding to a position of a first target sub-image in the plurality of fingerprint sub-images from the fusion image, and expanding an edge of the first image region according to the fusion image to obtain a first expanded image; determining an input image according to at least one fingerprint sub-image, and expanding an edge of the input image according to the fusion image to obtain a second expanded image, wherein the at least one fingerprint sub-image includes the first target sub-image; and determining one of the first expanded image and the second expanded image as an output image for fingerprint recognition.

Description

Technical Field

[0001] This application relates to the field of optical fingerprint image processing technology, and in particular to a fingerprint image processing method, apparatus, electronic device and storage medium. Background Technology

[0002] Current under-display ultra-thin optical fingerprint sensors typically employ multi-angle optical paths. This involves designing an array of multiple pixel units, each acquiring fingerprint images from different angles through multi-directional optical channels, thereby maximizing the collection of effective fingerprint signals. However, using multiple fingerprint images to compare against a registered fingerprint template for fingerprint recognition leads to a decrease in recognition speed and increases storage consumption. Conversely, using only one or a few fingerprint images fails to fully utilize the information from all acquired fingerprint images, potentially reducing the success rate of fingerprint recognition. Therefore, a new technical solution is needed to at least partially address these issues. Summary of the Invention

[0003] In view of the above, embodiments of this application provide a fingerprint image processing method, apparatus, electronic device, and storage medium to at least partially solve the above problems.

[0004] According to a first aspect of the embodiments of this application, a fingerprint image processing method is provided, comprising: Multiple fingerprint sub-images collected by multiple pixel units of the optical fingerprint sensor at the current moment are acquired, and the multiple fingerprint sub-images are fused to obtain a fused image; From the fused image, a first image region corresponding to the position of the first target sub-image among the plurality of fingerprint sub-images is determined, and the edges of the first image region are extended according to the fused image to obtain a first extended image; An input image is determined based on at least one fingerprint sub-image, and the edges of the input image are extended based on the fused image to obtain a second extended image, wherein the at least one fingerprint sub-image includes the first target sub-image; One of the first extended image and the second extended image is determined as the output image for fingerprint recognition.

[0005] In some optional embodiments, the first target sub-image is the fingerprint sub-image with the best image quality among the plurality of fingerprint sub-images; determining the first image region corresponding to the position of the first target sub-image among the plurality of fingerprint sub-images from the fused image includes: determining the first image region corresponding to the fingerprint sub-image with the best image quality from the fused image.

[0006] In some optional embodiments, determining the output image for fingerprint recognition from one of the first extended image and the second extended image includes: determining the one with higher image quality from the first extended image and the second extended image as the output image for fingerprint recognition.

[0007] In some optional embodiments, the step of extending the edge of the first image region according to the fused image to obtain a first extended image includes: determining a second image region from the fused image that includes the first image region, wherein the second image region extends beyond at least one edge of the first image region; extracting the second image region from the fused image, and using the extraction result as the first extended image.

[0008] In some optional embodiments, if the four side edges of the first image region do not coincide with the edge of the fused image, then the four side edges of the second image region extend beyond the corresponding four side edges of the first image region; or, if at least one side edge of the first image region coincides with the edge of the fused image, then the second image region extends beyond the side edges of the first image region that do not coincide with the edge of the fused image.

[0009] In some optional embodiments, determining the input image based on at least one fingerprint sub-image includes: locally fusing the overlapping portion between the first target sub-image and a second target sub-image among the plurality of fingerprint sub-images to obtain a target locally fused image, wherein the second target sub-image is any fingerprint sub-image among the plurality of fingerprint sub-images other than the first target sub-image; and stitching the unfused portion of the first target sub-image and the target locally fused image together to form the input image.

[0010] In some optional embodiments, the step of locally fusing the overlapping portion between the first target sub-image and the second target sub-image among the plurality of fingerprint sub-images to obtain a target locally fused image includes: determining target fusion weights corresponding to the first target sub-image and the second target sub-image respectively based on the first local image quality of the first target sub-image in the overlapping portion and the second local image quality of the second target sub-image in the overlapping portion, wherein the higher the local image quality, the higher the determined target fusion weight; and performing a weighted summation of the pixel values ​​of the first target sub-image and the second target sub-image in the overlapping portion based on the target fusion weights corresponding to the first target sub-image and the second target sub-image respectively, so as to locally fuse the overlapping portion between the first target sub-image and the second target sub-image to obtain a target locally fused image.

[0011] In some optional embodiments, the step of extending the edges of the input image based on the fused image to obtain a second extended image includes: determining a second image region including the first image region from the fused image; determining a third image region within the second image region other than the first image region; and correspondingly filling the third image region to the outside of the edges of the input image to form the second extended image.

[0012] In some optional embodiments, if the first target sub-image is the fingerprint sub-image with the best image quality among the plurality of fingerprint sub-images, then the second target sub-image is the fingerprint sub-image with the second best image quality among the plurality of fingerprint sub-images.

[0013] In some optional embodiments, determining the input image based on at least one fingerprint sub-image and extending the edges of the input image based on the fused image to obtain a second extended image includes: determining the first target sub-image as the input image and extending the edges of the first target sub-image based on the fused image to obtain the second extended image.

[0014] In some optional embodiments, the step of extending the edges of the first target sub-image according to the fused image to obtain the second extended image includes: determining a second image region including the first image region from the fused image, determining a third image region other than the first image region within the second image region; and correspondingly filling the third image region outside the edges of the first target sub-image to form the second extended image.

[0015] In some optional embodiments, the step of fusing the plurality of fingerprint sub-images to obtain a fused image includes: determining the highest quality sub-image among the plurality of fingerprint sub-images; for each fingerprint sub-image other than the highest quality sub-image, determining the similarity of the overlapping portion between the fingerprint sub-image and the highest quality sub-image based on reference offset data, and determining the real-time offset between the fingerprint sub-image and the highest quality sub-image based on the similarity; aligning the plurality of fingerprint sub-images based on the real-time offset between each fingerprint sub-image and the highest quality sub-image, and fusing the aligned plurality of fingerprint sub-images to obtain the fused image.

[0016] According to a second aspect of the embodiments of this application, a fingerprint image processing apparatus is provided, comprising: The fusion module is used to acquire multiple fingerprint sub-images collected by multiple pixel units of the optical fingerprint sensor at the current moment, and to fuse the multiple fingerprint sub-images to obtain a fused image; The first extension module is used to determine, from the fused image, a first image region corresponding to the position of a first target sub-image among the plurality of fingerprint sub-images, and to extend the edge of the first image region according to the fused image to obtain a first extended image; The second extension module is used to determine an input image based on at least one fingerprint sub-image, and to extend the edges of the input image based on the fused image to obtain a second extended image, wherein the at least one fingerprint sub-image includes the first target sub-image; The determining module is used to determine one of the first extended image and the second extended image as the output image for fingerprint recognition.

[0017] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory is used to store a computer program; and the processor is used to execute the method provided in the first aspect by running the computer program stored in the memory.

[0018] According to a fourth aspect of the embodiments of this application, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method provided in the first aspect.

[0019] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided in the first aspect.

[0020] According to a sixth aspect of the present application, a fingerprint recognition device is provided, applied to an electronic device having a display screen. The fingerprint recognition device includes: an optical fingerprint sensor for: imaging multiple light signals reflected from a finger above the display screen in multiple different directions to acquire multiple fingerprint sub-images; and a processing unit for executing the method provided in the first aspect.

[0021] According to the fingerprint image processing scheme provided in the embodiments of this application, multiple fingerprint sub-images collected by multiple pixel units of an optical fingerprint sensor can be fused to obtain a fused image. From the fused image, a first image region corresponding to the position of a first target sub-image among the multiple fingerprint sub-images can be determined. The edges of the first image region can be extended according to the fused image to obtain a first extended image. Then, an input image can be determined according to at least one fingerprint sub-image. The edges of the input image can be extended according to the fused image to obtain a second extended image. The at least one fingerprint sub-image includes the first target sub-image. Then, one of the first extended image and the second extended image can be determined as the output image for fingerprint recognition. Therefore, on the one hand, the solution of this application embodiment can use one of the first extended image and the second extended image as the output image for fingerprint recognition, thereby achieving the effect of using a single, better output image for fingerprint recognition when the optical fingerprint sensor acquires multiple fingerprint sub-images at a time. This effectively reduces the storage consumption during fingerprint recognition and also effectively improves the recognition speed and success rate. On the other hand, since both the first extended image and the second extended image are obtained by expanding the fused image, and the fused image better aggregates the information from the acquired multiple fingerprint sub-images, has a larger field of view, and lower spatial noise, using one of them as the output image for fingerprint recognition, compared to directly using the fingerprint sub-images for fingerprint recognition, utilizes the information from the large field of view of the fused image. Therefore, its recognition field of view is larger, its spatial noise is lower, and its signal-to-noise ratio is higher, which is beneficial to improving the recognition success rate. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0023] Figure 1 A flowchart of an optional fingerprint image processing method according to this application is shown.

[0024] Figure 2A The flowchart shows an optional sub-step of step S102, "fusing multiple fingerprint sub-images to obtain a fused image".

[0025] Figure 2B A schematic diagram of the multi-angle optical path of a pixel unit of an optional optical fingerprint sensor according to this application is shown.

[0026] Figure 2CA schematic diagram illustrating the real-time offset calculation between the fingerprint sub-image and the sub-image with the best quality is shown.

[0027] Figure 3A A flowchart of an optional sub-step in sub-step S1022 is shown, which is "determining the similarity of the overlapping portion of the fingerprint sub-image and the best quality sub-image based on the reference offset data, and determining the real-time offset between the fingerprint sub-image and the best quality sub-image at the current moment based on the similarity".

[0028] Figure 3B A flowchart of one optional sub-step for updating the reference offset data is shown.

[0029] Figure 3C A flowchart of an optional sub-step in sub-step S1023, "fusion of multiple aligned fingerprint sub-images to obtain a fused image," is shown.

[0030] Figure 3D A flowchart of an optional sub-step in sub-step S10231 is shown, which involves "fusing the local areas of each fingerprint sub-image in the overlapping region based on the local image quality of each fingerprint sub-image in the overlapping region and the similarity of the overlapping portion of each fingerprint sub-image in the overlapping region with the sub-image of the best quality".

[0031] Figure 3E This diagram illustrates the fusion of local portions of two images.

[0032] Figure 4A The flowchart shows an optional sub-step of step S104, "extending the edges of the first image region according to the fused image to obtain the first extended image".

[0033] Figure 4B A flowchart of an optional sub-step in step S106, "determine the input image based on at least one fingerprint sub-image," is shown.

[0034] Figure 4C An optional sub-step flowchart of sub-step S1061 is shown.

[0035] Figure 5 A schematic diagram showing the expansion direction of the edge of the first image region in the fused image is shown.

[0036] Figure 6 A schematic diagram of local fusion of the overlapping portion of the first target sub-image and the second target sub-image is shown.

[0037] Figure 7 This diagram illustrates how the edges of an input image are expanded to obtain a second expanded image by fusing images.

[0038] Figure 8 A block diagram illustrating the overall implementation of an optional fingerprint image processing scheme of this application is shown.

[0039] Figure 9 A block diagram illustrating the overall implementation of another alternative fingerprint image processing scheme of this application is shown.

[0040] Figure 10 A schematic diagram illustrating the implementation process of the fingerprint image processing scheme of this application is shown.

[0041] Figure 11 A schematic diagram of an exemplary fingerprint image processing apparatus of this application is shown.

[0042] Figure 12 A schematic diagram of an exemplary electronic device of this application is shown.

[0043] Figure 13 A schematic diagram of an exemplary fingerprint recognition device according to this application is shown. Detailed Implementation

[0044] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0045] In the optical path design of ultra-thin under-display optical fingerprint sensors, two extreme signal acquisition states need to be considered. The first is a scenario where the fingerprint fits well against the screen, such as finger recognition at room temperature. In this case, the smaller the light-receiving angle of the signal acquisition optical path, the more likely it is to receive a single fingerprint ridge signal (or fingerprint valley signal), resulting in a higher signal quantity and better recognition. The other scenario is a scenario where the fingerprint does not fit well, such as finger recognition or light pressing at low temperatures. In this case, it is desirable to increase the light-receiving angle of the signal acquisition optical path because the fingerprint is a three-dimensional surface. When light is incident at a large angle, the reflected light from the valley is more easily blocked by the sidewalls of the ridge, thus increasing the difference between the ridge and valley signals and increasing the signal strength. The actual fingerprint pressure signal is distributed between these two extreme cases. To accommodate both scenarios, some current ultra-thin under-display optical fingerprint sensor designs typically employ multi-angle optical paths. This is achieved by designing multiple pixel units arranged in an array. These pixel units acquire fingerprint images from different angles through multi-directional optical channels, thereby maximizing the acquisition of effective fingerprint signals.

[0046] However, using multiple fingerprint images to compare with the registered fingerprint template for fingerprint recognition will lead to a decrease in recognition speed and an increase in storage consumption during fingerprint recognition; while using only one or a few fingerprint images will not fully utilize the information in all the collected fingerprint images, which will easily reduce the success rate of fingerprint recognition.

[0047] In view of this, this application proposes a fingerprint image processing scheme. The specific implementation of the embodiments of this application is further described below with reference to the accompanying drawings.

[0048] Figure 1 A flowchart of an optional fingerprint image processing method according to this application is shown. Figure 1 As shown, the fingerprint image processing method includes steps S102, S104, S106, and S108, specifically: S102: Acquire multiple fingerprint sub-images collected by multiple pixel units of the optical fingerprint sensor at the current moment, and fuse the multiple fingerprint sub-images to obtain a fused image.

[0049] This application does not specifically limit the results of the optical fingerprint sensor. For example, the optical fingerprint sensor can be an under-display optical fingerprint sensor, which can be used to realize the under-display optical fingerprint recognition function of electronic devices. The optical fingerprint sensor can image multiple light signals reflected from the finger above the display screen of the electronic device in multiple different directions to acquire multiple fingerprint sub-images. It should be noted that the fingerprint image detection method of this application can be executed by any processing unit. For example, this method can be executed by the processing unit of the electronic device to which the optical fingerprint sensor is applied (e.g., including but not limited to mobile terminals, computers, fingerprint locks, etc.). This can include one or more chips capable of data processing, including but not limited to CPU (Central Processing Unit), MCU (Microcontroller Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), etc.), which is beneficial to improve the processing effect.

[0050] The optical fingerprint sensor in this application adopts a multi-angle optical path design, which includes multiple pixel units. Each pixel unit may include one or more optical pixels, and multiple pixel units may include the same number of optical pixels. Figure 2B A schematic diagram of the multi-angle optical path of a pixel unit in an optional optical fingerprint sensor is shown. Figure 2B As shown in the diagram, the optical fingerprint sensor includes multiple pixel units (such as... Figure 2B The simplified example uses 9 pixel units (though it should be understood that in practice, there can be more pixel units). Multiple pixel units are equipped with microlens units. When a finger is entered through the optical fingerprint sensor, the reflected light passes through different optical paths of the microlens units and is received by each pixel unit at various angles. Each pixel unit can then generate a fingerprint image using the received light, thus achieving fingerprint image acquisition by each pixel unit. The light received by each pixel unit at various angles represents multiple light signals from different directions. It should be understood that the multiple fingerprint sub-images mentioned in step S102 are fingerprint images acquired separately by multiple pixel units.

[0051] To facilitate the explanation of this embodiment, the following text will use the example of acquiring 9 fingerprint sub-images with 9 pixel units.

[0052] Due to the multi-angle optical path, there is a certain image offset between each pair of fingerprint sub-images. Because of the offset between the fingerprint sub-images, the area of ​​the fused image is larger than that of a single fingerprint sub-image. That is, the field of view of the fused image is larger than that of a single fingerprint sub-image. Therefore, the fused image can aggregate the effective information of multiple fingerprint sub-images. By processing the fused image, an output image for fingerprint recognition is obtained, which allows the output image to utilize the information of the large field of view of the fused image, thereby increasing the recognition field of view of fingerprint recognition.

[0053] This application does not limit the specific implementation of step S102. Optionally, step S102 in this application may involve aligning and offsetting multiple fingerprint sub-images before fusing them to obtain a fused image.

[0054] Optionally, when aligning and offsetting multiple fingerprint sub-images, a preset fixed image offset can be used. This is because in the aforementioned multi-optical fingerprint sensor design, the directions of the multi-directional optical channels are fixed, resulting in approximately stable image offsets between the multiple fingerprint images acquired along each directional optical channel. Therefore, a suitable fixed image offset can be obtained through a special method and preset to facilitate the aligning and offsetting of multiple fingerprint sub-images acquired from multiple pixel units before fusion processing. Other methods can also be used to achieve this, as long as they meet the requirements.

[0055] This application also notes that, since fingers are three-dimensional, factors such as the fingerprint's state (e.g., dry or wet) leading to contact, the pressure state (e.g., light or heavy press) causing slight deformation, and the effective pressure area can easily cause a deviation between the actual offset of the fingerprint images and the fixed image offset. This can lead to a potential deterioration in the quality of the fused image obtained after fusing multiple fingerprint sub-images. To improve this situation, in some optional embodiments of this application, reference is made to... Figure 2AThe flowchart shown includes step S102, which involves fusing multiple fingerprint sub-images to obtain a fused image. This step includes sub-steps S1021-S1023, specifically: S1021: Determine the best quality sub-image among multiple fingerprint sub-images.

[0056] The optimal quality sub-image in this application refers to the fingerprint sub-image with the best image quality among multiple fingerprint sub-images. The optimal quality sub-image can be determined using any suitable method, without limitation. For example, any suitable algorithm can be used to calculate one or more of the following metrics for each fingerprint sub-image: Harris Response (HR), Sobel Response (SR), Peak Signal-to-Noise Ratio (PSNR), and Normalized Stoichiometric Ratio (NSR, which represents the ratio between peak-to-peak value and local mean). Based on the calculation results, the image quality of the multiple fingerprint sub-images can be determined, thus identifying the optimal quality sub-image. Optionally, one or more of the above metrics can be used to score the image quality of each fingerprint sub-image, and the fingerprint sub-image with the highest image quality score can be determined as the optimal quality sub-image.

[0057] The reason these metrics can be used to determine image quality is that they essentially detect the clarity of the bands. Within a small region of a fingerprint sub-image, the fingerprint essentially resembles a periodic band of light and dark signals. The clearer the bands, the larger the normalized gradient (SR), and the greater the fluctuation of the difference between the maximum and minimum values ​​in the neighborhood relative to the maximum or minimum value (corresponding to the peak signal-to-noise ratio PSNR). Similarly, the Harris response (HR) eigenvalues ​​exhibit edge characteristics, and the ratio of the peak signal to the local mean (corresponding to the normalized signal strength NSR) is also larger (the local mean is linearly related to illumination intensity and exposure time; under the same conditions, a good fingerprint will have a larger signal strength, i.e., a higher normalized signal strength NSR). Therefore, determining the image quality of a fingerprint sub-image using one or more of these metrics ensures the reliability of the identified optimal quality sub-image and facilitates subsequent data processing.

[0058] Furthermore, in step S1021 of this application, multiple fingerprint sub-images at the current moment are acquired, and the sub-image with the best quality is determined, which is beneficial for real-time and accurate fingerprint recognition.

[0059] S1022: For each fingerprint sub-image (excluding the highest quality sub-image) among multiple fingerprint sub-images, determine the similarity of the overlapping portion between the fingerprint sub-image and the highest quality sub-image based on the reference offset data, and determine the real-time offset between the fingerprint sub-image and the highest quality sub-image at the current moment based on the similarity. (This sub-step S1022 should be continued in conjunction with...) Figure 2A (Flowchart understanding) In multiple fingerprint sub-images, there will be a certain offset between each pair of images (i.e., image offset). The offset between images is measured in pixels. Offsets can include lateral offsets along the horizontal direction of the image (e.g., the x-direction, which can also be understood as the horizontal direction) and vertical offsets along the vertical direction of the image (e.g., the y-direction, which can also be understood as the vertical direction). For ease of explanation, we will use rightward lateral offset as the positive x-direction, leftward lateral offset as the negative x-direction, downward vertical offset as the positive y-direction, and upward vertical offset as the negative y-direction. In the following text, each type of offset can be denoted as (x, y), where x represents the lateral offset and y represents the vertical offset. A positive x value indicates a lateral offset to the right along the horizontal direction of the image, and a negative x value indicates a lateral offset to the left along the horizontal direction of the image. A positive y value indicates a downward vertical offset along the vertical direction of the image, and a negative y value indicates a downward vertical offset along the vertical direction of the image.

[0060] For example, Figure 2C This diagram illustrates the real-time offset calculation between a fingerprint sub-image and the sub-image of optimal quality. It can also be combined with... Figure 2C As shown, the image is displayed in the x-direction (horizontal) and y-direction (vertical). It should be understood that... Figure 2C The fingerprint sub-images and the highest quality sub-images shown, as well as the overlapping portions between them, are merely examples and are not intended to limit the scope of this application.

[0061] Optionally, the reference offset data in this application includes reference offsets between multiple reference fingerprint sub-images and a reference sub-image of optimal quality, wherein the reference sub-image of optimal quality is acquired by a first pixel unit among multiple pixel units, and the multiple reference fingerprint sub-images are acquired by multiple second pixel units among multiple pixel units, wherein the multiple second pixel units are multiple pixel units other than the first pixel unit among the multiple pixel units. This reference offset data facilitates data processing in this application, making it easier to determine the similarity of the overlapping portions between the fingerprint sub-images and the reference sub-image of optimal quality.

[0062] Before executing the fingerprint image processing method of this application for the first time, the reference offset data can be pre-stored (which can be understood as initialized reference deviation data) and can be retrieved when needed. Optionally, the initialization of the reference offset data can be completed during the mass production stage using striped weights or weights with special patterns (striped weights or weights with special patterns can be used to simulate fingers).

[0063] For example, the process of determining the reference deviation data in this application can be understood by referring to the following procedure: Figure 2B Taking the nine pixel units of an optical fingerprint sensor as an example, during mass production, nine images (e.g., images 1 to 9) of a striped weight can be acquired using these nine pixel units (e.g., pixel unit 1 to pixel unit 9). The image with the best image quality among images 1 to 9 is then selected as the reference image with the best quality (e.g., image 1). The other eight images can be used as reference fingerprint sub-images (i.e., images 2 to 9). Pixel unit 1 is the first pixel unit, and pixel units 2 to 9 are the respective second pixel units. Then, the image offsets of images 2 to 9 relative to image 1 can be directly measured to obtain eight image offsets. These eight image offsets can then be used as eight reference offsets and stored as reference offset data, which can be retrieved directly when needed. It should be understood that this process is merely an example for ease of understanding and does not constitute any limitation on this application.

[0064] This application does not limit the method of similarity calculation. In some optional embodiments, step S1022, "determining the similarity of the overlapping portion of the fingerprint sub-image and the best quality sub-image based on the reference offset data, and determining the real-time offset between the fingerprint sub-image and the best quality sub-image at the current moment based on the similarity," includes sub-steps S10221 and S10222, specifically: S10221: Based on the reference offsets between multiple reference fingerprint sub-images and the reference sub-image with the best quality, calculate the current reference offset between the fingerprint sub-image and the sub-image with the best quality, and determine the real-time offset search range at the current moment based on the current reference offset.

[0065] Since multiple reference fingerprint sub-images and the reference best quality sub-image are also acquired from the same multiple pixel units as the multiple fingerprint sub-images, when executing the fingerprint image processing method of this scheme (for example, when performing fingerprint recognition in practice), if the best quality sub-image is not acquired from the first pixel unit, the current reference offset between each fingerprint sub-image and the best quality sub-image can be calculated by using the reference offset between each of the multiple reference fingerprint sub-images and the reference best quality sub-image.

[0066] For example, let's take the case where 9 images are acquired using 9 pixel units, with pixel unit 1 being the first pixel unit of the reference fingerprint sub-image with the best quality, and pixel units 2-9 being the second pixel units of the reference fingerprint sub-image. For instance, let's take the reference offset between the reference fingerprint sub-image acquired by pixel unit 2 and the reference fingerprint sub-image acquired by pixel unit 1 as (x1, y1). If the fingerprint sub-image with the best quality determined in step S1021 is the fingerprint sub-image acquired by pixel unit 2, then the current reference offset between the fingerprint sub-image acquired by pixel unit 1 and the fingerprint fingerprint sub-image acquired by pixel unit 2 can be converted to (-x1, -y1). Similarly, the current reference offset between the fingerprint sub-images acquired by other pixel units and the fingerprint fingerprint sub-image acquired by pixel unit 2 can also be calculated according to each reference offset. It should be understood that this is merely an example for ease of understanding and is not intended to limit this application.

[0067] After calculating the current reference offset, the real-time offset search range at the current moment can be determined so as to accurately determine the real-time offset corresponding to the fingerprint sub-image within the real-time offset search range.

[0068] In some optional embodiments, the sub-step S10221, "determine the real-time offset search range at the current moment based on the current reference offset", may include: determining the real-time offset search range at the current moment based on the current reference offset and the preset offset margin.

[0069] Based on this, the above embodiments in this application determine the real-time offset search range by using the current reference offset and the preset offset margin, so as to ensure that a suitable real-time offset can be found within the real-time offset search range.

[0070] For example, the preset offset margin includes positive and negative offset margins along the horizontal direction of the image, and positive and negative offset margins along the vertical direction of the image. Optionally, refer to Figure 2C In the example shown, the absolute values ​​of the positive and negative offset margins along the horizontal direction of the image are equal, both being wx; the absolute values ​​of the positive and negative offset margins along the vertical direction of the image are equal, both being wy. For example, the current reference offset can be denoted as (sx, sy), where sx represents the horizontal offset in the current reference offset and sy represents the vertical offset in the current reference offset, combined with... Figure 2C The determined real-time offset search range can then be expressed as: ;in, Of course, the fact that the absolute values ​​of the positive offset margin and the negative offset margin along the horizontal / vertical direction are equal here is a special example. They may not be equal in other alternative embodiments. For ease of explanation, we will use the example of equality below.

[0071] The range corresponding to the preset offset margin within the real-time offset search range can be understood as the neighborhood of the corresponding range of the current reference offset within the real-time offset search range. Therefore, this optional implementation method of the present application can be understood as a "current reference offset + neighborhood search" approach, so as to ensure that a suitable real-time offset is found within the real-time offset search range.

[0072] S10222: Determine the similarity between the overlapping portion of the fingerprint sub-image and the best-quality sub-image corresponding to each offset within the real-time offset search range at the current moment, and use the offset corresponding to the largest similarity as the real-time offset between the fingerprint sub-image and the best-quality sub-image at the current moment. (Please continue to combine this sub-step S10222 with...) Figure 3A (Flowchart understanding) Within the real-time offset search range, there are multiple offsets (each offset includes horizontal and vertical offsets). For each offset, the overlapping portion between the corresponding fingerprint sub-image and the best-quality sub-image may differ. For each offset, the similarity between the overlapping portion of its corresponding fingerprint sub-image and the best-quality sub-image can be calculated, and the offset corresponding to the largest similarity can be used as the real-time offset at the current moment.

[0073] Alternatively, the real-time offset between the fingerprint sub-image and the highest quality sub-image can be calculated using the following formula:

[0074] In the above formula, cx represents the horizontal offset in the real-time offset at the current moment, and cy represents the vertical offset in the real-time offset at the current moment; pic_A represents the fingerprint sub-image, and pic_B represents the sub-image with the best quality; S represents the similarity of the overlapping part of pic_A and pic_B under the offset; sx represents the horizontal offset in the current reference offset, and sy represents the vertical offset in the current reference offset; -wx and wx represent the negative and positive offset margins along the horizontal direction of the image in the preset offset margin, respectively; -wy and wy represent the negative and positive offset margins along the vertical direction of the image in the preset offset margin, respectively. Referring to the above, " , "It can also represent the real-time offset search range (which can be combined with...") Figure 2C (Understanding). The above formula can be understood as follows: the actual offset at the current moment needs to be traversed and searched within the above real-time offset search range to select the offset corresponding to the highest similarity as the final output actual offset at the current moment.

[0075] In this application, the aforementioned similarity can be used to describe the correlation between the overlapping portion of a fingerprint sub-image and the highest quality sub-image, and can be implemented using any suitable index. Optionally, the similarity can be calculated using at least one of the following indices: Normalized Cross Correlation (NCC), Structural Similarity (SSIM), and Mean-Square Error (MSE). Specifically, a higher NCC and SSIM value indicates a higher similarity, while a lower MSE value indicates a higher similarity.

[0076] Based on this, this application can accurately and reliably determine the real-time offset between the fingerprint sub-image and the sub-image with the best quality at the current moment through the optional methods of the above sub-steps S10221~S10222. This facilitates the subsequent steps to achieve the alignment and fusion of multiple fingerprint sub-images through reliable real-time offset, thereby more effectively adapting to the deviations caused by different fingerprint states and pressing states of the three-dimensional finger. This is more conducive to improving the image quality of the fused image obtained after fusing multiple fingerprint sub-images, so as to facilitate subsequent data processing using the fused image.

[0077] In some optional embodiments, the fingerprint image processing method of this application further includes: updating the reference offset data based on the real-time offset between each fingerprint sub-image and the sub-image with the best quality at the current moment.

[0078] Because the offset between the fingerprint sub-images detected by each pixel unit varies slightly with temperature affecting the optical fingerprint sensor, and also changes under extreme conditions such as abnormal drops (e.g., crashes) causing offset of the optical fingerprint sensor module, this application updates the reference offset data. The updated reference offset data can better adapt to the changes in the optical fingerprint sensor under different environments and conditions, thereby making the subsequently determined real-time offset more accurate and reliable, facilitating further data processing.

[0079] In some alternative embodiments, refer to Figure 3B The flowchart shown illustrates how updating the reference offset data includes the following sub-steps S202, S204, and S206, specifically: S202: Based on the real-time offset between each fingerprint sub-image and the best quality sub-image at the current moment, determine the reference real-time offset between the fingerprint sub-images acquired by multiple second pixel units and the fingerprint sub-images acquired by the first pixel unit.

[0080] As mentioned above, the reference offset data may include reference offsets between multiple reference fingerprint sub-images and the reference best quality sub-image. In this application, updating the reference offset data means updating each reference offset. Therefore, it is necessary to convert to the real-time offset between the corresponding reference fingerprint sub-image and the reference best quality sub-image (i.e., the real-time reference offset) for calculation.

[0081] Since the reference quality sub-image is acquired from the first pixel unit and the reference fingerprint sub-image is acquired from the second pixel unit, the real-time offset of each reference can be calculated through each real-time offset.

[0082] For example, let's take the case where 9 images are acquired using 9 pixel units, pixel unit 1 is the first pixel unit of the reference fingerprint sub-image with the best quality, and pixel units 2-9 are the second pixel units of the reference fingerprint sub-image. If the fingerprint sub-image with the best quality determined in step S1021 is the fingerprint sub-image acquired by pixel unit 2, then if the real-time deviation between the fingerprint sub-image acquired by pixel unit 1 (i.e., the first pixel unit) and the fingerprint sub-image with the best quality acquired by pixel unit 2 is (x2, y2), then the reference real-time offset between the fingerprint sub-image acquired by pixel unit 2 and the fingerprint sub-image acquired by pixel unit 1 (i.e., the first pixel unit) can be obtained as (-x2, -y2). Similarly, the reference real-time offsets between the fingerprint sub-images acquired by the other multiple second pixel units (pixel units 3-9) and the fingerprint sub-image acquired by the first pixel unit (pixel unit 1) can also be calculated according to each real-time offset. It should be understood that this is only an example for ease of understanding and is not intended to limit this application.

[0083] S204: For each real-time reference offset, determine the reference offset between the reference fingerprint sub-image acquired by the corresponding second pixel unit and the reference quality optimal sub-image, and perform a weighted sum of the real-time reference offset and the reference offset according to the update weights. (This sub-step S204 should be continued in conjunction with...) Figure 3B (Flowchart understanding) For example, let's take the case where 9 images are acquired using 9 pixel units, pixel unit 1 is the first pixel unit for acquiring the reference fingerprint sub-image with the best quality, and pixel units 2-9 are the second pixel units for acquiring the reference fingerprint sub-image. For the real-time reference offset between the fingerprint sub-image acquired by any pixel unit 2-8 and the fingerprint sub-image acquired by pixel unit 1, taking pixel unit 2 as an example, the reference offset between the reference fingerprint sub-image acquired by pixel unit 2 and the reference fingerprint sub-image with the best quality is determined. The real-time reference offset between pixel unit 2 and pixel unit 1, and the reference offset between the reference fingerprint sub-image acquired by pixel unit 2 and the reference fingerprint sub-image with the best quality are weighted and summed according to the update weights to obtain the weighted summation result for pixel unit 2. Similarly, for pixel units 3-9, the weighted summation can be performed in the same way as for pixel unit 2 to obtain the corresponding weighted summation results. That is, for the 8 real-time reference offsets corresponding to pixel units 2-9, 8 weighted summation results can be obtained. It should be understood that this is only an example for ease of understanding and is not intended to limit this application.

[0084] Optionally, in sub-step S204, the reference real-time offset and the reference offset can be weighted and summed according to the following two formulas to obtain the corresponding weighted summation result:

[0085]

[0086] Where w and 1-w are used to characterize the updated weights, and Cx represents the lateral offset in the real-time reference offset, and sx(t) and sx(t-1) are used to represent the lateral offsets in the reference offset at two different times. Corresponding to the above embodiment, sx(t) can be understood as the lateral offset in the updated reference offset, and sx(t-1) can be understood as the lateral offset in the reference offset before the update.

[0087] Where Cy represents the longitudinal offset in the real-time reference offset, and sy(t) and sy(t-1) are used to represent the longitudinal offsets in the reference offsets at two different times. Corresponding to the above embodiment, sy(t) can be understood as the longitudinal offset in the updated reference offset, and sy(t-1) can be understood as the longitudinal offset in the reference offset before the update.

[0088] The horizontal and vertical offsets in the updated reference offset are calculated using the two formulas above. This yields the updated reference offset between a reference fingerprint sub-image and the reference sub-image with the best quality, which is also a weighted summation result.

[0089] It should be understood that the updated reference offset can be different by using different update weights for weighted summation. Optionally, the update weights w and 1-w can be preset values ​​(which can be understood as initial update weights) before the fingerprint image processing method of this scheme is executed for the first time.

[0090] S206: Update the reference offset data based on the weighted summation results corresponding to each real-time reference offset. (Please continue to combine this sub-step S206 with...) Figure 3B (Flowchart understanding) Each weighted summation result is an updated reference offset between a reference fingerprint sub-image and the reference sub-image with the best reference quality. Based on each updated reference offset, each updated reference offset is used as the updated reference offset data, thereby realizing the update of the reference offset data.

[0091] Based on this, the optional implementation method including the above sub-steps S202 to S206 in this application can effectively and reliably update the reference offset data. The updated reference offset data can better adapt to the changes of the optical fingerprint sensor in different environments and states, thereby making the subsequently determined real-time offset more accurate and reliable, so as to facilitate further data processing.

[0092] In some optional embodiments, the fingerprint image processing method of this application further includes: adjusting the update weight based on at least one of the image quality of the fingerprint sub-image corresponding to the reference real-time offset and the similarity of the overlapping portion of the fingerprint sub-image corresponding to the reference real-time offset and the sub-image with the best quality.

[0093] In this application, the update weight is adjusted based on at least one of the image quality of the fingerprint sub-image corresponding to the reference real-time offset and the similarity of the overlapping part of the fingerprint sub-image corresponding to the reference real-time offset and the sub-image with the best quality, so as to better update the reference offset data. This allows the reference offset data to better adapt to the changes of the optical fingerprint sensor in different environments and states, making the subsequently determined real-time offset more accurate and reliable, so as to facilitate further data processing.

[0094] The image quality of fingerprint sub-images can be determined by calculating one or more of the following metrics: Harris Response (HR), Normalized Gradient (SR), Peak Signal-to-Noise Ratio (PSNR), and Normalized Signal-to-Rate (NSR). The image quality of the fingerprint sub-image is then determined based on the calculation results. Similarity can be calculated using at least one of the following similarity metrics: Normalized Cross-Correlation (NCC), Structural Similarity (SSIM), and Mean-Square Error (MSE). These related topics have been similarly introduced earlier and will not be repeated here.

[0095] Optionally, in some alternative embodiments, the better the image quality of the fingerprint sub-image corresponding to the reference real-time offset, the higher the adjusted update weight. This makes the reliability of the adjusted and calculated update weight higher.

[0096] Alternatively, in some other embodiments, the higher the similarity between the overlapping portion of the fingerprint sub-image corresponding to the reference real-time offset and the best-quality sub-image, the higher the adjusted update weight. This can also make the reliability of the adjusted calculated update weight higher.

[0097] Optionally, in some further embodiments, the update weights can be adjusted based on the image quality of the fingerprint sub-image corresponding to the reference real-time offset when the similarity between the overlapping portion of the fingerprint sub-image corresponding to the reference real-time offset and the best-quality sub-image exceeds a certain similarity threshold. Specifically, the better the image quality of the fingerprint sub-image corresponding to the reference real-time offset, the higher the adjusted update weights. This can also improve the reliability of the updated weights calculated after adjustment.

[0098] Optionally, when adjusting the update weight in this application, the update weight can be adjusted within a preset update weight range. The update weight range can be preset as needed. For example, the update weight range of the above-mentioned update weight w can be, for example, 0.2~0.7, 0.1~0.6, etc. It can be understood that the corresponding 1-w can change with the change of w.

[0099] S1023: Based on the real-time offset between each fingerprint sub-image and the highest quality sub-image at the current moment, align multiple fingerprint sub-images, and then fuse the aligned sub-images to obtain a fused image. (This sub-step S1023 should be continued in conjunction with...) Figure 2A (Flowchart understanding) Optionally, for each fingerprint sub-image other than the best-quality sub-image, the overlapping portion between the fingerprint sub-image and the best-quality sub-image can be aligned using its corresponding real-time offset (including lateral and longitudinal offsets) at the current moment. This alignment allows multiple fingerprint sub-images to be aligned. The aligned sub-images can then be fused to obtain a fused image, which better integrates information from the acquired fingerprint sub-images, has a larger field of view, and lower spatial noise. This facilitates the processing of the fused image to obtain an output image for fingerprint recognition in real-time.

[0100] In this application, since there is an offset between the fingerprint sub-images, the area of ​​the fused image is larger than that of a single fingerprint sub-image. That is, the field of view of the fused image is larger than that of a single fingerprint sub-image. Therefore, by processing the fused image to obtain the output image for fingerprint recognition, the effective information of multiple fingerprint sub-images can be better aggregated, so that the output image can utilize the information of the large field of view of the fused image to increase the recognition field of fingerprint recognition.

[0101] In some alternative embodiments, refer to Figure 3C The flowchart shown includes steps S1023, specifically "fusion of the aligned fingerprint sub-images to obtain a fused image," which comprises sub-steps S10231 and S10232. S10231: For any overlapping region of the aligned multiple fingerprint sub-images, based on the local image quality of each fingerprint sub-image in the overlapping region and the similarity between the overlapping portion of each fingerprint sub-image in the overlapping region and the sub-image with the best quality, the local fingerprint sub-images in the overlapping region are fused to obtain a local fusion result.

[0102] In this application, the aligned fingerprint sub-images may have multiple overlapping regions, and each overlapping region may include local contents of multiple fingerprint sub-images.

[0103] In sub-step S10231 of this application, the reason for fusing the local images of each fingerprint sub-image within the overlapping region based on the local image quality of each fingerprint sub-image within the overlapping region and the similarity of the overlapping portion of each fingerprint sub-image within the overlapping region with the highest quality sub-image is that: the local image quality of a fingerprint sub-image describes the clarity of a local area of ​​the fingerprint sub-image; the clearer the local area of ​​the fingerprint sub-image, the better the local image quality (it should be understood that the local image quality score in the following text is also higher); and the similarity of the overlapping portion of the fingerprint sub-image with the highest quality sub-image describes the pattern consistency between the fingerprint sub-image and the highest quality sub-image; after aligning multiple fingerprint sub-images, the more consistent the patterns at the same position, the higher the similarity. Therefore, in this application, when fusing the local images of each fingerprint sub-image within the overlapping region, multiple dimensions are comprehensively considered. First, the local image quality mentioned above is considered to ensure that the output local fusion result includes as many clear areas as possible from each fingerprint sub-image, avoiding excessive inclusion of blurred areas from each fingerprint sub-image, which would lead to a blurred output local fusion result and, consequently, a blurred fused image. Second, the similarity mentioned above is considered to suppress areas in each fingerprint sub-image where the ridge pattern does not match the ridge pattern of the highest quality sub-image, thus preventing image ridge distortion and blurring in the output local fusion result and, consequently, image ridge distortion and blurring in the fused image.

[0104] In some alternative embodiments, refer to Figure 3D The flowchart shown illustrates that step S10231, "based on the local image quality of each fingerprint sub-image within the overlapping region and the similarity of the overlapping portion of each fingerprint sub-image within the overlapping region with the sub-image of the best quality, fuses the local areas of each fingerprint sub-image within the overlapping region," includes sub-steps S10231A, S10231B, and S10231C, specifically: S10231A: Determine the local image quality score of each fingerprint sub-image within the overlapping region.

[0105] Optionally, for any overlapping region, one or more of the following metrics can be used to perform local image quality scoring on each fingerprint sub-image using an algorithm, such as Harris Response (HR), Normalized Gradient (SR), Peak Signal-to-Noise Ratio (PSNR), and Normalized Signal-to-Rate (NSR), to obtain the local image quality score for each fingerprint sub-image within the overlapping region. These related concepts have been similarly introduced earlier and will not be repeated here.

[0106] Next, the fingerprint sub-images within the overlapping region can be sorted according to their local image quality scores. This requires calculating the scores for all fingerprint sub-images within the overlapping region (including any sub-image with the highest quality). For example, we can assume that the local image quality scores of the sorted fingerprint sub-images, from highest to lowest, are: Q1, Q2, Q3, ..., Q... n Correspondingly, the fingerprint sub-images corresponding to the above sorting can be assumed to be: P1, P2, P3, ..., P n .

[0107] S10231B: Determine the fusion weights corresponding to each fingerprint sub-image in the overlapping region based on the obtained quality scores of each local image and the similarity between the overlapping parts of each fingerprint sub-image in the overlapping region and the sub-image with the best quality.

[0108] For example, the similarity between the overlapping portions of each fingerprint sub-image within the overlapping region and the best-quality sub-image (as mentioned above, similarity can be calculated using at least one of the following indicators: correlation NCC, structural similarity SSIM, mean squared error MSE, etc., where a larger correlation NCC and structural similarity SSIM indicate greater similarity, while a smaller mean squared error MSE indicates greater similarity), can be assumed to be S1, S2, S3, ..., S... n It should be noted that this calculation needs to be performed on all fingerprint sub-images within the overlapping region (including any sub-image of optimal quality). Optionally, the similarity of the sub-image of optimal quality relative to itself can be defined as 100.

[0109] Then, based on the quality scores of each local image, Q1, Q2, Q3, ..., Q... n And each similarity S1, S2, S3, ..., S n Identify the fingerprint sub-images P1, P2, P3, ..., P within the overlapping region. n The corresponding fusion weights. For example, in the case of the above 9 fingerprint sub-images, if the local content of all 9 fingerprint sub-images exists in a certain overlapping area, then n=9; if t (t<9) of the 9 fingerprint sub-images exist in a certain overlapping area, then n=t.

[0110] S10231C: The local pixel values ​​of each fingerprint sub-image are weighted and summed according to the obtained fusion weights to fuse the local areas of each fingerprint sub-image in the overlapping region.

[0111] The local fusion result is a local image. The pixel values ​​of each fingerprint sub-image can be weighted and summed using the obtained fusion weights to obtain the pixel values ​​of each pixel in the local fusion result. This achieves the purpose of fusing the local areas of each fingerprint sub-image in the overlapping region to obtain the local fusion result.

[0112] Based on this, the optional implementation method including the above sub-steps S10231A to S10231C in this application can reliably and effectively fuse the local areas of each fingerprint sub-image in the overlapping area, thereby obtaining a reliable local fusion result, so as to facilitate the subsequent generation of a fused image.

[0113] In some optional embodiments, if the local image quality score corresponding to the fingerprint sub-image is higher and the similarity is greater, then the fusion weight corresponding to the determined fingerprint sub-image is higher.

[0114] In this application, by determining higher fusion weights for fingerprint sub-images with higher local image quality scores and greater similarity, the image quality of the fused local fusion result can be effectively and better guaranteed. This can reduce the noise brought by fingerprint sub-images with poor local image quality during local image fusion, and help avoid the situation where the image quality of the fused local fusion result is not as good as the best quality sub-image before fusion.

[0115] It is understood that, in the optional implementation of this application, when fusing the local areas of each fingerprint sub-image within the overlapping region, firstly, the output local fusion result can be made to include the clear local areas of each fingerprint sub-image with as high a weight as possible, avoiding the inclusion of blurred local areas of each fingerprint sub-image with high weight, which would lead to a blurred output local fusion result, and thus avoid blurring of the subsequently obtained fused image; secondly, areas in each fingerprint sub-image where the ridge pattern is inconsistent with the ridge pattern direction of the best quality sub-image can be suppressed, avoiding image ridge pattern distortion and blurring of the output local fusion result, and thus avoiding image ridge pattern distortion and blurring of the subsequently obtained fused image.

[0116] For example, you can refer to Figure 3E To understand the effects of the aforementioned technologies. Figure 3E This illustrates a schematic diagram of fusing portions of two images. (For example...) Figure 3EThe image shows nine overlapping regions between two images (Image A and Image B) (a1 and b1 overlap, a2 and b2 overlap, ..., a9 and b9 overlap). In some overlapping regions, the local image quality score of Image A is higher than that of Image B (e.g., a1 is better than b1, a4 is better than b4, a5 is better than b5, a7 is better than b7). In other overlapping regions, the local image quality score of some regions in Image B is better than that of Image A (e.g., b2 is better than a2, b3 is better than a3, b6 is better than a6, b8 is better than a8, b9 is better than a9). Image C shows the image obtained by directly superimposing and fusing Images A and B within each overlapping region. Each overlapping region of Images A and B is directly fused with a 50% fusion weight. The resulting image C has a local image quality score lower than the image with the best local image quality in the corresponding overlapping region. Image D shows the image obtained by fusing local areas of image A and image B within overlapping regions, using a higher fusion weight based on the local image quality score. (It should be noted that, for easier understanding, image D illustrates a special case where local areas with higher quality scores are fused with a 100% fusion weight, allowing any local area of ​​image A or image B with a higher quality score within the overlapping region to be fully incorporated into image D.) Therefore, image D has better image quality than image C. In other words, assigning higher fusion weights to images with higher local image quality scores within overlapping regions effectively and better guarantees the image quality of the fused local image, thus reflecting the aforementioned technical effect. It should be understood that regarding... Figure 3E The description is not intended to limit this application.

[0117] Optionally, the above sub-step S10231C can be implemented in this application by the following formula:

[0118] Where BP is used to represent the local fusion result, P i Used to represent the various fingerprint sub-images (P1, P2, P3, ..., P) within the overlapping area. n When substituting into the calculation, it can be calculated using the pixel values ​​of the pixels; Q i Used to represent local image quality scores (Q1, Q2, Q3, ..., Q...). n Q1 is the maximum value among the local image quality scores; S i Used to represent fingerprint image P i Similarity (S1, S2, S3, ..., S) of the overlapping portion with the highest quality sub-image nS1 represents the similarity of fingerprint sub-image P1 with a local image quality score of Q1; n is the total number of fingerprint sub-images in the overlapping region.

[0119] As can be seen from the above formula, for any fingerprint sub-image P within the overlapping region i Its fusion weight is: Furthermore, it can be seen that the fingerprint sub-image P i The corresponding local image quality score Q i The higher the similarity S i The larger the value, the higher the value of the determined fingerprint sub-image P. i The higher the corresponding fusion weight, the better the image quality of the local fusion result.

[0120] When calculating using the above formula, each fingerprint sub-image P can be... i By substituting the pixel values ​​of the corresponding pixels in the overlapping region, a weighted summation operation is performed, resulting in the pixel values ​​of each pixel in the local fusion result. This allows for the local fusion of various fingerprint sub-images within the overlapping region.

[0121] S10232: The local fusion results corresponding to the overlapping regions of the aligned fingerprint sub-images are stitched together to obtain the fused image. (This sub-step S10232 should be continued in conjunction with...) Figure 3C (Flowchart understanding) After obtaining the local fusion results corresponding to each overlapping region, these local fusion results can be stitched together to form a large fused image, i.e., a fused image. Then, the fused image can be further processed in subsequent steps S104~S108 to obtain an output image for fingerprint recognition.

[0122] Based on this, this application employs an optional implementation method including the aforementioned sub-steps S10231~S10232, which simultaneously considers the local image quality of each fingerprint sub-image in the overlapping region and its similarity to the sub-image with the best quality, to perform local fusion of the fingerprint sub-images in the overlapping region. This makes the fused image formed by stitching together the results of each local fusion more reliable, and allows the obtained fused image to better aggregate the effective information of multiple fingerprint sub-images. Furthermore, the fused image is processed to obtain an output image for fingerprint recognition, enabling the output image to better utilize the large field of view information of the fused image. This results in a larger recognition field of view, lower spatial noise, and a higher signal-to-noise ratio for fingerprint recognition in the output image, which is beneficial for improving the recognition success rate.

[0123] It should also be noted that in other optional embodiments, when "for any overlapping region of the aligned multiple fingerprint sub-images, the local areas of each fingerprint sub-image within the overlapping region are fused to obtain a local fusion result," it can also be implemented based solely on the local image quality of each fingerprint sub-image within the overlapping region, or solely on the similarity of the overlapping portion of each fingerprint sub-image within the overlapping region with the sub-image of the best quality. That is, the fusion weight can also consider only the local image quality or similarity, as long as it meets the requirements.

[0124] This application, through the optional implementation of the above sub-steps S1021~S1023, can more accurately determine and realize the alignment and image fusion of multiple fingerprint sub-images based on the real-time offset of each fingerprint sub-image. Therefore, it can more effectively adapt to the deviations caused by different fingerprint states and pressing states of the three-dimensional finger, which is conducive to improving the quality of the fused image obtained after the fusion of multiple fingerprint sub-images, so as to facilitate the output image obtained by subsequent processing based on the fused image. Furthermore, since this scheme processes the fused image obtained by fusion of multiple fingerprint sub-images after alignment based on each real-time offset to obtain the output image for fingerprint recognition, it can better aggregate the effective information of multiple fingerprint sub-images, so that the output image can better utilize the large field of view information of the fused image, making the fingerprint recognition field of view of the output image larger, the spatial noise lower, and the signal-to-noise ratio higher, which is conducive to improving the recognition success rate.

[0125] S104: From the fused image, determine the first image region corresponding to the position of the first target sub-image among multiple fingerprint sub-images, and extend the edges of the first image region according to the fused image to obtain the first extended image. (Please continue to combine this step S104 with...) Figure 1 (Flowchart understanding) In this application, after obtaining a fused image with a large image size, a first image region corresponding to the location of the first target sub-image can be determined from the fused image. The first target sub-image can be any one of multiple fingerprint sub-images. The first image region and the first target sub-image have the same size.

[0126] Then, the edges of the first image region can be expanded based on the fused image to obtain a first expanded image. Because it has been expanded, the size of the first expanded image is larger than that of the first image region, and its recognition field of view is larger relative to a single fingerprint sub-image and the first image region.

[0127] Optionally, the first target sub-image can be the fingerprint sub-image with the best image quality among multiple fingerprint sub-images. That is, the first target sub-image can be the sub-image with the best quality. Therefore, step S104 above can be to determine the first image region corresponding to the fingerprint sub-image with the best image quality among multiple fingerprint sub-images from the fused image.

[0128] Based on this, the image quality of the first extended image obtained subsequently can be effectively improved, as can the image quality of the second extended image obtained by extending the edges of the input image. This ensures the image quality of the output image used for fingerprint recognition, and thus effectively improves the recognition success rate of fingerprint image recognition.

[0129] In other alternative embodiments, the first target sub-image can also be selected from multiple fingerprint sub-images, choosing the one with the second-best image quality. Alternatively, it can be any other fingerprint sub-image that meets the requirements.

[0130] As previously mentioned, the optimal sub-image quality can be determined using any suitable method, without limitation. For example, any suitable algorithm can be used to calculate one or more of the following metrics for each fingerprint sub-image: Harris Response (HR), Sobel Response (SR), Peak Signal-to-Noise Ratio (PSNR), and Normalized Stoichiometric Ratio (NSR, representing the ratio between peak-to-peak value and local mean). Based on the calculation results, the image quality of multiple fingerprint sub-images can be determined to identify the optimal sub-image quality. Optionally, one or more of the above metrics can be used to score the image quality of each fingerprint sub-image, and the fingerprint sub-image with the highest image quality score can be identified as the optimal sub-image quality, thus making it the first target sub-image. Furthermore, the fingerprint sub-image with the second highest image quality score is the second-best fingerprint sub-image quality (which may also be referred to as the second-best sub-image quality below). In addition, as previously mentioned, multiple fingerprint sub-images can be sorted according to image quality. The relevant content can be understood by referring to the previous text, and will not be repeated here.

[0131] This application does not limit the specific method of "extending the edges of the first image region according to the fused image to obtain the first extended image" in step S104. For example, in some optional embodiments, refer to Figure 4A The flowchart shown may include sub-steps S1041 and S1042, specifically: S1041: Determine a second image region from the fused image, including a first image region, wherein the second image region extends beyond at least one edge of the first image region.

[0132] In this application, after determining a first image region from the fused image, at least one edge of the first image region can be extended outward, so that the first image region is expanded into a larger second image region in the fused image. In this optional embodiment, the second image region can also be understood as the ROI (Region of Interest) in the fused image.

[0133] It is understood that both the fingerprint sub-image and the first image region include four side edges. In this application, when determining the second image region, which includes the first image region, from the fused image, it can be extended to one or more side edges of the first image region as needed, as long as the requirements are met.

[0134] The size of the image expansion should be chosen to meet the performance requirements of the recognition system. It should be understood that larger expansion sizes are not always better; the ideal size is one that meets the requirements for recognition success rate and speed. For example, for some blurred, fused images that affect recognition success rate, excessively large expansion sizes may lead to mismatches in texture, resulting in a decrease in recognition success rate. Similarly, large images generally have slower recognition speeds, so excessively large expansion sizes may also reduce recognition speed. Therefore, an appropriate expansion size should be selected based on actual needs.

[0135] For example, in some optional embodiments, if the four side edges of the first image region do not coincide with the edges of the fused image, then the four side edges of the second image region extend beyond the corresponding four side edges of the first image region.

[0136] In this situation, you can refer to Figure 5 As shown in Figure a, the first image region corresponding to the first target sub-image can be located in the middle of the fused image. Its four side edges do not coincide with the edges of the fused image. Therefore, the four side edges of the first image region can be extended in four directions in the fused image (the arrows indicate the directions of extension) to determine the second image region.

[0137] For example, in some alternative embodiments, if at least one edge of the first image region coincides with the edge of the fused image, then the second image region extends beyond the edges of the first image region that do not coincide with the edge of the fused image.

[0138] In this situation, you can refer to Figure 5As shown in Figures b and c, the first image region corresponding to the first target sub-image can be located at the corner of the fused image. Its two side edges do not coincide with the edges of the fused image. Therefore, the other two side edges of the first image region can be extended in the fused image in two other directions (the arrows indicate the direction of extension, as shown in Figure b, the example can be extended downwards and to the right; the example in Figure c can be extended upwards and to the left), thereby determining the second image region.

[0139] Of course, if the first image region corresponding to the first target sub-image is located at one edge of the fused image, that is, when only one edge of the first image region coincides with one edge of the fused image (e.g., coincides with the upper edge of the fused image), then the other three edges of the first image region can be extended in the fused image in three other directions (e.g., downward, left, and right) to determine the second image region.

[0140] Based on this, the above-mentioned alternative solutions can adapt to the different positional states of the first image regions in the fused image, effectively determine the second image regions including the first image regions, and effectively realize the extension of the edges of the first image regions according to the fused image.

[0141] It should be understood that when extending the edges of the first image region, the edges of the first image region can be extended to the four edges of the merged image at most; that is, the entire merged image is defined as the second image region. It should also be understood that when extending the edges of the first image region, the extension size on each side can be the same or different, as long as the requirements are met. For example, using… Figure 5 In Figure a, the first image region can extend to the left and right edges by a size that is slightly larger than the top and bottom edges. Of course, this is just an example and not a limitation of this application.

[0142] S1042: Extract the second image region from the fused image and use the extracted result as the first extended image.

[0143] After determining the second image region, which includes the first image region, from the fused image, the second image region can be extracted from the fused image, and the result of the extraction is the first extended image.

[0144] Combination Figure 5 Figure ac illustrates three examples of extracting a second image region from the fused image to obtain a first expanded image. It should be understood that this is not intended to limit the scope of this application.

[0145] Based on this, the optional implementation method including the above sub-steps S1041 to S1042 in this application can effectively realize the expansion of the edge of the first image region to obtain the first expanded image, so as to facilitate the subsequent determination of the output image for fingerprint recognition.

[0146] S106: Determine the input image based on at least one fingerprint sub-image, and extend the edges of the input image based on the fused image to obtain a second extended image, wherein the at least one fingerprint sub-image includes a first target sub-image. (Please continue with this step S106) Figure 1 (Flowchart understanding) In this application, the input image is determined based on at least one fingerprint sub-image, including a first target sub-image, and has the same size as the fingerprint sub-image. In step S106, the size of the second extended image obtained by expanding the edges of the input image based on the fused image is the same as the size of the first extended image obtained in step S104 above.

[0147] This application does not limit the method for determining the input image. For example, in some optional embodiments, refer to... Figure 4B The flowchart shown includes the following sub-steps S106, "determine the input image based on at least one fingerprint sub-image", which specifically includes the following sub-steps S1061 and S1062: S1061: Locally fuse the overlapping portion between the first target sub-image and the second target sub-image among multiple fingerprint sub-images to obtain a target local fusion image.

[0148] S1062: The input image is formed by stitching together the unfused parts of the first target sub-image and the target locally fused image.

[0149] Based on this, this application obtains a target locally fused image by locally fusing the overlapping portion of the first target sub-image and the second target sub-image. Then, the input image is stitched together with the unfused portion of the first target sub-image and the target locally fused image, thus effectively obtaining the input image. This facilitates subsequent edge expansion based on the input image to obtain the second expanded image. Furthermore, using two fingerprint sub-images to generate the input image reduces the computational load while maintaining the quality of the input image.

[0150] Reference Figure 6 The diagram illustrates the process of locally fusing the overlapping portion between the first and second target sub-images to generate a locally fused target image and an input image. It can be seen that the input image includes the locally fused target image (i.e., the result of the local fusion) and the portion of the first target sub-image that was not locally fused. It should be understood that this is merely an example for ease of understanding.

[0151] In this application, the second target sub-image can be any fingerprint sub-image other than the first target sub-image among multiple fingerprint sub-images, and can be selected as needed.

[0152] For example, in some optional embodiments, if the first target sub-image is the fingerprint sub-image with the best image quality among a plurality of fingerprint sub-images (i.e., the best quality sub-image), then the second target sub-image is the fingerprint sub-image with the second best image quality among the plurality of fingerprint sub-images (i.e., the second best quality sub-image).

[0153] Based on this, by locally fusing the overlapping portion of the best-quality sub-image and the second-best-quality sub-image, the image quality of the input image can be better guaranteed, thereby improving the image quality of the second extended image obtained by expanding the input image.

[0154] In some optional embodiments, sub-step S1061 may involve aligning the offsets of the overlapping portions between the first target sub-image and the second target sub-image before performing local fusion to obtain a target locally fused image. This is beneficial for adapting to deviations caused by different fingerprint states and pressing states on a three-dimensional surface, and for improving the effect of locally fusing the overlapping portions between the first target sub-image and the second target sub-image, thereby ensuring the quality of the obtained input image and ensuring a better effect for the subsequent obtained second extended image.

[0155] In some alternative embodiments, refer to Figure 4C The flowchart shown may include sub-steps S1061A and S1061B, specifically: S1061A: Based on the first local image quality of the first target sub-image in the overlapping part and the second local image quality of the second target sub-image in the overlapping part, determine the target fusion weights corresponding to the first target sub-image and the second target sub-image respectively, and if the local image quality is higher, the determined target fusion weight is higher.

[0156] In this application, the higher the local image quality of the first target sub-image and the second target sub-image, the higher the target fusion weight. This can ensure that the target local fusion image obtained by subsequent local fusion can retain more features of the image with higher local image quality, thereby improving the effect of local fusion.

[0157] Optionally, local image quality scores can be calculated for the first target sub-image and the second target sub-image in the overlapping region, respectively. The calculated local image quality scores can represent the quality of the first local image and the quality of the second local image, respectively. Based on the local image quality scores corresponding to the two, target fusion weights are determined for each. The higher the local image quality score, the higher the determined target fusion weight.

[0158] It should be understood that when calculating the local image quality score, one or more of the following metrics can be used: Harris Response (HR), Normalized Gradient (SR), Peak Signal-to-Noise Ratio (PSNR), and Normalized Signal Scale (NSR). Similar content has been introduced previously and will not be repeated here.

[0159] The target fusion weights corresponding to the first and second target sub-images can be determined in any suitable manner. Optionally, assuming the local image quality score (i.e., the first local image quality) of the first target sub-image in the overlapping region is 'a', and the local image quality score (i.e., the second local image quality) of the second target sub-image in the overlapping region is 'b', then the target fusion weight corresponding to the first target sub-image can be determined as a / (a+b), and the target fusion weight corresponding to the second target sub-image as b / (a+b). This also ensures that the higher the local image quality, the higher the determined target fusion weight.

[0160] S1061B: Based on the target fusion weights corresponding to the first target sub-image and the second target sub-image respectively, the pixel values ​​of the first target sub-image and the second target sub-image in the overlapping part are weighted and summed to locally fuse the overlapping part between the first target sub-image and the second target sub-image to obtain a target local fusion image.

[0161] Specifically, by using the target fusion weights corresponding to the first target sub-image and the second target sub-image respectively, the pixel values ​​of the first target sub-image and the second target sub-image in the overlapping part are weighted and summed to obtain the pixel values ​​of each pixel in the target local fusion image, thereby achieving local fusion and obtaining the target local fusion image.

[0162] Based on this, through the optional implementation including the above sub-steps S1061A~S1061B, the overlapping portion between the first target sub-image and the second target sub-image can be effectively fused to obtain a target locally fused image, so as to facilitate the determination of the input image in sub-step S1062; and since the higher the local image quality of the first target sub-image and the second target sub-image, the higher the target fusion weight, it can be ensured that the target locally fused image obtained by subsequent local fusion can retain more features of the image with higher local image quality, thereby improving the effect of local fusion. (Similarly, the technical effect here can also be combined with the previous text regarding...) Figure 3E (The explanation is provided below and will not be repeated here.) In some optional embodiments, step S106, "extending the edges of the input image according to the fused image to obtain a second extended image", includes: determining a second image region including a first image region from the fused image, determining a third image region other than the first image region within the second image region, and filling the third image region to the outside of the edges of the input image to form a second extended image.

[0163] In this application, by correspondingly filling the third image region (other than the first image region) within the second image region to the edge of the input image, the edge of the input image can be extended quickly and effectively based on the fused image to obtain a second extended image, so as to facilitate the subsequent determination of the output image for fingerprint recognition.

[0164] For example, refer to Figure 7 The diagram illustrates the position of a third image region in the fused image. It can be seen that the third image region corresponds to the portion of the first expanded image that extends the first image region. By correspondingly filling the edges of the input image with this third image region, edge expansion can be achieved to obtain the second expanded image. Since the input image, the first target sub-image, and the first image region all have the same size, the second expanded image has the same size as the first expanded image.

[0165] In some alternative embodiments, step S106 includes: determining a first target sub-image as an input image, and expanding the edges of the first target sub-image according to the fused image to obtain a second expanded image.

[0166] In this application, through such an optional embodiment, the first target sub-image is directly determined as the input image, thereby effectively obtaining the input image, so that the edge is subsequently extended based on the input image to obtain the second extended image.

[0167] Alternatively, the first target sub-image here can also be the fingerprint sub-image with the best image quality among multiple fingerprint sub-images (i.e., the highest quality sub-image). It can be determined based on the first target sub-image used by the first extended image.

[0168] In some optional embodiments, the above-mentioned "extending the edge of the first target sub-image according to the fused image to obtain a second extended image" includes: determining a second image region including the first image region from the fused image, determining a third image region other than the first image region within the second image region; and filling the third image region to the outside of the edge of the first target sub-image to form the second extended image.

[0169] In this application, by filling the edge of the first target sub-image (i.e., the input image) with the third image region other than the first image region in the second image region, the edge of the first target sub-image (i.e., the input image) can be extended quickly and effectively according to the fused image to obtain the second extended image, so as to facilitate the subsequent determination of the output image for fingerprint recognition.

[0170] For example, still refer to Figure 7 The diagram illustrates the position of a third image region in the fused image. It can be seen that the third image region corresponds to the portion of the first expanded image that extends the first image region. By correspondingly filling the edge of the input image (the first target sub-image) with this third image region, edge expansion can be achieved to obtain the second expanded image. Since the input image, the first target sub-image, and the first image region all have the same size, the second expanded image has the same size as the first expanded image.

[0171] Optionally, when filling the third image region to the edge of the first target sub-image, the boundary between the edges of the third image region and the first target sub-image can be fused to avoid boundary effects between them.

[0172] S108: Select one of the first extended image and the second extended image as the output image for fingerprint recognition. (Please continue with step S108...) Figure 1 (Flowchart understanding) After obtaining the first extended image and the second extended image, one of the first extended image and the second extended image can be determined as the output image so that fingerprint recognition can be performed using the output image.

[0173] Based on this, through the optional implementation of steps S102 to S108 above, this application can, on the one hand, use one of the first extended image and the second extended image as the output image for fingerprint recognition, thereby achieving the effect of using a single, better output image for fingerprint recognition when the optical fingerprint sensor acquires multiple fingerprint sub-images at a time, effectively reducing storage consumption during fingerprint recognition, and also effectively improving the recognition speed and success rate of fingerprint recognition; on the other hand, since both the first extended image and the second extended image are obtained by expanding the fused image, and the fused image better aggregates the information from the acquired multiple fingerprint sub-images, has a larger field of view, and lower spatial noise, using one of them as the output image for fingerprint recognition, compared to directly using the fingerprint sub-images for fingerprint recognition, the output image utilizes the information of the large field of view of the fused image, thus its recognition field of view is larger, spatial noise is lower, and signal-to-noise ratio is higher, which is beneficial to improving the recognition success rate.

[0174] In this application, any rule can be used to determine one of the first extended image and the second extended image as the output image. For example, in some optional embodiments, in step S108, the one with higher image quality between the first extended image and the second extended image can be determined as the output image for fingerprint recognition. In this way, the image quality of the output image used for fingerprint recognition can be better guaranteed, thereby effectively improving the fingerprint recognition effect.

[0175] Optionally, the image quality scores corresponding to the first and second extended images can be calculated separately according to the above method (the higher the image quality score, the higher the image quality, which can be determined by calculating relevant indicators (such as Harris response HR, normalized gradient SR, peak signal-to-noise ratio PSNR, normalized signal quantity NSR, etc., one or more of these indicators have been introduced above and will not be repeated here). Thus, the image with higher quality can be determined based on the image quality scores of the two images, so as to determine the output image.

[0176] The following is combined Figure 8 As shown, the overall implementation of an example fingerprint image processing scheme in this application is described. This scheme corresponds to the optional method described above, which involves "firstly locally fusing the overlapping portion of the first target sub-image and the second target sub-image to obtain a target locally fused image, and then stitching the unfused portion of the first target sub-image and the target locally fused image together to form an input image" (i.e., the optional methods of sub-steps S1061 to S1062 described above). See also... Figure 8 First, multiple fingerprint sub-images 1 to N are acquired from the pixel units of the optical fingerprint sensor. Then, the image quality of the fingerprint sub-images 1 to N is evaluated and sorted, and the offset relative to the highest quality sub-image is calculated for each, resulting in fingerprint sub-images ranked 1 to N in image quality. Next, all fingerprint sub-images (i.e., fingerprint images 1 to N) are aligned and offset before being fused to obtain a fused image. Then, the highest quality sub-image is used as the first target sub-image, and the edges of the first target sub-image are extended using the fused image to obtain a first extended image. Finally, the fingerprint sub-image ranked 1st in image quality is used as the first target sub-image (i.e., the highest quality sub-image). The fingerprint sub-image (ranked second in image quality) is first used to obtain a target local fused image. The input image (i.e., the best-to-second-best fused image) is then obtained based on the target local fused image. The edges of the input image are then expanded according to the fused image to obtain a second expanded image at the same position as the first expanded image. After that, the image quality of the first and second expanded images can be evaluated, and the output with the best image quality can be selected as the output image for fingerprint recognition. The output image can then be used for fingerprint recognition (it can also be used for fingerprint registration in other scenarios).

[0177] Optionally, combined Figure 9 As shown, it illustrates the overall implementation of another example fingerprint image processing scheme. Figure 9 and Figure 8 The difference is: Figure 8 It involves using the fingerprint sub-image with the highest image quality (i.e., the best quality sub-image) and the fingerprint sub-image with the second highest image quality (i.e., the second best quality sub-image), achieving local fusion of the best and second best fingerprint sub-images, and then obtaining the input image; while Figure 9 The process involves iterating through the fingerprint sub-images ranked 2nd to Nth in terms of image quality and performing local fusion with the sub-image of best quality to obtain multiple input images (N-1 images). Then, for each of the N-1 input images, N-1 second extended images are obtained. The image quality of the N-1 second extended images is evaluated with the first extended image, and the output with the best image quality is selected as the output image for fingerprint recognition. The output image can then be used for fingerprint recognition or registration. Figure 9 This implementation method is suitable for situations where memory is sufficient and storage pressure is low.

[0178] It should be understood that the above regarding Figure 8 and Figure 9 The descriptions provided are for illustrative purposes only and are not intended to limit the scope of this application.

[0179] Next, let's combine... Figure 10 As shown, the implementation of the fingerprint image processing scheme of this application is described above. Figure 10 The implementation process of the technical solution in this application can be more intuitively seen by combining various image diagrams. It should be understood that the following text regarding... Figure 10 The description provided is not intended to limit the scope of this application.

[0180] Reference Figure 10 As can be seen, the fingerprint image processing scheme of this application includes at least two optional schemes. Optional scheme 1 is the above-mentioned scheme of "first local fusion, then expansion and image selection", and optional scheme 2 is the above-mentioned scheme of "no local fusion, direct expansion and image selection". Figure 10 For ease of representation, "fingerprint sub-image" is referred to as "sub-image".

[0181] Reference Figure 10 As shown, in option 1: Step 1: Acquire multiple fingerprint sub-images 1~N collected by multiple pixel units of the optical fingerprint sensor at the current moment, and fuse the multiple fingerprint sub-images 1~N to obtain a fused image.

[0182] Step 2: From the fused image, determine the first image region corresponding to the position of the first target sub-image among the multiple fingerprint sub-images 1 to N; then determine the second image region including the first image region (the second image region extends beyond at least one edge of the first image region) from the fused image, extract the second image region from the fused image, and use the extraction result as the first extended image (based on this, the edge of the first image region can be extended according to the fused image to obtain the first extended image).

[0183] It should be understood that the example of optional scheme 1 here is with sub Figure 1 The first target sub-image, and with sub Figure 1 The fingerprint sub-image with the best image quality among multiple fingerprint sub-images 1 to N (i.e., the sub-image with the best quality).

[0184] Step 3: Locally fuse the overlapping parts between the first target sub-image and the second target sub-image among multiple fingerprint sub-images to obtain a target local fused image. Then, stitch the unfused parts of the first target sub-image and the target local fused image together to form an input image (based on this, the input image can be determined based on at least one fingerprint sub-image). Next, determine the second image region, including the first image region, from the fused image, and determine the third image region within the second image region other than the first image region. Fill the third image region to the edge of the input image to form a second extended image (based on this, the edge of the input image can be extended based on the fused image to obtain the second extended image).

[0185] It should be understood that the example of optional scheme 1 here is with sub Figure 1 The first target sub-image is 1, and the second target sub-image is 2. The sub-image 2 is the fingerprint sub-image with the second best image quality among the multiple fingerprint sub-images 1 to N.

[0186] Step 4: Based on image quality, the image with higher quality between the first extended image and the second extended image is determined as the output image for fingerprint recognition (based on this, it is possible to determine the output image from the first extended image and the second extended image).

[0187] Reference Figure 10 As shown, in option 2: Step 1: Acquire multiple fingerprint sub-images 1~N collected by multiple pixel units of the optical fingerprint sensor at the current moment, and fuse the multiple fingerprint sub-images 1~N to obtain a fused image.

[0188] Step 2: From the fused image, determine the first image region corresponding to the position of the first target sub-image among the multiple fingerprint sub-images 1 to N; then determine the second image region including the first image region (the second image region extends beyond at least one edge of the first image region) from the fused image, extract the second image region from the fused image, and use the extraction result as the first extended image (based on this, the edge of the first image region can be extended according to the fused image to obtain the first extended image).

[0189] It should be understood that in the example of Option 2 here, sub-image 3 is the first target sub-image, and sub-image 3 is the fingerprint sub-image with the best image quality among multiple fingerprint sub-images 1 to N (i.e., the sub-image with the best quality). Here, sub-image 3 is used as the first target sub-image mainly for the purpose of distinguishing it from Option 1 above and for ease of understanding, and does not mean that there is a substantial difference between the two.

[0190] Step 3: Determine the first target sub-image as the input image, and determine the second image region, including the first image region, from the fused image. Determine the third image region, excluding the first image region, within the second image region. Fill the third image region to the edge of the first target sub-image to form the second extended image. (Based on this, it is possible to determine the input image based on at least one fingerprint sub-image and extend the edge of the input image based on the fused image to obtain the second extended image.)

[0191] Step 4: Based on image quality, the image with higher quality between the first extended image and the second extended image is determined as the output image for fingerprint recognition (based on this, it is possible to determine the output image from the first extended image and the second extended image).

[0192] It is understood that the foregoing description of the fingerprint image processing method is merely an exemplary description of the embodiments of this application and is not intended to limit the embodiments of this application in any way.

[0193] According to a second aspect of this application, and a second aspect of the embodiments thereof, a fingerprint image processing apparatus is provided. (Refer to...) Figure 11 As shown, the fingerprint image processing device 1100 includes: The fusion module 1102 is used to acquire multiple fingerprint sub-images collected by multiple pixel units of the optical fingerprint sensor at the current moment, and to fuse the multiple fingerprint sub-images to obtain a fused image; The first extension module 1104 is used to determine, from the fused image, a first image region corresponding to the position of the first target sub-image among the plurality of fingerprint sub-images, and to extend the edge of the first image region according to the fused image to obtain a first extended image; The second extension module 1106 is used to determine an input image based on at least one fingerprint sub-image, and to extend the edges of the input image based on the fused image to obtain a second extended image, wherein the at least one fingerprint sub-image includes the first target sub-image; The determining module 1108 is used to determine one of the first extended image and the second extended image as the output image for fingerprint recognition.

[0194] The solution provided in this application embodiment, on the one hand, can use one of the first extended image and the second extended image as the output image for fingerprint recognition, thereby enabling the use of a single, better output image for fingerprint recognition when the optical fingerprint sensor acquires multiple fingerprint sub-images at a time. This effectively reduces storage consumption during fingerprint recognition and also effectively improves the recognition speed and success rate. On the other hand, since both the first extended image and the second extended image are obtained by expanding a fused image, and the fused image better aggregates the information from the acquired multiple fingerprint sub-images, has a larger field of view, and lower spatial noise, using one of them as the output image for fingerprint recognition, compared to directly using fingerprint sub-images for fingerprint recognition, utilizes the large field of view information of the fused image. Therefore, its recognition field of view is larger, its spatial noise is lower, and its signal-to-noise ratio is higher, which is beneficial to improving the recognition success rate.

[0195] In some optional embodiments, the first target sub-image is the fingerprint sub-image with the best image quality among the plurality of fingerprint sub-images; the first extension module 1104 is specifically used to: determine a first image region corresponding to the fingerprint sub-image with the best image quality from the fused image.

[0196] In some optional embodiments, the determining module 1108 is specifically used to: determine the one with higher image quality between the first extended image and the second extended image as the output image for fingerprint recognition.

[0197] In some optional embodiments, the first extension module 1104 is specifically configured to: determine a second image region from the fused image, including the first image region, wherein the second image region extends beyond at least one edge of the first image region; extract the second image region from the fused image, and use the extraction result as the first extension image.

[0198] In some optional embodiments, if the four side edges of the first image region do not coincide with the edge of the fused image, then the four side edges of the second image region extend beyond the corresponding four side edges of the first image region; or, if at least one side edge of the first image region coincides with the edge of the fused image, then the second image region extends beyond the side edges of the first image region that do not coincide with the edge of the fused image.

[0199] In some optional embodiments, the first extension module 1104 is specifically used to: locally fuse the overlapping portion between the first target sub-image and the second target sub-image among the plurality of fingerprint sub-images to obtain a target locally fused image, wherein the second target sub-image is any fingerprint sub-image among the plurality of fingerprint sub-images other than the first target sub-image; and stitch the unfused portion of the first target sub-image and the target locally fused image together to form the input image.

[0200] In some optional embodiments, the first extension module 1104 is specifically configured to: determine the target fusion weights corresponding to the first target sub-image and the second target sub-image respectively based on the first local image quality of the first target sub-image in the overlapping portion and the second local image quality of the second target sub-image in the overlapping portion, wherein the higher the local image quality, the higher the determined target fusion weight; and perform a weighted summation of the pixel values ​​of the first target sub-image and the second target sub-image in the overlapping portion based on the target fusion weights corresponding to the first target sub-image and the second target sub-image respectively, so as to locally fuse the overlapping portion between the first target sub-image and the second target sub-image to obtain a target locally fused image.

[0201] In some optional embodiments, the first extension module 1104 is specifically configured to: determine a second image region including the first image region from the fused image, determine a third image region other than the first image region within the second image region, and fill the third image region correspondingly to the edge of the input image to form the second extended image.

[0202] In some optional embodiments, if the first target sub-image is the fingerprint sub-image with the best image quality among the plurality of fingerprint sub-images, then the second target sub-image is the fingerprint sub-image with the second best image quality among the plurality of fingerprint sub-images.

[0203] In some optional embodiments, the second expansion module 1106 is specifically used to: determine the first target sub-image as the input image, and expand the edges of the first target sub-image according to the fused image to obtain the second expanded image.

[0204] In some optional embodiments, the second extension module 1106 is specifically used to: determine a second image region including the first image region from the fused image, determine a third image region other than the first image region within the second image region, and fill the third image region correspondingly outside the edge of the first target sub-image to form the second extended image.

[0205] In some optional embodiments, the fusion module 1102 is specifically configured to: determine the highest quality sub-image among the plurality of fingerprint sub-images; for each fingerprint sub-image other than the highest quality sub-image, determine the similarity of the overlapping portion between the fingerprint sub-image and the highest quality sub-image based on reference offset data, and determine the real-time offset between the fingerprint sub-image and the highest quality sub-image based on the similarity; align the plurality of fingerprint sub-images based on the real-time offset between each fingerprint sub-image and the highest quality sub-image, and fuse the aligned plurality of fingerprint sub-images to obtain the fused image.

[0206] The fingerprint image processing apparatus 1100 provided in the second aspect of this application is based on the same inventive concept as the fingerprint image processing method provided in the first aspect, corresponds to the corresponding fingerprint image processing method in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding fingerprint image processing method embodiments. Therefore, it will not be described again here. In addition, the implementation of each module in the fingerprint image processing apparatus 300 of this embodiment can refer to the description of the corresponding part in the foregoing fingerprint image processing method embodiments, and will not be described again here.

[0207] According to a third aspect of the embodiments of this application, an electronic device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, which causes the processor to perform an operation corresponding to the method of the first aspect.

[0208] Reference Figure 12 This illustration shows a structural schematic diagram of an electronic device according to an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device. Figure 12 As shown, the electronic device 1200 may include: a processor 1202, a communications interface 1204, a memory 1206, and a communications bus 1208.

[0209] in: The processor 1202, communication interface 1204, and memory 1206 communicate with each other via communication bus 1208.

[0210] Communication interface 1204 is used for communication with other electronic devices or servers.

[0211] The processor 1202 is used to execute the computer program 1210, specifically the relevant steps in the above-described fingerprint image processing method embodiment.

[0212] Specifically, computer program 1210 may include program code that includes computer operation instructions.

[0213] The processor 1202 may be a CPU, a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0214] Memory 1206 is used to store computer program 1210. Memory 1206 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0215] The computer program 1210 may include multiple computer instructions. Specifically, the computer program 1210 may use multiple computer instructions to cause the processor 1202 to perform the operation corresponding to the fingerprint image processing method described in any of the foregoing multiple method embodiments.

[0216] The specific implementation of each step in computer program 1210 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0217] According to a fourth aspect of the embodiments of this application, the embodiments of this application also provide a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the fingerprint image processing method described in any of the multiple method embodiments provided in the first aspect. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk, etc.

[0218] According to a fifth aspect of the embodiments of this application, the embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the fingerprint image processing method as described in any of the multiple method embodiments provided in the first aspect.

[0219] According to a sixth aspect of the embodiments of this application, the embodiments of this application also provide a fingerprint recognition device, applied to an electronic device having a display screen, referring to... Figure 13 As shown, the fingerprint recognition device 1300 includes: an optical fingerprint sensor 1302, used to: image multiple light signals reflected from a finger above the display screen in multiple different directions to acquire multiple fingerprint sub-images; and a processing unit 1304, used to execute the fingerprint image processing method described in any one of the first aspects.

[0220] It should be understood that optical fingerprint sensors employ a multi-angle optical path design and may include multiple pixel units. The relevant details of optical fingerprint sensors have already been explained in the method embodiments of the first aspect above, and will not be repeated here.

[0221] In addition, as mentioned above, the processing unit can be any processing unit. For example, the processing unit can be the processing unit of the electronic device to which the optical fingerprint sensor is applied (e.g., including but not limited to mobile terminals, computers, fingerprint locks, etc.). (For example, it may include one or more chips capable of data processing, including but not limited to CPU (Central Processing Unit), MCU (Microcontroller Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), etc.), which is beneficial to improving the processing effect.

[0222] The fingerprint image processing device 1100 / electronic device 1200 / computer storage medium / computer program product / fingerprint recognition device 1300 embodiments in this application have been described in detail in the foregoing fingerprint image processing method embodiments. Therefore, their related content and beneficial effects can be understood by referring to the above method embodiments, and will not be repeated here.

[0223] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0224] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application. It should be understood that the various technical features in the technical solutions of the embodiments of this application can be combined in any suitable manner.

[0225] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded over a network. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0226] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for specific applications, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0227] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

[0228] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". It should be noted that the concepts of "first", "second", etc., mentioned in the embodiments of this application are only used to distinguish different devices, modules, or units, and are not used to limit the order of functions performed by these devices, modules, or units or their interdependencies. It should be noted that the modifications of "a" and "a plurality" mentioned in the embodiments of this application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0229] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of this application, and are not intended to limit them; although the embodiments of this application have been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A fingerprint image processing method, comprising: Multiple fingerprint sub-images collected by multiple pixel units of the optical fingerprint sensor at the current moment are acquired, and the multiple fingerprint sub-images are fused to obtain a fused image; From the fused image, a first image region corresponding to the position of the first target sub-image among the plurality of fingerprint sub-images is determined, and the edges of the first image region are extended according to the fused image to obtain a first extended image; An input image is determined based on at least one fingerprint sub-image, and the edges of the input image are extended based on the fused image to obtain a second extended image, wherein the at least one fingerprint sub-image includes the first target sub-image; One of the first extended image and the second extended image is determined as the output image for fingerprint recognition.

2. The method according to claim 1, wherein, The first target sub-image is the fingerprint sub-image with the best image quality among the plurality of fingerprint sub-images; Determining the first image region from the fused image that corresponds to the location of the first target sub-image among the plurality of fingerprint sub-images includes: From the fused image, a first image region is determined that corresponds to the fingerprint sub-image with the best image quality.

3. The method according to claim 1, wherein, The step of determining one of the first extended image and the second extended image as the output image for fingerprint recognition includes: The image with higher quality between the first extended image and the second extended image is selected as the output image for fingerprint recognition.

4. The method according to any one of claims 1-3, wherein, The step of extending the edges of the first image region based on the fused image to obtain a first extended image includes: A second image region is determined from the fused image, including the first image region, wherein the second image region extends beyond at least one edge of the first image region; The second image region is extracted from the fused image, and the extracted result is used as the first extended image.

5. The method according to claim 4, wherein, If the four edges of the first image region do not coincide with the edges of the fused image, then the four edges of the second image region extend beyond the corresponding four edges of the first image region; or... If at least one edge of the first image region coincides with the edge of the fused image, then the second image region extends beyond the edges of the first image region that do not coincide with the edge of the fused image.

6. The method according to any one of claims 1-3, wherein, The step of determining the input image based on at least one fingerprint sub-image includes: The overlapping portion between the first target sub-image and the second target sub-image among the plurality of fingerprint sub-images is locally fused to obtain a target local fused image, wherein the second target sub-image is any fingerprint sub-image among the plurality of fingerprint sub-images other than the first target sub-image; The input image is formed by stitching together the unfused portions of the first target sub-image and the locally fused target image.

7. The method according to claim 6, wherein, The step of locally fusing the overlapping portion between the first target sub-image and the second target sub-image among the plurality of fingerprint sub-images to obtain a target locally fused image includes: Based on the first local image quality of the first target sub-image in the overlapping portion and the second local image quality of the second target sub-image in the overlapping portion, the target fusion weights corresponding to the first target sub-image and the second target sub-image are determined respectively. Furthermore, if the local image quality is higher, the determined target fusion weight is higher. Based on the target fusion weights corresponding to the first target sub-image and the second target sub-image respectively, the pixel values ​​of the first target sub-image and the second target sub-image in the overlapping part are weighted and summed to locally fuse the overlapping part between the first target sub-image and the second target sub-image to obtain a target local fusion image.

8. The method according to claim 6, wherein, The step of expanding the edges of the input image based on the fused image to obtain a second expanded image includes: From the fused image, a second image region including the first image region is determined, and a third image region other than the first image region is determined within the second image region; The third image region is correspondingly filled to the outside of the edge of the input image to form the second extended image.

9. The method according to claim 6, wherein, If the first target sub-image is the fingerprint sub-image with the best image quality among the plurality of fingerprint sub-images, then the second target sub-image is the fingerprint sub-image with the second best image quality among the plurality of fingerprint sub-images.

10. The method according to any one of claims 1-3, wherein, The step of determining an input image based on at least one fingerprint sub-image and expanding the edges of the input image based on the fused image to obtain a second expanded image includes: The first target sub-image is determined as the input image, and the edges of the first target sub-image are expanded according to the fused image to obtain the second expanded image.

11. The method according to claim 10, wherein, The step of expanding the edges of the first target sub-image based on the fused image to obtain the second expanded image includes: From the fused image, a second image region including the first image region is determined, and a third image region other than the first image region is determined within the second image region; The third image region is correspondingly filled to the outside of the edge of the first target sub-image to form the second extended image.

12. The method according to any one of claims 1-3, wherein, The step of fusing the multiple fingerprint sub-images to obtain a fused image includes: Determine the highest quality sub-image among the plurality of fingerprint sub-images; For each fingerprint sub-image other than the best quality sub-image among the plurality of fingerprint sub-images, the similarity of the overlapping part between the fingerprint sub-image and the best quality sub-image is determined according to the reference offset data, and the real-time offset between the fingerprint sub-image and the best quality sub-image at the current time is determined according to the similarity. Based on the real-time offset between each fingerprint sub-image and the highest quality sub-image at the current moment, the multiple fingerprint sub-images are aligned, and the aligned multiple fingerprint sub-images are fused to obtain the fused image.

13. A fingerprint image processing apparatus, comprising: The fusion module is used to acquire multiple fingerprint sub-images collected by multiple pixel units of the optical fingerprint sensor at the current moment, and to fuse the multiple fingerprint sub-images to obtain a fused image; The first extension module is used to determine, from the fused image, a first image region corresponding to the position of a first target sub-image among the plurality of fingerprint sub-images, and to extend the edge of the first image region according to the fused image to obtain a first extended image; The second extension module is used to determine an input image based on at least one fingerprint sub-image, and to extend the edges of the input image based on the fused image to obtain a second extended image, wherein the at least one fingerprint sub-image includes the first target sub-image; The determining module is used to determine one of the first extended image and the second extended image as the output image for fingerprint recognition.

14. An electronic device, comprising: The processor, the communication interface, the memory, and the communication bus are provided, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus. The memory is used to store computer programs; The processor is configured to perform the method of any one of claims 1-12 by running the computer program stored in the memory.

15. A computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1-12.

16. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-12.

17. A fingerprint recognition device, applied to an electronic device having a display screen, the fingerprint recognition device comprising: An optical fingerprint sensor is used to: image light signals reflected from a finger above the display screen in multiple different directions to acquire multiple fingerprint sub-images; A processing unit for performing the method as described in any one of claims 1-12.

Images