Fingerprint image processing method and apparatus, electronic device, and storage medium

Abstract

Provided are a fingerprint image processing method and apparatus, an electronic device, and a storage medium. The fingerprint image processing method comprises: acquiring a plurality of fingerprint sub-images respectively collected by a plurality of pixel units of an optical fingerprint sensor at a current moment, and performing fusion processing on the plurality of fingerprint sub-images, to obtain a fusion image; from within the fusion image, determining a first image area corresponding to the position of a first target sub-image among the plurality of fingerprint sub-images, and expanding the edge of the first image area according to the fusion image, to obtain a first expanded image; determining an input image according to at least one fingerprint sub-image, and expanding the edge of the input image according to the fusion image, to obtain a second expanded image, wherein the at least one fingerprint sub-image comprises the first target sub-image; and determining one of the first expanded image and the second expanded image to be an output image used for performing fingerprint recognition.

Description

Fingerprint image processing method, device, electronic device and storage medium Technical Field

[0001] The embodiments of the present application relate to the technical field of optical fingerprint image processing, and in particular to a fingerprint image processing method, device, electronic device, and storage medium. Background Art

[0002] The optical path designs of some current ultra-thin under-screen optical fingerprint sensors typically utilize multi-angle optical paths. By designing multiple pixel units arranged in an array, these units capture fingerprint images from multiple angles through multi-directional optical channels, thereby maximizing the capture of valid fingerprint signals. However, using multiple fingerprint images to compare against a registered fingerprint template for fingerprint recognition reduces recognition speed and increases memory usage during fingerprint recognition. Furthermore, using only one or a few of the multiple fingerprint images fails to fully utilize the information from all captured fingerprint images, which can easily reduce the success rate of fingerprint recognition. Therefore, a new technical solution is needed to at least partially address these issues.

[0003] Summary of the Invention

[0004] In view of this, embodiments of the present application provide a fingerprint image processing method, apparatus, electronic device, and storage medium to at least partially solve the above-mentioned problems.

[0005] According to a first aspect of an embodiment of the present application, a fingerprint image processing method is provided, comprising:

[0006] Acquire multiple fingerprint sub-images respectively collected by multiple pixel units of the optical fingerprint sensor at the current moment, and fuse the multiple fingerprint sub-images to obtain a fused image;

[0007] Determining, from the fused image, a first image region corresponding to a position of a first target sub-image in the multiple fingerprint sub-images, and extending an edge of the first image region according to the fused image to obtain a first extended image;

[0008] determining an input image according to at least one fingerprint sub-image, and extending an edge of the input image according to the fused image to obtain a second extended image, wherein the at least one fingerprint sub-image includes the first target sub-image;

[0009] One of the first extended image and the second extended image is determined as an output image for fingerprint recognition.

[0010] In some optional embodiments, the first target sub-image is the fingerprint sub-image with the best image quality among the multiple fingerprint sub-images; and determining, from the fused image, a first image area corresponding to the position of the first target sub-image among the multiple fingerprint sub-images includes: determining, from the fused image, a first image area corresponding to the fingerprint sub-image with the best image quality.

[0011] In some optional embodiments, determining one of the first extended image and the second extended image as the output image for fingerprint recognition includes: determining the one with higher image quality between the first extended image and the second extended image as the output image for fingerprint recognition.

[0012] In some optional embodiments, the extending the edge of the first image area according to the fused image to obtain the first extended image includes: determining a second image area including the first image area from the fused image, wherein the second image area exceeds the edge of at least one side of the first image area; and cutting out the second image area from the fused image, and using the cutout result as the first extended image.

[0013] In some optional embodiments, if the four side edges of the first image area do not overlap with the edges of the fused image, the four side edges of the second image area respectively exceed the corresponding four side edges of the first image area; or, if the edge of at least one side of the first image area overlaps with the edge of the fused image, the second image area exceeds the side edges of the first image area that do not overlap with the edges of the fused image.

[0014] In some optional embodiments, determining the input image based on at least one fingerprint sub-image includes: locally fusing the overlapping portion between the first target sub-image and the second target sub-image among the multiple fingerprint sub-images to obtain a target locally fused image, wherein the second target sub-image is any fingerprint sub-image among the multiple fingerprint sub-images except the first target sub-image; and splicing the input image based on the portion of the first target sub-image that is not locally fused and the target locally fused image.

[0015] In some optional embodiments, locally fusing the overlapping portion between the first target sub-image and the second target sub-image among the multiple fingerprint sub-images to obtain a target locally fused image includes: determining target fusion weights corresponding to the first target sub-image and the second target sub-image respectively according to a first local image quality of the first target sub-image in the overlapping portion and a second local image quality of the second target sub-image in the overlapping portion, and the higher the local image quality, the higher the determined target fusion weight; and performing weighted summation of pixel values ​​of the first target sub-image and pixel values ​​of the second target sub-image in the overlapping portion according to the target fusion weights corresponding to the first target sub-image and the second target sub-image respectively, so as to locally fuse the overlapping portion between the first target sub-image and the second target sub-image to obtain a target locally fused image.

[0016] In some optional embodiments, the extending the edge of the input image according to the fused image to obtain the second extended image includes: determining a second image area including the first image area from the fused image, and determining a third image area excluding the first image area within the second image area; and filling the third image area correspondingly to the outside of the edge of the input image to form the second extended image.

[0017] In some optional embodiments, if the first target sub-image is the fingerprint sub-image with the best image quality among the multiple fingerprint sub-images, then the second target sub-image is the fingerprint sub-image with the second best image quality among the multiple fingerprint sub-images.

[0018] In some optional embodiments, determining the input image based on at least one fingerprint sub-image and extending the edge of the input image based on the fused image to obtain the second extended image includes: determining the first target sub-image as the input image, and extending the edge of the first target sub-image based on the fused image to obtain the second extended image.

[0019] In some optional embodiments, the edge of the first target sub-image is extended according to the fused image to obtain the second extended image, including: determining a second image area including the first image area from the fused image, and determining a third image area within the second image area excluding the first image area; and filling the third image area correspondingly to the outside of the edge of the first target sub-image to form the second extended image.

[0020] In some optional embodiments, the fusing of the multiple fingerprint sub-images to obtain a fused image includes: determining a best-quality sub-image having the best image quality among the multiple fingerprint sub-images; determining, for each fingerprint sub-image other than the best-quality sub-image among the multiple fingerprint sub-images, a similarity between the fingerprint sub-image and an overlapping portion of the best-quality sub-image based on reference offset data, and determining, based on the similarity, a real-time offset between the fingerprint sub-image and the best-quality sub-image at a current moment; aligning the multiple fingerprint sub-images based on the real-time offset between each fingerprint sub-image and the best-quality sub-image at a current moment, and fusing the aligned multiple fingerprint sub-images to obtain the fused image.

[0021] According to a second aspect of an embodiment of the present application, there is provided a fingerprint image processing device, comprising:

[0022] a fusion module, configured to obtain a plurality of fingerprint sub-images respectively collected by a plurality of pixel units of the optical fingerprint sensor at a current moment, and fuse the plurality of fingerprint sub-images to obtain a fused image;

[0023] a first expansion module, configured to determine, from the fused image, a first image region corresponding to a position of a first target sub-image in the plurality of fingerprint sub-images, and to expand an edge of the first image region according to the fused image to obtain a first expanded image;

[0024] a second expansion module, configured to determine an input image based on at least one fingerprint sub-image, and expand an edge of the input image based on the fused image to obtain a second expanded image, wherein the at least one fingerprint sub-image includes the first target sub-image;

[0025] The determination module is configured to determine one of the first extended image and the second extended image as an output image for fingerprint recognition.

[0026] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store a computer program; and the processor is used to execute the method provided in the first aspect by running the computer program stored on the memory.

[0027] According to a fourth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method provided in the first aspect is implemented.

[0028] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising a computer program, which implements the method provided in the first aspect when executed by a processor.

[0029] According to a sixth aspect of an embodiment of the present application, a fingerprint recognition device is provided, which is applied to an electronic device with a display screen. The fingerprint recognition device includes: an optical fingerprint sensor, which is used to: image light signals in multiple directions reflected by a finger above the display screen to obtain multiple fingerprint sub-images; and a processing unit, which is used to execute the method provided in the first aspect.

[0030] According to the fingerprint image processing solution provided in the embodiment of the present application, multiple fingerprint sub-images respectively collected by multiple pixel units of the optical fingerprint sensor can be fused to obtain a fused image, and a first image area corresponding to the position of a first target sub-image in the multiple fingerprint sub-images can be determined from the fused image, and the edge of the first image area can be extended according to the fused image to obtain a first extended image. Then, an input image is determined according to at least one fingerprint sub-image, and the edge of the input image is extended according to the fused image to obtain a second extended image, and the at least one fingerprint sub-image includes the first target sub-image. Then, one of the first extended image and the second extended image is determined as the output image for fingerprint recognition. Therefore, on the one hand, the solution of the embodiment of the present application can output one of the first extended image and the second extended image as the output image for fingerprint recognition, so that when the optical fingerprint sensor collects multiple fingerprint sub-images at a time, a single better output image can be used for fingerprint recognition, thereby effectively reducing the storage consumption during fingerprint recognition, and can also effectively improve the recognition speed and recognition success rate of fingerprint recognition; on the other hand, since the first extended image and the second extended image are both obtained by expanding the fused image, and the fused image better aggregates the information in the multiple collected fingerprint sub-images, has a larger field of view, and has lower spatial noise, one of the two is used as the output image for fingerprint recognition. Compared with directly using the fingerprint sub-image for fingerprint recognition, the output image utilizes the information of the large field of view of the fused image, so its recognition field of view is larger, the spatial noise is lower, and the signal-to-noise ratio is higher, which is conducive to improving the recognition success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0032] FIG1 shows a flow chart of an optional fingerprint image processing method of the present application.

[0033] FIG2A shows a flowchart of an optional sub-step of “fusing multiple fingerprint sub-images to obtain a fused image” in step S102 .

[0034] FIG2B shows a schematic diagram of multi-angle light paths of a pixel unit of an optional optical fingerprint sensor of the present application.

[0035] FIG2C is a schematic diagram showing the real-time offset calculation between the fingerprint sub-image and the sub-image with the best quality.

[0036] FIG3A shows a flowchart of an optional sub-step of “determining the similarity between the overlapping portion of the fingerprint sub-image and the optimal quality sub-image based on the reference offset data, and determining the real-time offset between the fingerprint sub-image and the optimal quality sub-image at the current moment based on the similarity” in sub-step S1022.

[0037] FIG3B shows an optional sub-step flow chart of a method for updating reference offset data.

[0038] FIG3C shows a flowchart of an optional sub-step of “fusing the aligned multiple fingerprint sub-images to obtain a fused image” in sub-step S1023 .

[0039] FIG3D shows a flowchart of an optional sub-step of “merging the parts of each fingerprint sub-image in the overlapping area according to the local image quality of each fingerprint sub-image in the overlapping area and the similarity between each fingerprint sub-image in the overlapping area and the overlapping part of the sub-image with the best quality” in sub-step S10231.

[0040] FIG3E shows a schematic diagram of fusing parts of two images.

[0041] FIG4A shows a flowchart of an optional sub-step of “extending the edge of the first image region according to the fused image to obtain a first extended image” in step S104 .

[0042] FIG4B shows a flowchart of an optional sub-step of “determining the input image according to at least one fingerprint sub-image” in step S106 .

[0043] FIG4C shows an optional sub-step flowchart of sub-step S1061.

[0044] FIG5 is a schematic diagram showing the extension direction of the edge of the first image region in the fused image of the present application.

[0045] FIG6 is a schematic diagram showing the local fusion of the overlapping portion of the first target sub-image and the second target sub-image.

[0046] FIG7 is a schematic diagram showing a second extended image obtained by extending the edge of an input image by fusing images.

[0047] FIG8 shows a block diagram of an overall implementation of an optional fingerprint image processing solution of the present application.

[0048] FIG9 shows a block diagram of an overall implementation of another optional fingerprint image processing solution of the present application.

[0049] FIG10 is a schematic diagram showing the implementation process of the fingerprint image processing solution of the present application.

[0050] FIG11 shows a schematic diagram of an exemplary fingerprint image processing device of the present application.

[0051] FIG12 shows a schematic diagram of an exemplary electronic device of the present application.

[0052] FIG13 shows a schematic diagram of an exemplary fingerprint recognition device of the present application. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and in detail described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0054] The optical path design of ultra-thin under-screen optical fingerprint sensors requires consideration of two extreme signal acquisition conditions. The first is when the fingerprint is well adhered to the screen, such as when fingerprint recognition is performed at room temperature. In this case, the smaller the signal acquisition optical path's light collection angle, the more likely it is to receive a single fingerprint ridge signal (or fingerprint valley signal), resulting in a higher signal strength and improved recognition. The other is when the fingerprint is poorly adhered to the screen, such as when fingerprint recognition is performed at low temperatures or when a light press is performed. In these cases, it is desirable to increase the signal acquisition optical path's light collection angle. Because a fingerprint is a three-dimensional surface, at high angles of incident light, valley-reflected light is more easily blocked by the ridge sidewalls, increasing the difference between the ridge and valley signals and boosting the signal. Actual fingerprint press signals fall somewhere between these two extremes. To address these considerations, the optical path designs of some current ultra-thin under-screen optical fingerprint sensors typically employ a multi-angle optical path. By designing an array of multiple pixel units, these units capture fingerprint images from different angles through multi-directional light channels, thereby maximizing the number of valid fingerprint signals captured.

[0055] However, using multiple fingerprint images to compare with the registered fingerprint template for fingerprint recognition will lead to a decrease in recognition speed and increase the storage consumption during fingerprint recognition; and if only one or a few of the multiple fingerprint images are used, the information in all the collected fingerprint images cannot be fully utilized, which can easily reduce the success rate of fingerprint recognition.

[0056] In view of this, a fingerprint image processing solution is proposed in this application. The specific implementation of the embodiment of this application is further described below with reference to the accompanying drawings.

[0057] FIG1 shows a flowchart of an optional fingerprint image processing method of the present application. As shown in FIG1 , the fingerprint image processing method includes steps S102, S104, S106 and S108, specifically:

[0058] S102: Acquire multiple fingerprint sub-images respectively collected by multiple pixel units of the optical fingerprint sensor at the current moment, and fuse the multiple fingerprint sub-images to obtain a fused image.

[0059] This application does not specifically limit the specific results of the optical fingerprint sensor. For example, the optical fingerprint sensor can be an under-screen optical fingerprint sensor, which can be used to implement the under-screen optical fingerprint recognition function of an electronic device. The optical fingerprint sensor can image light signals in multiple directions reflected by a finger above the display screen of the electronic device to collect multiple fingerprint sub-images. It should be noted that the fingerprint image detection method of the present application can be executed by any processing unit. For example, the method can be executed by a processing unit (for example, including but not limited to one or more chips capable of data processing, including but not limited to a CPU (Central Processing Unit), an MCU (Microcontroller Unit), a GPU (Graphic Processing Unit), an FPGA (Field Programmable Gate Array), etc.) of an electronic device to which the optical fingerprint sensor is applied (for example, including but not limited to a mobile terminal, a computer, a fingerprint lock, etc.), which is conducive to improving the processing effect.

[0060] The optical fingerprint sensor in this application adopts a multi-angle optical path design, which includes multiple pixel units, each pixel unit may include one or more optical pixels, and multiple pixel units may include the same number of optical pixels. Figure 2B shows a schematic diagram of the multi-angle optical path of a pixel unit of an optional optical fingerprint sensor. As shown in Figure 2B, the schematic diagram shows that the optical fingerprint sensor includes multiple pixel units (as shown in the simplified example of Figure 2B, a total of 9 pixel units, but it should be understood that there may be more pixel units in practice), and microlens units are provided on the multiple pixel units. When a finger records a fingerprint through the optical fingerprint sensor, the reflected light can pass through the different optical paths of the microlens unit and be received by each pixel unit at various light receiving angles. Each pixel unit can generate a fingerprint image based on the received light, thereby realizing the collection of the fingerprint image by each pixel unit. The light at each angle received by each pixel unit is also a plurality of light signals in different directions. It should be understood that the multiple fingerprint sub-images described in step S102 are also fingerprint images collected by multiple pixel units respectively.

[0061] To facilitate the description of this embodiment, 9 fingerprint sub-images collected by 9 pixel units are taken as an example.

[0062] Due to the multi-angle optical path, there is a certain amount of image offset between each of the multiple fingerprint sub-images. Because of the offset between the individual fingerprint sub-images, the area of ​​the fused image is larger than that of a single fingerprint sub-image. In other words, the field of view of the fused image is larger than that of a single fingerprint sub-image. Therefore, the fused image can aggregate the effective information of multiple fingerprint sub-images. The output image for fingerprint recognition is obtained through fusion image processing, allowing the output image to utilize the information of the large field of view of the fused image to increase the recognition field of view of fingerprint recognition.

[0063] The specific implementation of step S102 is not limited in this application. Optionally, in this application, step S102 may be to align and offset multiple fingerprint sub-images and then perform fusion processing to obtain a fused image.

[0064] Optionally, a preset fixed image offset can be used to align and offset multiple fingerprint sub-images. This is because, in the aforementioned multi-optical-path optical fingerprint sensor, the directions of the multi-directional optical channels are fixed, resulting in a nearly stable image offset between the multiple fingerprint images captured along each directional optical channel. Therefore, a suitable fixed image offset can be obtained through a special method and preset to facilitate the fusion processing of multiple fingerprint sub-images captured by multiple pixel units after alignment and offset. Other methods can also be used to achieve this, as long as they meet the requirements.

[0065] The present application also notes that since the finger is a three-dimensional surface, the actual offset between the fingerprint images and the fixed image offset can easily deviate due to factors such as the contact state caused by the fingerprint state (such as dry or wet fingers), the slight deformation caused by the pressing state (such as light pressing or heavy pressing), and the effective pressing area. This can then lead to a deterioration in the quality of the fused image obtained after the fusion of multiple fingerprint sub-images. To improve this situation, in some optional embodiments of the present application, with reference to the flowchart shown in FIG2A , step S102 of "fusing multiple fingerprint sub-images to obtain a fused image" includes sub-steps S1021 to S1023, specifically:

[0066] S1021: Determine the best-quality sub-image with the best image quality among the multiple fingerprint sub-images.

[0067] The optimal quality sub-image in this application is the fingerprint sub-image with the best image quality among multiple fingerprint sub-images. The optimal quality sub-image can be determined according to any suitable method, which is not limited here. For example, any suitable algorithm can be used to calculate one or more of the indicators such as Harris Response (HR), Normalized Gradient (SR), Peak Signal-to-Noise Ratio (PSNR), Normalized Stoichiometric Ratio (NSR, which can characterize the ratio between the peak-to-peak value and the local mean) of each fingerprint sub-image, and the image quality of the multiple fingerprint sub-images is determined based on the calculation results to determine the optimal quality sub-image with the best image quality. Optionally, by calculating one or more of the above indicators, the image quality of each fingerprint sub-image can be scored, and the fingerprint sub-image with the highest image quality score can be determined as the optimal quality sub-image.

[0068] The above metrics can be used to determine image quality because they essentially measure the clarity of banded stripes. Within a small region of a fingerprint subimage, a fingerprint is essentially a periodic light and dark band signal. The clearer the banded stripes, the larger the normalized gradient (SR). Similarly, the fluctuation of the neighborhood's maximum and minimum differences relative to the maximum or minimum values ​​(corresponding to the peak signal-to-noise ratio (PSNR)) increases. Similarly, the Harris response (HR) eigenvalue exhibits edge characteristics, and the ratio of the peak signal to the local mean (corresponding to the normalized signal strength (NSR)) increases. (The local mean is linearly related to light intensity and exposure time. Under the same conditions, a good fingerprint will have a larger signal strength, i.e., a higher normalized signal strength (NSR)). Therefore, determining the image quality of a fingerprint subimage using one or more of the above metrics can ensure the reliability of the optimal subimage, facilitating subsequent data processing.

[0069] In addition, in step S1021 of the present application, multiple fingerprint sub-images at the current moment are obtained, and the sub-image with the best quality is determined, which is conducive to real-time and accurate fingerprint recognition.

[0070] S1022: For each fingerprint sub-image other than the sub-image with the best quality among the multiple fingerprint sub-images, determine the similarity of the overlapping portion between the fingerprint sub-image and the sub-image with the best quality based on the reference offset data, and determine the current real-time offset between the fingerprint sub-image and the sub-image with the best quality based on the similarity. (Please continue to understand this sub-step S1022 in conjunction with the flowchart of Figure 2A.)

[0071] There will be a certain offset between multiple fingerprint sub-images (also known as image offset). The offset between images is measured in pixels. The offset can include a lateral offset along the horizontal direction of the image (for example, called the x direction, which can also be understood as the horizontal direction) and a longitudinal offset along the vertical direction of the image (for example, called the y direction, which can also be understood as the vertical direction). For ease of explanation, the horizontal rightward direction of the image is regarded as the positive direction of the x direction, the horizontal leftward direction is regarded as the negative direction of the x direction, the vertical downward direction is regarded as the positive direction of the y direction, and the vertical upward direction is regarded as the negative direction of the y direction. In the following, each type of offset can be recorded in the form of (x, y), where x represents the lateral offset in the offset and y represents the vertical offset in the offset. A positive x value represents a lateral offset to the right along the horizontal direction of the image, a negative x value represents a lateral offset to the left along the horizontal direction of the image, a positive y value represents a longitudinal offset downward along the vertical direction of the image, and a negative y value represents a longitudinal offset downward along the vertical direction of the image.

[0072] For example, FIG2C shows a schematic diagram of real-time offset calculation between a fingerprint sub-image and an optimal quality sub-image. FIG2C also illustrates the x-direction (horizontal) and y-direction (vertical) of an image. It should be understood that the fingerprint sub-image and optimal quality sub-image, as well as the overlap between them, shown in FIG2C are merely examples and do not constitute any limitation on the present application.

[0073] Optionally, the reference offset data in the present application includes reference offsets between a plurality of reference fingerprint sub-images and a reference optimal quality sub-image, wherein the reference optimal quality sub-image is acquired by a first pixel unit among the plurality of pixel units, and the plurality of reference fingerprint sub-images are acquired by a plurality of second pixel units among the plurality of pixel units, wherein the plurality of second pixel units are a plurality of pixel units other than the first pixel unit among the plurality of pixel units. In the present application, such reference offset data can be used to facilitate data processing to determine the similarity between the overlapping portions of the fingerprint sub-image and the optimal quality sub-image.

[0074] Before executing the fingerprint image processing method of the present application for the first time, reference offset data can be pre-stored (which can be understood as initialized reference deviation data) and can be retrieved when needed. Optionally, the reference offset data can be initialized by calibration during the mass production phase using striped weights or weights with special patterns (striped weights or weights with special patterns can be used to simulate fingers).

[0075] For example, the reference deviation data determination process in this application can be understood by referring to the following process: Taking the 9 pixel units of the optical fingerprint sensor in FIG2B as an example, in the mass production stage, 9 images of the striped weight (e.g., images 1 to 9) can be collected through 9 pixel units (e.g., pixel units 1 to 9), and then the image with the best image quality among images 1 to 9 is determined. The image with the best image quality is used as the reference sub-image with the best quality (e.g., image 1). The other 8 images can be used as reference fingerprint sub-images (i.e., images 2 to 9), with pixel unit 1 being the first pixel unit and pixel units 2 to 9 being the second pixel units. Afterwards, the image offsets of images 2 to 9 relative to image 1 can be directly measured to obtain 8 image offsets. These 8 image offsets can then be used as 8 reference offsets, which can be stored as reference offset data and directly retrieved when needed. It should be understood that this process is merely an example for ease of understanding and does not constitute any limitation to this application.

[0076] The present application does not limit the method for calculating the similarity. In some optional embodiments, step S1022 of "determining the similarity of the overlapping portion of the fingerprint sub-image and the sub-image with the best quality based on the reference offset data, and determining the real-time offset between the fingerprint sub-image and the sub-image with the best quality at the current moment based on the similarity" includes sub-steps S10221 and S10222, specifically:

[0077] S10221: Calculate the current reference offset between the fingerprint sub-image and the sub-image with the best quality based on the reference offsets between the multiple reference fingerprint sub-images and the reference sub-image with the best quality, and determine the real-time offset search range at the current moment based on the current reference offset.

[0078] Since the multiple reference fingerprint sub-images and the reference optimal quality sub-image are also collected by the same multiple pixel units as the multiple fingerprint sub-images, when executing the fingerprint image processing method of this scheme (for example, when fingerprint recognition is performed in practice), if the optimal quality sub-image is not collected by the first pixel unit, the current reference offset between each fingerprint sub-image and the optimal quality sub-image can be converted by using the reference offsets between the multiple reference fingerprint sub-images and the reference optimal quality sub-image.

[0079] For example, let's take the example of 9 pixel units capturing 9 images respectively, pixel unit 1 being the first pixel unit capturing the reference sub-image with the best quality, and pixel units 2 to 9 being the second pixel units capturing the reference fingerprint sub-image. For example, let's take the example of the reference offset between the reference fingerprint sub-image captured by pixel unit 2 and the reference sub-image with the best quality captured by pixel unit 1 being (x1, y1). If the sub-image with the best quality determined in step S1021 is the fingerprint sub-image captured by pixel unit 2, then the current reference offset between the fingerprint sub-image captured by pixel unit 1 and the sub-image with the best quality captured by pixel unit 2 can be converted to (-x1, -y1); similarly, the current reference offset between the fingerprint sub-image captured by other pixel units and the sub-image with the best quality captured by pixel unit 2 can also be converted according to each reference offset. It should be understood that this is merely an example for ease of understanding and is not intended to limit the present application.

[0080] After calculating the current reference offset, the real-time offset search range at the current moment can be determined to accurately determine the real-time offset corresponding to the fingerprint sub-image within the real-time offset search range.

[0081] In some optional embodiments, "determining the real-time offset search range at the current moment based on the current reference offset" in sub-step S10221 may include: determining the real-time offset search range at the current moment based on the current reference offset and a preset offset margin.

[0082] Based on this, the above embodiments of the present application determine the real-time offset search range by using the current reference offset and the preset offset margin to ensure that a suitable real-time offset can be searched within the real-time offset search range.

[0083] For example, the preset offset margin includes a positive offset margin and a negative offset margin along the horizontal direction of the image, and a positive offset margin and a negative offset margin along the vertical direction of the image. Optionally, referring to the example shown in FIG2C , the absolute values ​​of the positive offset margin and the negative offset margin along the horizontal direction of the image are equal, both being wx; the absolute values ​​of the positive offset margin and the negative offset margin along the vertical direction of the image are equal, both being wy. For example, the current reference offset can be recorded as (sx, sy), where sx represents the horizontal offset in the current reference offset, and sy represents the vertical offset in the current reference offset. Combined with FIG2C , the determined real-time offset search range can be expressed as: sx+i, sy+j; wherein i=[-wx, wx], j=[-wy, wy]. Of course, the absolute values ​​of the positive offset margin and the negative offset margin along the horizontal / vertical direction are equal here, which is a special example. In other optional embodiments, they may also be unequal. For ease of explanation, equality can be used as an example below.

[0084] The range corresponding to the preset offset margin within the real-time offset search range can be understood as the neighborhood of the range corresponding to the current reference offset within the real-time offset search range. Therefore, this optional implementation of the present application can be understood as a "current reference offset + neighborhood search" approach to ensure that a suitable real-time offset is found within the real-time offset search range.

[0085] S10222: Determine the similarity between the overlapping portions of the fingerprint sub-image and the best-quality sub-image, corresponding to each offset within the current real-time offset search range. The offset corresponding to the greatest similarity is used as the current real-time offset between the fingerprint sub-image and the best-quality sub-image. (Please continue to understand this sub-step S10222 in conjunction with the flowchart in FIG3A .)

[0086] Within the real-time offset search range, there are multiple offsets (each offset includes a horizontal offset and a vertical offset). For each offset, the corresponding overlapping portion of the fingerprint sub-image and the optimal quality sub-image may be different. For each offset, the similarity between the corresponding overlapping portion of the fingerprint sub-image and the optimal quality sub-image is calculated, and the offset corresponding to the maximum similarity is used as the real-time offset at the current moment.

[0087] Alternatively, the real-time offset between the fingerprint sub-image and the sub-image with the best quality at the current moment can be calculated by the following formula:

[0088] {cx,cy}=arg_max{S(pic_A,pic_B,sx+i,sy+j),i=[-wx,wx],j=[-wy,wy]}

[0089] In the above formula, cx represents the lateral offset in the real-time offset at the current moment, cy represents the longitudinal offset in the real-time offset at the current moment; pic_A represents the fingerprint sub-image, pic_B represents the optimal quality sub-image; S represents the similarity of the overlapping parts of pic_A and pic_B under offset; sx represents the lateral offset in the current reference offset, sy represents the longitudinal offset in the current reference offset; -wx and wx represent the negative offset margin and positive offset margin along the horizontal direction of the image in the preset offset margin, respectively, and -wy and wy represent the negative offset margin and positive offset margin along the vertical direction of the image in the preset offset margin, respectively. Referring to the above, it can be seen that "sx+i,sy+j, i=[-wx,wx], j=[-wy,wy]" can also represent the real-time offset search range (which can be understood in conjunction with Figure 2C). The above formula can be understood as follows: the actual offset at the current moment needs to be traversed and searched within the above real-time offset search range to select the offset corresponding to the maximum similarity as the actual offset at the current moment in the final output.

[0090] In the present application, the aforementioned similarity can be used to describe the correlation between the overlapping portions of the fingerprint sub-image and the optimal quality sub-image, and can be implemented using any suitable indicator. Optionally, the similarity can be calculated using at least one indicator selected from the group consisting of Normalized Cross Correlation (NCC), SSIM (structural similarity), and MSE (mean-square error). The greater the NCC and SSIM values, the greater the similarity, while the smaller the MSE value, the greater the similarity.

[0091] Based on this, the present application can accurately and reliably determine the real-time offset between the fingerprint sub-image and the sub-image with the best quality at the current moment through the optional method of the above-mentioned sub-steps S10221 to S10222, thereby facilitating the subsequent steps to achieve alignment and image fusion of multiple fingerprint sub-images through reliable real-time offset, thereby being able to more effectively adapt to the deviations caused by different fingerprint states and pressing states of fingers on the three-dimensional surface, and is more conducive to improving the image quality of the fused image obtained after the fusion of multiple fingerprint sub-images, so as to facilitate the subsequent use of the fused image for data processing.

[0092] In some optional embodiments, the fingerprint image processing method of the present application further includes: updating the reference offset data according to the real-time offset between each fingerprint sub-image and the sub-image with the best quality at the current moment.

[0093] The offset between the fingerprint sub-images detected by each pixel unit will change slightly due to the temperature of the optical fingerprint sensor, and will also change due to the offset of the optical fingerprint sensor module in extreme cases, such as an abnormal drop (for example, a fall) of the electronic device equipped with the optical fingerprint sensor. Therefore, by updating the reference offset data in this application, the updated reference offset data can better adapt to the changes of the optical fingerprint sensor in different environments and different states, thereby making the subsequent real-time offset more accurate and reliable, and facilitating further data processing.

[0094] In some optional embodiments, referring to the flowchart shown in FIG. 3B , the method for updating the reference offset data includes the following sub-steps S202 , S204 , and S206 . Specifically:

[0095] S202: Determine reference real-time offsets between fingerprint sub-images captured by multiple second pixel units and fingerprint sub-images captured by the first pixel unit, respectively, among the multiple fingerprint sub-images based on the current real-time offsets between each fingerprint sub-image and the sub-image with the best quality.

[0096] As described above, the reference offset data may include reference offsets between multiple reference fingerprint sub-images and the reference optimal quality sub-image. In this application, updating the reference offset data is to update each reference offset. Therefore, it is necessary to convert it into the real-time offset between the corresponding reference fingerprint sub-image and the reference optimal quality sub-image (that is, the reference real-time offset) for calculation.

[0097] Since the reference optimal quality sub-image is collected by the first pixel unit and the reference fingerprint sub-image is collected by the second pixel unit, each reference real-time offset can be converted into each real-time offset.

[0098] For example, let's still take the example of nine pixel units capturing nine images, pixel unit 1 being the first pixel unit capturing the reference sub-image with the best quality, and pixel units 2 to 9 being the second pixel units capturing the reference fingerprint sub-images. If the sub-image with the best quality determined in step S1021 is the fingerprint sub-image captured by pixel unit 2, then if the real-time deviation between the fingerprint sub-image captured by pixel unit 1 (i.e., the first pixel unit) and the sub-image with the best quality captured by pixel unit 2 is (x2, y2), then the reference real-time offset between the fingerprint sub-image captured by pixel unit 2 and the fingerprint sub-image captured by pixel unit 1 (i.e., the first pixel unit) can be converted to (-x2, -y2). Similarly, the reference real-time offsets between the fingerprint sub-images captured by the other multiple second pixel units (pixel units 3 to 9) and the fingerprint sub-image captured by the first pixel unit (pixel unit 1) can also be calculated based on the respective real-time offsets. It should be understood that this is merely an example for ease of understanding and is not intended to limit the present application.

[0099] S204: For each reference real-time offset, determine the reference offset between the reference fingerprint sub-image captured by the corresponding second pixel unit and the reference optimal quality sub-image, and perform a weighted sum of the reference real-time offset and the reference offset according to the updated weight. (Please continue to understand this sub-step S204 in conjunction with the flowchart of Figure 3B.)

[0100] For example, let's continue with the example of nine pixel units capturing nine images, with pixel unit 1 being the first pixel unit capturing the reference sub-image of optimal quality, and pixel units 2-9 being the second pixel units capturing the reference fingerprint sub-image. For the reference real-time offset between the fingerprint sub-image captured by any pixel unit from 2-8 and the fingerprint sub-image captured by pixel unit 1, for example, taking pixel unit 2 as an example, the reference offset between the reference fingerprint sub-image captured by pixel unit 2 and the reference sub-image of optimal quality is determined. The reference real-time offset between pixel unit 2 and pixel unit 1, and the reference offset between the reference fingerprint sub-image captured by pixel unit 2 and the reference sub-image of optimal quality are weightedly summed according to the updated weights to obtain the weighted summation result corresponding to pixel unit 2. Similarly, for pixel units 3-9, the weighted summation can be performed in accordance with the calculation method for pixel unit 2, obtaining the corresponding weighted summation results. That is, for the eight reference real-time offsets corresponding to pixel units 2-9, eight weighted summation results can be obtained. It should be understood that this is merely an example for ease of understanding and is not intended to limit the present application.

[0101] Optionally, in sub-step S204, the reference real-time offset and the reference offset may be weighted and summed according to the following two formulas to obtain the corresponding weighted summation result: sx(t)=(1-w)*sx(t-1)+w*Cx sy(t)=(1-w)*sy(t-1)+w*Cy

[0102] Where w and 1-w represent the update weights, with w∈(0,1). Cx represents the lateral offset of the reference real-time offset. sx(t) and sx(t-1) represent the lateral offsets of the reference offsets at the previous and next moments, respectively. In the above embodiment, sx(t) can be understood as the lateral offset of the updated reference offset, and sx(t-1) can be understood as the lateral offset of the reference offset before the update.

[0103] Where Cy represents the longitudinal offset in the reference real-time offset, and sy(t) and sy(t-1) represent the longitudinal offsets in the reference offsets at the preceding and subsequent moments, respectively. In the above embodiment, sy(t) can be understood as the longitudinal offset in the updated reference offset, and sy(t-1) can be understood as the longitudinal offset in the reference offset before the update.

[0104] The horizontal offset and the vertical offset in the updated reference offset are calculated by the above two formulas, that is, an updated reference offset between a reference fingerprint sub-image and a reference optimal quality sub-image is obtained, that is, a weighted summation result is obtained.

[0105] It should be understood that the updated reference offsets obtained by weighted summation with different update weights may be different. Optionally, the update weights w and 1-w may be preset values ​​(which may be understood as initialized update weights) before the fingerprint image processing method of this solution is executed for the first time.

[0106] S206: Update the reference offset data based on the weighted summation results corresponding to each reference real-time offset. (Please continue to understand this sub-step S206 in conjunction with the flowchart of Figure 3B)

[0107] Each weighted summation result is an updated reference offset between a reference fingerprint sub-image and a reference sub-image with the best quality. According to each updated reference offset, each updated reference offset is used as updated reference offset data, thereby achieving the update of the reference offset data.

[0108] Based on this, the present application includes the optional implementation of the above-mentioned sub-steps S202 to S206, which can effectively and reliably update the reference offset data. The updated reference offset data can better adapt to the changes of the optical fingerprint sensor in different environments and different states, thereby making the subsequently determined real-time offset more accurate and reliable, so as to facilitate further data processing.

[0109] In some optional embodiments, the fingerprint image processing method in the present application further includes: adjusting the update weight based on at least one of the image quality of the fingerprint sub-image corresponding to the reference real-time offset and the similarity between the overlapping part of the fingerprint sub-image corresponding to the reference real-time offset and the sub-image with the best quality.

[0110] In the present application, the update weight is adjusted based on at least one of the image quality of the fingerprint sub-image corresponding to the reference real-time offset and the similarity between the overlapping part of the fingerprint sub-image corresponding to the reference real-time offset and the sub-image with the best quality, so as to better update the reference offset data, so that the reference offset data can better adapt to the changes of the optical fingerprint sensor in different environments and different states, and make the subsequently determined real-time offset more accurate and reliable, so as to facilitate further data processing.

[0111] The image quality of the fingerprint sub-image can be determined by calculating one or more indicators such as Harris response (HR), normalized gradient (SR), peak signal-to-noise ratio (PSNR), and normalized signal size (NSR), and determining the image quality of the fingerprint sub-image based on the calculation results. Similarity can be calculated using at least one similarity indicator such as normalized cross correlation (NCC), structural similarity (SSIM), and mean-square error (MSE). These related contents have been similarly introduced above and will not be repeated here.

[0112] Optionally, in some optional embodiments, the better the image quality of the fingerprint sub-image corresponding to the reference real-time offset, the higher the adjusted update weight, which can make the calculated update weight after adjustment more reliable.

[0113] Alternatively, in some other optional embodiments, the higher the similarity between the overlapping portion of the fingerprint sub-image corresponding to the reference real-time offset and the sub-image with the best quality, the higher the adjusted update weight. This can also make the calculated update weight after adjustment more reliable.

[0114] Alternatively, in some further embodiments, when the similarity between the overlapping portion of the fingerprint sub-image corresponding to the reference real-time offset and the optimal quality sub-image exceeds a certain similarity threshold, the update weight may be adjusted based on the image quality of the fingerprint sub-image corresponding to the reference real-time offset. Specifically, the better the image quality of the fingerprint sub-image corresponding to the reference real-time offset, the higher the adjusted update weight. This can also improve the reliability of the calculated update weight after adjustment.

[0115] Optionally, in the present application, when adjusting the update weight, the update weight may be adjusted within a preset update weight range. The update weight range may be preset as needed. For example, the update weight range of the update weight w may be, for example, 0.2-0.7, 0.1-0.6, etc. It is understood that the corresponding 1-w may vary with changes in w.

[0116] S1023: Align the multiple fingerprint sub-images based on the current real-time offset between each fingerprint sub-image and the sub-image with the best quality, and fuse the aligned multiple fingerprint sub-images to obtain a fused image. (Please continue to understand this sub-step S1023 in conjunction with the flowchart in Figure 2A.)

[0117] Optionally, each fingerprint subimage other than the subimage with the best quality can be aligned with its corresponding real-time offset (including horizontal and vertical offsets) at the current moment, so that the overlapping portion between the fingerprint subimage and the subimage with the best quality is aligned. After all fingerprint subimages are aligned with the subimage with the best quality, multiple fingerprint subimages can be aligned. The aligned multiple fingerprint subimages can then be fused to produce a fused image. This image effectively aggregates the information from the multiple captured fingerprint subimages, has a larger field of view, and exhibits low spatial noise. This facilitates processing the fused image to produce an output image for fingerprint recognition, thereby enabling real-time fingerprint recognition.

[0118] In the present application, since there is an offset between each fingerprint sub-image, the area of ​​the fused image is larger than that of a single fingerprint sub-image, that is, the field of view of the fused image is larger than that of a single fingerprint sub-image. Therefore, by processing the fused image to obtain an output image for fingerprint recognition, the effective information of multiple fingerprint sub-images can be better aggregated, so that the output image can utilize the information of the large field of view of the fused image and increase the recognition field of view of fingerprint recognition.

[0119] In some optional embodiments, referring to the flowchart shown in FIG3C , step S1023 of “fusing the aligned multiple fingerprint sub-images to obtain a fused image” includes sub-steps S10231 and S10232. Specifically:

[0120] S10231: For any overlapping area of ​​the aligned multiple fingerprint sub-images, based on the local image quality of each fingerprint sub-image in the overlapping area and the similarity between each fingerprint sub-image in the overlapping area and the overlapping part of the sub-image with the best quality, the local areas of each fingerprint sub-image in the overlapping area are fused to obtain a local fusion result.

[0121] In the present application, the aligned multiple fingerprint sub-images may have multiple overlapping areas, and each overlapping area may include local content of the multiple fingerprint sub-images.

[0122] In sub-step S10231 of the present application, the local image quality of each fingerprint sub-image in the overlapping area and the similarity between each fingerprint sub-image in the overlapping area and the overlapping part of the best quality sub-image are used to fuse the parts of each fingerprint sub-image in the overlapping area. This is because: the local image quality of the fingerprint sub-image can describe the clarity of the local fingerprint sub-image. The clearer the local fingerprint sub-image, the better the local image quality (it should be understood that the local image quality score below is also higher); and the similarity between the overlapping part of the fingerprint sub-image and the best quality sub-image can describe the consistency of the texture between the fingerprint sub-image and the best quality sub-image. After multiple fingerprint sub-images are aligned, the more consistent the texture at the same position, the higher the similarity. Therefore, in the present application, when fusing the parts of each fingerprint sub-image in the overlapping area, the above multiple dimensions are comprehensively considered. First, the above-mentioned local image quality is taken into consideration, so that the output local fusion result contains the local clear area of ​​each fingerprint sub-image as much as possible, avoiding the local blurred areas of each fingerprint sub-image from entering the fusion too much, resulting in blurred output local fusion result, and thus avoiding the blurring of the subsequent fusion image; secondly, the above-mentioned similarity is taken into consideration to suppress the areas where the texture of each fingerprint sub-image is inconsistent with the texture of the sub-image with the best quality, so as to avoid the distortion and blurring of the image texture of the output local fusion result, and thus avoid the distortion and blurring of the image texture of the subsequent fusion image.

[0123] In some optional embodiments, referring to the flowchart shown in FIG3D , step S10231 of “fusing the parts of the fingerprint sub-images within the overlapping area according to the local image quality of the fingerprint sub-images within the overlapping area and the similarity between the fingerprint sub-images within the overlapping area and the overlapping parts of the sub-image with the best quality” includes sub-steps S10231A, S10231B, and S10231C. Specifically:

[0124] S10231A: Determine the local image quality score of each fingerprint sub-image in the overlapping area.

[0125] Optionally, for any overlapping region, an image quality score can be calculated for each local portion of each fingerprint sub-image using one or more of the following indicators: Harris response (HR), normalized gradient (SR), peak signal-to-noise ratio (PSNR), normalized signal strength (NSR), etc., to obtain a local image quality score for each fingerprint sub-image in the overlapping region. These related contents have been similarly introduced above and will not be repeated here.

[0126] Afterwards, the fingerprint sub-images in the overlapped area can be sorted according to the local image quality score. Here, all fingerprint sub-images in the overlapped area (including the sub-image with the best quality if there is one) need to be calculated. For example, it can be assumed that the local image quality scores of the fingerprint sub-images after sorting are from high to low: Q1, Q2, Q3, ..., Q n Correspondingly, the fingerprint sub-images corresponding to the above order can be assumed to be: P1, P2, P3, ..., P n .

[0127] S10231B: Determine the fusion weight corresponding to each fingerprint sub-image in the overlapped area according to the obtained quality scores of each local image and the similarity between the overlapping portion of each fingerprint sub-image in the overlapped area and the sub-image with the best quality.

[0128] For example, the similarity between each fingerprint sub-image in the overlapping area and the overlapping part of the sub-image with the best quality (as mentioned above, the similarity can be calculated by at least one indicator among the correlation NCC, structural similarity SSIM, mean square error MSE, etc., wherein the larger the correlation NCC and structural similarity SSIM are, the greater the similarity is, and the smaller the mean square error MSE is, the greater the similarity is.) can be assumed to be S1, S2, S3, ..., S n It should be noted that the calculation needs to be performed here for all fingerprint sub-images in the overlapping area (including the best quality sub-image if there is one). Optionally, the similarity of the best quality sub-image to itself can be defined as 100.

[0129] Afterwards, the quality scores Q1, Q2, Q3, ..., Q n and each similarity S1, S2, S3, ..., S n Determine the fingerprint sub-images P1, P2, P3, ..., P in the overlapping area n Corresponding fusion weights. For example, corresponding to the above example of 9 fingerprint sub-images, if the local contents of these 9 fingerprint sub-images all exist in a certain overlapping area, then n = 9; if t (t < 9) of these 9 fingerprint sub-images exist in a certain overlapping area, then n = t.

[0130] S10231C: performing weighted summation on the local pixel values ​​of each fingerprint sub-image according to the obtained fusion weights, so as to fuse the local portions of each fingerprint sub-image in the overlapping area.

[0131] The local fusion result is a local image. The local pixel values ​​of each fingerprint sub-image can be weighted and summed by the obtained fusion weights to obtain the pixel values ​​of each pixel point of the local fusion result, thereby achieving the purpose of fusing the local parts of each fingerprint sub-image in the overlapping area and obtaining the local fusion result.

[0132] Based on this, the present application includes the optional implementation of the above sub-steps S10231A to S10231C, which can reliably and effectively fuse the parts of each fingerprint sub-image in the overlapping area, thereby obtaining a reliable local fusion result to facilitate the subsequent generation of a fused image.

[0133] In some optional embodiments, if the local image quality score corresponding to the fingerprint sub-image is higher and the similarity is greater, the fusion weight corresponding to the fingerprint sub-image is determined to be higher.

[0134] In the present application, by determining a higher fusion weight for a fingerprint sub-image with a higher local image quality score and a greater similarity, the image quality of the local fusion result after fusion can be effectively and better guaranteed, and the noise brought by the fingerprint sub-image with poor local image quality during local image fusion can be weakened, which is conducive to avoiding the situation where the image quality of the local fusion result after fusion is worse than the sub-image with the best quality before fusion.

[0135] It can be understood that, in the optional implementation manner of the present application, when locally fusing each fingerprint sub-image within the overlapping area, first, the output local fusion result can be made to include the local clear area of ​​each fingerprint sub-image with a high weight as much as possible, so as to avoid the local blurred area of ​​each fingerprint sub-image entering the fusion with a high weight, resulting in the blurring of the output local fusion result, thereby avoiding the blurring of the subsequently obtained fused image; secondly, the area in which the texture of each fingerprint sub-image is inconsistent with the texture direction of the sub-image with the best quality can be suppressed, so as to avoid the distortion and blurring of the image texture of the output local fusion result, thereby avoiding the distortion and blurring of the image texture of the subsequently obtained fused image.

[0136] For example, the above technical effects can be understood with reference to FIG3E . FIG3E shows a schematic diagram of partially fusing two images. As shown in FIG3E , two images (i.e., image A and image B) are shown with a total of 9 overlapping regions (a1 and b1 overlap, a2 and b2 overlap, ..., a9 and b9 overlap). In some overlapping regions, the local image quality score of image A is higher than that of image B (for example, a1 is better than b1, a4 is better than b4, a5 is better than b5, and a7 is better than b7). In other overlapping regions, the local image quality score of some regions of image B is better than that of image A (for example, b2 is better than a2, b3 is better than a3, b6 is better than a6, b8 is better than a8, and b9 is better than a9). Image C shows an image obtained by directly superimposing and fusing image A and image B in each overlapping region. Each overlapping region of image A and image B is directly fused with a fusion weight of 50%. The local image quality score of each region of the resulting image C is lower than that of the image with the best local image quality in the corresponding overlapping region. Image D shows an image obtained by fusing parts of image A and parts of image B with higher fusion weights according to the higher local image quality scores in each overlapping area (it should be noted that, for the sake of more intuitive understanding, image D shows a special case, which fuses the parts with higher local image quality scores with a fusion weight of 100%, so that the parts of image A or the parts of image B with higher local image quality scores in the overlapping area can completely enter image D). It can be seen that image D has better image quality than image C. In other words, determining a higher fusion weight for fusing images with higher local image quality scores in the overlapping area can effectively and better guarantee the image quality of the local fusion results after fusion, thereby reflecting the above-mentioned technical effects. It should be understood that the description of Figure 3E does not constitute any limitation to this application.

[0137] Optionally, in this application, the above sub-step S10231C can be implemented by the following formula:

[0138] Among them, BP is used to represent the local fusion result, P i It is used to represent each fingerprint sub-image (P1, P2, P3, ..., P n ), when substituted into the calculation, it can be calculated by the pixel value of the pixel point; Q i Used to represent the local image quality score (Q1, Q2, Q3, ..., Q n ), Q1 is the maximum value of each local image quality score; S i Used to represent the fingerprint sub-image P i The similarity with the overlapping part of the best quality sub-image (S1, S2, S3, ..., S n), S1 is used to represent the similarity corresponding to the fingerprint sub-image P1 with a local image quality score of Q1; n is the total number of fingerprint sub-images in the overlapping area.

[0139] From the above formula, it can be seen that for any fingerprint sub-image P in the overlapping area i , and its fusion weight is: And it can be seen that the fingerprint sub-image P i The corresponding local image quality score Q i The higher the similarity S i The larger the value is, the more accurate the fingerprint sub-image P is. i The higher the corresponding fusion weight, the more effectively the image quality of the local fusion result can be guaranteed.

[0140] When calculating through the above formula, each fingerprint sub-image P i The pixel values ​​of the corresponding pixels in the overlapping area are substituted to implement the weighted sum operation, and the pixel values ​​of each pixel of the local fusion result are obtained, so that the local fusion of each fingerprint sub-image in the overlapping area can be achieved.

[0141] S10232: The local fusion results corresponding to the overlapping areas of the aligned multiple fingerprint sub-images are stitched together to obtain the fused image. (Please continue to understand this sub-step S10232 in conjunction with the flowchart in Figure 3C)

[0142] After obtaining the local fusion results corresponding to the overlapping areas, the local fusion results can be stitched together into a fusion image, that is, a fusion image. The fusion image can then be further processed in subsequent steps S104 to S108 to obtain an output image for fingerprint recognition.

[0143] Based on this, the present application includes an optional implementation method of the above-mentioned sub-steps S10231 to S10232, while taking into account the local image quality of each fingerprint sub-image in the overlapping area and its similarity to the sub-image with the best quality, so as to perform local fusion of the fingerprint sub-images in the overlapping area, so that the fused image spliced ​​by the various local fusion results is more reliable, so that the obtained fused image can better aggregate the effective information of multiple fingerprint sub-images, and then the output image for fingerprint recognition is obtained by processing the fused image, so that the output image can better utilize the information of the large field of view of the fused image, so that the fingerprint recognition field of the output image is larger, the spatial noise is lower, and the signal-to-noise ratio is higher, which is conducive to improving the recognition success rate.

[0144] It should also be noted that in other optional embodiments, when "for any overlapping region of the aligned multiple fingerprint sub-images, locally fusing the fingerprint sub-images within the overlapping region to obtain a local fusion result," this can also be achieved based solely on the local image quality of the fingerprint sub-images within the overlapping region, or solely on the similarity between the overlapping portions of the fingerprint sub-images within the overlapping region and the sub-image with the best quality. In other words, the fusion weight can also only consider local image quality or similarity, as long as it meets the requirements.

[0145] Through the optional implementation of the above-mentioned sub-steps S1021 to S1023, the present application can more accurately determine and realize the alignment and image fusion of multiple fingerprint sub-images according to the real-time offset of each fingerprint sub-image, thereby more effectively adapting to the deviations caused by different fingerprint states and pressing states of the fingers on the three-dimensional surface, which is beneficial to improving the quality of the fused image obtained after the fusion of multiple fingerprint sub-images, so as to facilitate the subsequent output image obtained by processing the fused image; and, since the present solution obtains the output image for fingerprint recognition by processing the fused image obtained by fusing multiple fingerprint sub-images after alignment according to each real-time offset, it can better aggregate the effective information of multiple fingerprint sub-images, so that the output image can better utilize the information of the large field of view of the fused image, so that the recognition field of the fingerprint recognition of the output image is larger, the spatial noise is lower, and the signal-to-noise ratio is higher, which is beneficial to improving the recognition success rate.

[0146] S104: Determine, from the fused image, a first image region corresponding to the position of the first target sub-image in the multiple fingerprint sub-images, and extend the edges of the first image region based on the fused image to obtain a first extended image. (Please continue to understand this step S104 in conjunction with the flowchart in Figure 1.)

[0147] In the present application, after obtaining a fused image of a larger image size, a first image region corresponding to the location of a first target sub-image can be determined from the fused image. The first target sub-image can be any one of a plurality of fingerprint sub-images. The first image region and the first target sub-image have the same size.

[0148] Afterwards, the edge of the first image region can be expanded based on the fused image to obtain a first expanded image. Due to the expansion, the size of the first expanded image is larger than the first image region, and its recognition field of view is larger than that of a single fingerprint sub-image and the first image region.

[0149] Optionally, the first target sub-image may be the fingerprint sub-image with the best image quality among the multiple fingerprint sub-images. In other words, the first target sub-image may be the sub-image with the best quality. Therefore, step S104 may be to determine, from the fused image, the first image region corresponding to the fingerprint sub-image with the best image quality among the multiple fingerprint sub-images.

[0150] Based on this, the image quality of the first extended image obtained subsequently can be effectively improved, and the image quality of the second extended image obtained by subsequently extending the edge of the input image can also be effectively improved, thereby ensuring the image quality of the output image used for fingerprint recognition, and further effectively improving the recognition success rate of fingerprint image recognition.

[0151] In other optional embodiments, the first target sub-image may also be a fingerprint sub-image with the second best image quality among the multiple fingerprint sub-images, or may be another fingerprint sub-image that meets the requirements.

[0152] As previously mentioned, the optimal quality sub-image in this application can be determined using any suitable method, which is not limited here. For example, any suitable algorithm can be used to calculate one or more indicators such as the Harris Response (HR), Normalized Sobel Response (SR), Peak Signal-to-Noise Ratio (PSNR), and Normalized Stoichiometric Ratio (NSR, which represents the ratio between the peak-to-peak value and the local mean) of each fingerprint sub-image. The image quality of multiple fingerprint sub-images can be determined based on the calculation results to determine the optimal quality sub-image with the best image quality. Optionally, by calculating one or more of the above indicators, each fingerprint sub-image can be scored for image quality, and the fingerprint sub-image with the highest image quality score can be determined as the optimal quality sub-image, thereby determining the optimal quality sub-image as the first target sub-image. Furthermore, the fingerprint sub-image with the second highest image quality score is the fingerprint sub-image with the second-best image quality (e.g., also referred to as the sub-optimal quality sub-image hereinafter). In addition, as previously mentioned, multiple fingerprint sub-images can be sorted according to image quality. The relevant content can be understood by referring to the previous text and will not be repeated here.

[0153] The present application does not limit the specific method of "extending the edge of the first image region according to the fused image to obtain the first extended image" in step S104. For example, in some optional embodiments, referring to the flowchart shown in FIG4A, it may include sub-steps S1041 and S1042, specifically:

[0154] S1041: Determine a second image region including the first image region from the fused image, wherein the second image region exceeds an edge of at least one side of the first image region.

[0155] In this application, after determining the first image region from the fused image, the edge of at least one side of the first image region can be extended outward, so that the first image region is extended to a larger second image region in the fused image. In this optional embodiment, the second image region can also be understood as an ROI (Region of Interest) in the fused image.

[0156] It is understandable that both the fingerprint sub-image and the first image region include four side edges. In this application, when determining the second image region including the first image region from the fused image, the second image region can be extended to one or more side edges of the first image region as needed, as long as the requirements are met.

[0157] The expansion size can be selected to meet recognition performance requirements. It should be understood that when expanding an image, larger is not necessarily better; rather, it should be selected to meet the recognition success rate and speed requirements. For example, for some blurred fused images that affect recognition success rate, an expansion size that is too large may cause texture mismatch, resulting in a decrease in recognition success rate. For another example, large images generally have slower recognition speeds, so an expansion size that is too large may also result in a decrease in recognition speed. Therefore, the appropriate expansion size can be selected based on actual needs.

[0158] For example, in some optional embodiments, if the four side edges of the first image region do not overlap with the edges of the fused image, the four side edges of the second image region respectively exceed the corresponding four side edges of the first image region.

[0159] For this situation, as shown in Figure a in Figure 5, the first image area corresponding to the first target sub-image can be located in the middle of the fused image, and its four side edges do not overlap with the edges of the fused image. Then, the four side edges of the first image area can be extended in four directions in the fused image (the arrows indicate the directions of extension) to determine the second image area.

[0160] For example, in some other optional embodiments, if at least one side edge of the first image region coincides with an edge of the fused image, the second image region extends beyond the side edges of the first image region that do not coincide with the edge of the fused image.

[0161] For this situation, referring to Figures b and c in Figure 5, the first image area corresponding to the first target sub-image can be located at the corner of the fused image, and its two side edges do not coincide with the edges of the fused image. Then, the other two side edges of the first image area can be extended in the other two directions in the fused image (the arrows indicate the directions of extension, as shown in the figure, the example in Figure b can be extended downward and right; the example in Figure c can be extended upward and left), thereby determining the second image area.

[0162] Of course, when the first image area corresponding to the first target sub-image is located at an edge of the fused image, that is, when only one side edge of the first image area coincides with one side edge of the fused image (for example, coincides with the upper edge of the fused image), the other three side edges of the first image area can be extended in the other three directions in the fused image (for example, downward, left, and right), thereby determining the second image area.

[0163] Based on this, the above optional solution can adapt to different position states of the first image area in the fused image, effectively determine the second image area including the first image area, and effectively realize the expansion of the edge of the first image area according to the fused image.

[0164] It should be understood that when expanding the edges of the first image area, at most four sides of the first image area can be extended to the four sides of the fused image. That is, the entire fused image is determined to be the second image area. It should also be understood that when expanding the side edges of the first image area, the extension size on each side can be the same or different, as long as the requirements are met. For example, in Figure 5a, the extension size of the first image area to the left and right edges can be slightly larger than the extension size to the top and bottom edges. Of course, this is merely an example and not a limitation of this application.

[0165] S1042: Cut out the second image area from the fused image, and use the cutout result as the first extended image.

[0166] After the second image region including the first image region is determined from the fused image, the second image region can be cut out from the fused image, and the cutout result is the first extended image.

[0167] 5 ac, as shown in FIG, which shows three examples of extracting the second image region from the fused image to obtain the first extended image. It should be understood that this is not a limitation of the present application.

[0168] Based on this, the present application includes the optional implementation of the above-mentioned sub-steps S1041 to S1042, which can effectively expand the edge of the first image area to obtain a first extended image, so as to facilitate the subsequent determination of the output image for fingerprint recognition.

[0169] S106: Determine the input image based on the at least one fingerprint sub-image, and extend the edges of the input image based on the fused image to obtain a second extended image, wherein the at least one fingerprint sub-image includes the first target sub-image. (Please continue to understand this step S106 in conjunction with the flowchart in Figure 1.)

[0170] In the present application, the input image is determined based on at least one fingerprint sub-image including the first target sub-image, and is the same size as the fingerprint sub-image. In step S106, the size of the second extended image obtained by extending the edge of the input image based on the fused image is the same as the size of the first extended image obtained in step S104 above.

[0171] The present application does not limit the method for determining the input image. For example, in some optional embodiments, referring to the flowchart shown in FIG4B , step S106 of “determining the input image based on at least one fingerprint sub-image” includes the following sub-steps S1061 and S1062, specifically:

[0172] S1061: Locally fuse the overlapping portion between the first target sub-image and the second target sub-image in the multiple fingerprint sub-images to obtain a target local fused image.

[0173] S1062: Splicing an input image based on the portion of the first target sub-image that has not been locally fused and the target locally fused image.

[0174] Based on this, the present application obtains a target local fused image by locally fusing the overlapping portions of the first target sub-image and the second target sub-image. The input image is then spliced ​​together using the unfused portions of the first target sub-image and the target local fused image. This effectively obtains the input image, facilitating subsequent edge extension based on the input image to obtain the second extended image. Furthermore, using two fingerprint sub-images to generate the input image reduces the computational effort while maintaining the quality of the input image.

[0175] Referring to Figure 6, a schematic diagram illustrating locally fusing the overlapping portion of a first target sub-image and a second target sub-image to generate a target locally fused image and an input image is shown. It can be seen that the input image includes the target locally fused image (i.e., the result of local fusion) and the portion of the first target sub-image that has not been locally fused. It should be understood that this is merely an example for ease of understanding.

[0176] In the present application, the second target sub-image can be any fingerprint sub-image among the multiple fingerprint sub-images except the first target sub-image, and can be selected as needed.

[0177] For example, in some optional embodiments, if the first target sub-image is the fingerprint sub-image with the best image quality among multiple fingerprint sub-images (i.e., the best quality sub-image), then the second target sub-image is the fingerprint sub-image with the second best image quality among the multiple fingerprint sub-images (i.e., the second best quality sub-image).

[0178] Based on this, in this application, by locally fusing the overlapping parts of the sub-image with the best quality and the sub-image with the second best quality, the image quality of the input image can be better guaranteed, thereby improving the image quality of the second extended image obtained by expanding the input image.

[0179] In some optional embodiments, sub-step S1061 may be performed after aligning the offsets of the overlapping portion between the first target sub-image and the second target sub-image, and then performing local fusion to obtain the target local fused image. This helps accommodate deviations caused by different fingerprint states and pressing states of fingers on the three-dimensional surface, and helps improve the effect of local fusion of the overlapping portion between the first target sub-image and the second target sub-image, thereby ensuring the quality of the obtained input image and ensuring better quality of the second expanded image obtained subsequently.

[0180] In some optional embodiments, referring to the flowchart shown in FIG. 4C , sub-step S1061 may include sub-steps S1061A and S1061B. Specifically:

[0181] S1061A: Determine the target fusion weights corresponding to the first target sub-image and the second target sub-image respectively based on the first local image quality of the first target sub-image in the overlapping part and the second local image quality of the second target sub-image in the overlapping part, and the higher the local image quality, the higher the determined target fusion weight.

[0182] In the present application, the higher the local image quality of the first target sub-image and the second target sub-image, the higher the target fusion weight, which can ensure that the target local fusion image obtained by subsequent local fusion can retain more features of the image with higher local image quality, thereby improving the effect of local fusion.

[0183] Optionally, local image quality scores may be calculated for the first target sub-image and the second target sub-image in the overlapping portion. The calculated local image quality scores may represent the first local image quality and the second local image quality, respectively. Target fusion weights may be determined for the first and second local image qualities based on their corresponding local image quality scores. The higher the local image quality score, the higher the target fusion weight.

[0184] It should be understood that when calculating the local image quality score, one or more indicators such as Harris response HR, normalized gradient SR, peak signal-to-noise ratio PSNR, and normalized signal quantity NSR can be used for calculation. The relevant content has been similarly introduced in the previous article and will not be repeated here.

[0185] The target fusion weights corresponding to the first target sub-image and the second target sub-image, respectively, can be determined in any suitable manner. Optionally, assuming that the local image quality score (i.e., the first local image quality) corresponding to the first target sub-image in the overlapping portion is a, and the local image quality score (i.e., the second local image quality) corresponding to the second target sub-image in the overlapping portion is b, then the target fusion weight corresponding to the first target sub-image can be determined as: a / (a+b), and the target fusion weight corresponding to the second target sub-image can be determined as: b / (a+b). This also ensures that the higher the local image quality, the higher the target fusion weight determined.

[0186] S1061B: According to the target fusion weights corresponding to the first target sub-image and the second target sub-image respectively, the pixel values ​​of the first target sub-image and the pixel values ​​of the second target sub-image in the overlapping part are weightedly summed to locally fuse the overlapping part between the first target sub-image and the second target sub-image to obtain a target local fused image.

[0187] Specifically, by using the target fusion weights corresponding to the first target sub-image and the second target sub-image respectively, the pixel values ​​of the first target sub-image and the pixel values ​​of the second target sub-image in the overlapping part are weightedly summed to obtain the pixel values ​​of each pixel point of the target local fusion image, thereby realizing local fusion and obtaining the target local fusion image.

[0188] Based on this, by including the optional implementation of sub-steps S1061A-S1061B described above, it is possible to effectively locally fuse the overlapping portion of the first target sub-image and the second target sub-image to obtain a target locally fused image, thereby facilitating the determination of the input image in sub-step S1062. Furthermore, since the higher the local image quality of the first target sub-image and the second target sub-image, the higher the target fusion weight, the target locally fused image obtained by the subsequent local fusion can retain more features of the image with higher local image quality, thereby improving the local fusion effect. (Similarly, the technical effects here can also be understood in conjunction with the above description of FIG. 3E and will not be repeated here.)

[0189] In some optional embodiments, the step S106 of "extending the edge of the input image according to the fused image to obtain a second extended image" includes: determining a second image area including the first image area from the fused image, determining a third image area excluding the first image area within the second image area; and filling the third image area correspondingly to the outside of the edge of the input image to form a second extended image.

[0190] In the present application, by filling the third image area other than the first image area within the second image area to the edge of the input image accordingly, the edge of the input image can be quickly and effectively extended according to the fused image to obtain a second extended image, so as to facilitate the subsequent determination of the output image for fingerprint recognition.

[0191] For example, Figure 7 illustrates the location of a third image region in a fused image. As can be seen, the third image region corresponds to the portion of the first extended image that extends the first image region. This third image region is correspondingly filled beyond the edge of the input image, thereby achieving edge extension to produce the second extended image. The input image, the first target sub-image, and the first image region are all the same size, so the second extended image has the same size as the first extended image.

[0192] In some other optional embodiments, step S106 includes: determining the first target sub-image as the input image, and extending the edge of the first target sub-image according to the fused image to obtain a second extended image.

[0193] In the present application, through such an optional embodiment, the first target sub-image is directly determined as the input image, so that the input image can be effectively obtained, so as to facilitate subsequent edge extension based on the input image to obtain the second extended image.

[0194] Optionally, the first target sub-image here may also be the fingerprint sub-image with the best image quality among the multiple fingerprint sub-images (ie, the sub-image with the best quality), which may be determined based on the first target sub-image used in the first extended image.

[0195] In some optional embodiments, the above-mentioned “expanding the edge of the first target sub-image according to the fused image to obtain a second extended image” includes: determining a second image area including the first image area from the fused image, and determining a third image area excluding the first image area within the second image area; and filling the third image area correspondingly to the edge of the first target sub-image to form a second extended image.

[0196] In the present application, by filling the third image area other than the first image area within the second image area to the edge of the first target sub-image (i.e., the input image), the edge of the first target sub-image (i.e., the input image) can be quickly and effectively extended according to the fused image to obtain a second extended image, so as to facilitate the subsequent determination of the output image for fingerprint recognition.

[0197] For example, referring still to Figure 7, an example of the position of the third image region in the fused image is shown. It can be seen that the third image region corresponds to the portion of the first extended image that is an extension of the first image region. This third image region is correspondingly filled to the edge of the input image (first target sub-image), thereby achieving edge extension to obtain the second extended image. The input image, the first target sub-image, and the first image region are all the same size, so the second extended image has the same size as the first extended image.

[0198] Optionally, when the third image area is correspondingly filled outside the edge of the first target sub-image, the boundary between the third image area and the edge of the first target sub-image may be fused to avoid a boundary effect between the two.

[0199] S108: Determine one of the first expanded image and the second expanded image as the output image for fingerprint recognition. (Please continue to understand this step S108 in conjunction with the flowchart in Figure 1)

[0200] After the first extended image and the second extended image are obtained, one of the first extended image and the second extended image may be determined as an output image, so as to perform fingerprint recognition using the output image.

[0201] Based on this, the present application adopts the optional implementation method of the above steps S102 to S108. On the one hand, by outputting one of the first extended image and the second extended image as the output image for fingerprint recognition, it is possible to use a single better output image for fingerprint recognition when the optical fingerprint sensor collects multiple fingerprint sub-images at a single time, thereby effectively achieving the effect of reducing storage consumption during fingerprint recognition, and can also effectively improve the recognition speed and recognition success rate of fingerprint recognition; on the other hand, since the first extended image and the second extended image are both obtained by expanding the fused image, and the fused image better aggregates the information in the collected multiple fingerprint sub-images, has a larger field of view, and has lower spatial noise, one of the two is used as the output image for fingerprint recognition. Compared with directly using the fingerprint sub-image for fingerprint recognition, the output image utilizes the information of the large field of view of the fused image, so its recognition field of view is larger, the spatial noise is lower, and the signal-to-noise ratio is higher, which is conducive to improving the recognition success rate.

[0202] In this application, any rule can be used to determine one of the first extended image and the second extended image as the output image. For example, in some optional embodiments, in step S108, the one with higher image quality between the first extended image and the second extended image can be determined as the output image for fingerprint recognition. This can better ensure the image quality of the output image for fingerprint recognition, thereby effectively improving the fingerprint recognition effect.

[0203] Optionally, the image quality scores corresponding to the first extended image and the second extended image can be calculated respectively according to the method described above (the higher the image quality score, the higher the image quality, which can be determined by calculating relevant indicators (such as one or more of the Harris response HR, normalized gradient SR, peak signal-to-noise ratio PSNR, normalized signal quantity NSR, etc.). Similar content has been introduced in the previous article and will not be repeated here). Therefore, based on the image quality scores of the two, the one with higher image quality can be determined to facilitate determination of the output image.

[0204] The following describes the overall implementation of an example fingerprint image processing solution in the present application in conjunction with FIG8 . This solution corresponds to the above-mentioned optional method of "first locally fusing the overlapping parts of the first target sub-image and the second target sub-image to obtain a target local fusion image, and then splicing the input image based on the non-locally fused parts of the first target sub-image and the target local fusion image" (i.e., the optional method of the above-mentioned sub-steps S1061 to S1062). Referring to FIG8 , first, a plurality of fingerprint sub-images 1 to N respectively collected by a plurality of pixel units of the optical fingerprint sensor are obtained; then, after image quality evaluation and sorting of the fingerprint sub-images 1 to N, and then calculating the offset of the sub-image with the best quality relative to each other, the fingerprint sub-images with the image quality ranked 1 to N are obtained; then, all fingerprint sub-images (i.e., fingerprint sub-images 1 to N) can be aligned and offset and then fused to obtain a fused image; then, the sub-image with the best quality is used as the first target sub-image, and the fused image is used to extend the edge of the first target sub-image to obtain a first extended image; then, the fingerprint sub-image with the image quality ranked 1 is used as the first target sub-image (i.e., the sub-image with the best quality). sub-image), align the offsets of the overlapping parts of the fingerprint sub-image ranked second in image quality (i.e., the fingerprint sub-image with the second best image quality) and then perform local fusion to obtain a target local fusion image, and then obtain an input image (i.e., the optimal sub-optimal fusion image) based on the target local fusion image; then, expand the edge of the input image according to the fused image to obtain a second expanded image at the same position as the first expanded image; thereafter, the image quality of the first expanded image and the second expanded image can be evaluated, and the one with the best image quality can be selected as the output image for fingerprint recognition, and then the output image can be used for fingerprint recognition (it can also be used for fingerprint registration in other scenarios).

[0205] Alternatively, in conjunction with FIG9 , another example of an overall implementation of a fingerprint image processing solution is shown. The difference between FIG9 and FIG8 is that FIG8 uses the fingerprint sub-image ranked first in image quality (i.e., the sub-image with the best quality) and the fingerprint sub-image ranked second in image quality (i.e., the fingerprint sub-image with the second best quality) to achieve local fusion of the best and second best fingerprint sub-images to obtain the input image; while FIG9 traverses the fingerprint sub-images ranked 2nd to Nth in image quality and performs local fusion with the sub-image with the best quality to obtain multiple input images (N-1). Then, for each of the N-1 input images, N-1 second extended images are obtained. The image quality of the N-1 second extended images is evaluated with the first extended image, and the one with the best image quality is selected as the output image for fingerprint recognition. The output image can then be used for fingerprint recognition or registration. This implementation method in FIG9 is suitable for situations where memory is available and storage pressure is low.

[0206] It should be understood that the above description of FIG. 8 and FIG. 9 is only for facilitating understanding and does not constitute any limitation to the present application.

[0207] The implementation of the fingerprint image processing solution of the present application will be described below in conjunction with FIG10. FIG10 provides a more intuitive understanding of the implementation process of the technical solution of the present application in combination with schematic diagrams of various images. It should be understood that the following description of FIG10 does not limit the present application in any way.

[0208] As shown in FIG10 , the fingerprint image processing solution of the present application includes at least two optional solutions. Option 1 is the aforementioned "local fusion first, then expansion and image selection" solution, and Option 2 is the aforementioned "direct expansion and image selection without local fusion" solution. For ease of illustration, "fingerprint sub-image" is represented by "sub-image" in FIG10 .

[0209] Referring to FIG10 , in Option 1:

[0210] Step 1: Obtain multiple fingerprint sub-images 1 to N respectively collected by multiple pixel units of the optical fingerprint sensor at the current moment, and fuse the multiple fingerprint sub-images 1 to N to obtain a fused image.

[0211] Step 2: Determine from the fused image the first image region corresponding to the first target sub-image position in the plurality of fingerprint sub-images 1 to N; then determine from the fused image the second image region including the first image region (the second image region extends beyond the edge of at least one side of the first image region), cut out the second image region from the fused image, and use the cutout result as the first extended image (based on this, the edge of the first image region can be extended according to the fused image to obtain the first extended image).

[0212] It should be understood that in the example of Option 1, sub-image 1 is used as the first target sub-image, and sub-image 1 is used as the fingerprint sub-image with the best image quality (ie, the best quality sub-image) among the multiple fingerprint sub-images 1 to N.

[0213] Step 3: Locally fuse the overlapping parts of the first target sub-image and the second target sub-image in the multiple fingerprint sub-images to obtain a target local fused image, and then splice the un-locally fused parts of the first target sub-image and the target local fused image into an input image (based on this, the input image can be determined based on at least one fingerprint sub-image); then determine the second image area including the first image area from the fused image, determine the third image area excluding the first image area within the second image area, and fill the third image area to the edge of the input image accordingly to form a second extended image (based on this, the edge of the input image can be extended according to the fused image to obtain the second extended image).

[0214] It should be understood that in the example of Option 1, sub-image 1 is the first target sub-image, sub-image 2 is the second target sub-image, and sub-image 2 is the fingerprint sub-image with the second best image quality among the multiple fingerprint sub-images 1 to N.

[0215] Step 4: According to the image quality, the one with higher image quality between the first extended image and the second extended image is determined as the output image for fingerprint recognition (based on this, it can be achieved that one of the first extended image and the second extended image is determined as the output image).

[0216] Referring to FIG10 , in Option 2:

[0217] Step 1: Obtain multiple fingerprint sub-images 1 to N respectively collected by multiple pixel units of the optical fingerprint sensor at the current moment, and fuse the multiple fingerprint sub-images 1 to N to obtain a fused image.

[0218] Step 2: Determine from the fused image the first image region corresponding to the first target sub-image position in the plurality of fingerprint sub-images 1 to N; then determine from the fused image the second image region including the first image region (the second image region extends beyond the edge of at least one side of the first image region), cut out the second image region from the fused image, and use the cutout result as the first extended image (based on this, the edge of the first image region can be extended according to the fused image to obtain the first extended image).

[0219] It should be understood that in the example of Option 2, sub-image 3 is used as the first target sub-image, and sub-image 3 is the fingerprint sub-image with the best image quality among the multiple fingerprint sub-images 1 to N (i.e., the sub-image with the best quality). Sub-image 3 is used as the first target sub-image here mainly to facilitate distinction from Option 1 and to facilitate understanding, and does not represent a substantial difference between the two.

[0220] Step 3: Determine the first target sub-image as the input image, and determine the second image area including the first image area from the fused image, determine the third image area excluding the first image area within the second image area, and fill the third image area to the edge of the first target sub-image to form a second extended image (based on this, it is possible to determine the input image according to at least one fingerprint sub-image, and extend the edge of the input image according to the fused image to obtain the second extended image).

[0221] Step 4: According to the image quality, the one with higher image quality between the first extended image and the second extended image is determined as the output image for fingerprint recognition (based on this, it can be achieved that one of the first extended image and the second extended image is determined as the output image).

[0222] It should be understood that the above description of the fingerprint image processing method is merely an exemplary description of the embodiments of the present application and does not constitute any limitation to the embodiments of the present application.

[0223] According to the second aspect of the present application, according to the second aspect of the embodiment of the application, a fingerprint image processing device is provided. Referring to FIG11 , the fingerprint image processing device 1100 includes:

[0224] A fusion module 1102 is configured to obtain a plurality of fingerprint sub-images respectively collected by a plurality of pixel units of the optical fingerprint sensor at a current moment, and fuse the plurality of fingerprint sub-images to obtain a fused image;

[0225] a first expansion module 1104 configured to determine, from the fused image, a first image region corresponding to a position of a first target sub-image in the plurality of fingerprint sub-images, and to expand an edge of the first image region according to the fused image to obtain a first expanded image;

[0226] a second expansion module 1106, configured to determine an input image based on at least one fingerprint sub-image, and expand edges of the input image based on the fused image to obtain a second expanded image, wherein the at least one fingerprint sub-image includes the first target sub-image;

[0227] The determination module 1108 is configured to determine one of the first extended image and the second extended image as an output image for fingerprint recognition.

[0228] The solution provided in the embodiments of the present application, on the one hand, can output one of the first extended image and the second extended image as the output image for fingerprint recognition, so that when the optical fingerprint sensor collects multiple fingerprint sub-images at a time, a single better output image can be used for fingerprint recognition, thereby effectively reducing the storage consumption during fingerprint recognition, and can also effectively improve the recognition speed and recognition success rate of fingerprint recognition; on the other hand, since the first extended image and the second extended image are both obtained by expanding the fused image, and the fused image better aggregates the information in the multiple collected fingerprint sub-images, has a larger field of view, and has lower spatial noise, therefore, one of the two is used as the output image for fingerprint recognition. Compared with directly using the fingerprint sub-image for fingerprint recognition, the output image utilizes the information of the large field of view of the fused image, so its recognition field of view is larger, the spatial noise is lower, and the signal-to-noise ratio is higher, which is conducive to improving the recognition success rate.

[0229] In some optional embodiments, the first target sub-image is a fingerprint sub-image with the best image quality among the multiple fingerprint sub-images; and the first expansion module 1104 is specifically configured to determine, from the fused image, a first image region corresponding to the fingerprint sub-image with the best image quality.

[0230] In some optional embodiments, the determining module 1108 is specifically configured to determine the one with higher image quality between the first extended image and the second extended image as the output image for fingerprint recognition.

[0231] In some optional embodiments, the first expansion module 1104 is specifically used to: determine a second image area including the first image area from the fused image, wherein the second image area exceeds the edge of at least one side of the first image area; and cut out the second image area from the fused image, and use the cutout result as the first extended image.

[0232] In some optional embodiments, if the four side edges of the first image area do not overlap with the edges of the fused image, the four side edges of the second image area respectively exceed the corresponding four side edges of the first image area; or, if the edge of at least one side of the first image area overlaps with the edge of the fused image, the second image area exceeds the side edges of the first image area that do not overlap with the edges of the fused image.

[0233] In some optional embodiments, the first expansion module 1104 is specifically configured to: locally fuse the overlapping portion between the first target sub-image and a second target sub-image among the multiple fingerprint sub-images to obtain a target locally fused image, wherein the second target sub-image is any fingerprint sub-image among the multiple fingerprint sub-images except the first target sub-image; and splice the input image based on the portion of the first target sub-image that is not locally fused and the target locally fused image.

[0234] In some optional embodiments, the first expansion module 1104 is specifically used to: determine the target fusion weights corresponding to the first target sub-image and the second target sub-image respectively according to the first local image quality of the first target sub-image in the overlapping part and the second local image quality of the second target sub-image in the overlapping part, and if the local image quality is higher, the determined target fusion weight is higher; according to the target fusion weights corresponding to the first target sub-image and the second target sub-image respectively, perform weighted summation on the pixel values ​​of the first target sub-image and the pixel values ​​of the second target sub-image in the overlapping part to locally fuse the overlapping part between the first target sub-image and the second target sub-image to obtain a target local fusion image.

[0235] In some optional embodiments, the first expansion module 1104 is specifically used to: determine a second image area including the first image area from the fused image, determine a third image area excluding the first image area within the second image area; and fill the third image area correspondingly to the edge of the input image to form the second extended image.

[0236] In some optional embodiments, if the first target sub-image is the fingerprint sub-image with the best image quality among the multiple fingerprint sub-images, then the second target sub-image is the fingerprint sub-image with the second best image quality among the multiple fingerprint sub-images.

[0237] In some optional embodiments, the second expansion module 1106 is specifically configured to: determine the first target sub-image as the input image, and expand the edge of the first target sub-image according to the fused image to obtain the second expanded image.

[0238] In some optional embodiments, the second expansion module 1106 is specifically used to: determine a second image area including the first image area from the fused image, determine a third image area excluding the first image area within the second image area; and fill the third image area correspondingly to the edge of the first target sub-image to form the second extended image.

[0239] In some optional embodiments, the fusion module 1102 is specifically used to: determine the best quality sub-image with the best image quality among the multiple fingerprint sub-images; for each fingerprint sub-image other than the best quality sub-image among the multiple fingerprint sub-images, determine the similarity between the overlapping part of the fingerprint sub-image and the best quality sub-image according to the reference offset data, and determine the real-time offset between the fingerprint sub-image and the best quality sub-image at the current moment according to the similarity; align the multiple fingerprint sub-images according to the real-time offset between each fingerprint sub-image and the best quality sub-image at the current moment, and fuse the aligned multiple fingerprint sub-images to obtain the fused image.

[0240] The fingerprint image processing device 1100 provided in the second aspect of the present application is based on the same inventive concept as the fingerprint image processing method provided in the first aspect, corresponds to the corresponding fingerprint image processing methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding fingerprint image processing method embodiments. Therefore, a detailed description thereof will not be given here. Furthermore, the implementation of each module in the fingerprint image processing device 300 of this embodiment can refer to the corresponding descriptions of the aforementioned fingerprint image processing method embodiments, and will not be given here.

[0241] According to the third aspect of the embodiment of the present application, an electronic device is provided, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method of the first aspect.

[0242] 12 shows a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device. As shown in FIG12 , the electronic device 1200 may include: a processor 1202, a communications interface 1204, a memory 1206, and a communication bus 1208.

[0243] in:

[0244] The processor 1202 , the communication interface 1204 , and the memory 1206 communicate with each other via a communication bus 1208 .

[0245] The communication interface 1204 is used to communicate with other electronic devices or servers.

[0246] The processor 1202 is configured to execute the computer program 1210 , and specifically to execute the relevant steps in the above-mentioned fingerprint image processing method embodiment.

[0247] Specifically, the computer program 1210 may include program codes including computer operation instructions.

[0248] The processor 1202 may be a CPU, a GPU (Graphic Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0249] The memory 1206 is used to store the computer program 1210. The memory 1206 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0250] The computer program 1210 may include multiple computer instructions. Specifically, the computer program 1210 may enable the processor 1202 to execute operations corresponding to the fingerprint image processing method described in any of the aforementioned method embodiments through the multiple computer instructions.

[0251] The specific implementation of each step in the computer program 1210 can refer to the corresponding description of the corresponding steps and units in the above-mentioned method embodiment, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working process of the above-mentioned devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, and will not be repeated here.

[0252] According to a fourth aspect of the embodiments of the present application, the embodiments of the present application further provide a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the fingerprint image processing method described in any one of the multiple method embodiments provided in the first aspect. The computer storage medium includes, but is not limited to, a compact disc read-only memory (CD-ROM), random access memory (RAM), a floppy disk, a hard disk, or a magneto-optical disk.

[0253] According to a fifth aspect of the embodiments of the present application, the embodiments of the present application further provide a computer program product, including a computer program, which, when executed by a processor, implements the fingerprint image processing method as described in any one of the multiple method embodiments provided in the first aspect.

[0254] According to the sixth aspect of the embodiments of the present application, the embodiments of the present application further provide a fingerprint recognition device, which is applied to an electronic device with a display screen. As shown in Figure 13, the fingerprint recognition device 1300 includes: an optical fingerprint sensor 1302, which is used to: image light signals in multiple directions reflected by the finger above the display screen to collect multiple fingerprint sub-images; and a processing unit 1304, which is used to execute the fingerprint image processing method described in any one of the first aspects.

[0255] It should be understood that the optical fingerprint sensor adopts a multi-angle optical path design and can include multiple pixel units. The relevant content of the optical fingerprint sensor has been explained in the method embodiment of the first aspect above and will not be repeated here.

[0256] In addition, as mentioned above, the processing unit can be any processing unit. For example, the processing unit can be a processing unit of an electronic device (for example, including but not limited to a mobile terminal, a computer, a fingerprint lock, etc.) to which the optical fingerprint sensor is applied (for example, it may include one or more chips capable of data processing, including but not limited to a CPU (Central Processing Unit), an MCU (Microcontroller Unit), a GPU (Graphic Processing Unit), an FPGA (Field Programmable Gate Array), etc.), which is conducive to improving the processing effect.

[0257] The fingerprint image processing device 1100 / electronic device 1200 / computer storage medium / computer program product / fingerprint recognition device 1300 embodiments in the embodiments of the present application have been described in detail in the aforementioned fingerprint image processing method embodiments. Therefore, their relevant contents and beneficial effects can be understood with reference to the aforementioned method embodiments and will not be repeated here.

[0258] In addition, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used to train the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0259] It should be noted that, depending on the needs of implementation, the various components / steps described in the embodiments of this application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application. It should be understood that the various technical features in the technical solutions of the embodiments of this application can be combined in any appropriate manner.

[0260] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA)). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., random access memory (RAM), read-only memory (ROM), flash memory, etc.) that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown here.

[0261] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for specific applications, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0262] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.

[0263] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". It should be noted that the concepts of "first", "second", etc. mentioned in the embodiments of the present application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. It should be noted that the modifications of "one" and "multiple" mentioned in the embodiments of the present application are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0264] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present application, rather than to limit them. Although the embodiments of the present application have been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A fingerprint image processing method, comprising: Obtaining a plurality of fingerprint sub-images respectively collected by a plurality of pixel units of an optical fingerprint sensor at the current moment, and performing fusion processing on the plurality of fingerprint sub-images to obtain a fused image; Determining, from the fused image, a first image region corresponding to the position of a first target sub-image among the plurality of fingerprint sub-images, and expanding the edge of the first image region according to the fused image to obtain a first expanded image; Determining an input image according to at least one fingerprint sub-image, and expanding the edge of the input image according to the fused image to obtain a second expanded image, wherein the at least one fingerprint sub-image includes the first target sub-image; Determining one of the first expanded image and the second expanded image as an output image for fingerprint recognition.

2. The method according to claim 1, wherein The first target sub-image is the fingerprint sub-image with the best image quality among the plurality of fingerprint sub-images; The determining, from the fused image, a first image region corresponding to the position of a first target sub-image among the plurality of fingerprint sub-images includes: Determining, from the fused image, a first image region corresponding to the fingerprint sub-image with the best image quality.

3. The method according to claim 1, wherein The determining one of the first expanded image and the second expanded image as an output image for fingerprint recognition includes: Determining the one with higher image quality among the first expanded image and the second expanded image as the output image for fingerprint recognition.

4. The method according to any one of claims 1 to 3, wherein The expanding the edge of the first image region according to the fused image to obtain a first expanded image includes: Determining a second image region including the first image region from the fused image, wherein the second image region extends beyond the edge of at least one side of the first image region; Cropping out the second image region from the fused image, and using the cropping result as the first expanded image.

5. The method according to claim 4, wherein, If the four side edges of the first image region do not coincide with the edges of the fused image, the four side edges of the second image region respectively extend beyond the corresponding four side edges of the first image region; or, If at least one side edge of the first image region coincides with the edge of the fused image, the second image region extends beyond the side edges of the first image region that do not coincide with the edge of the fused image.

6. The method according to any one of claims 1 to 3, wherein The determining an input image according to at least one fingerprint sub-image includes: Performing local fusion on the overlapping part between the first target sub-image and a second target sub-image among the plurality of fingerprint sub-images to obtain a target local fused image, wherein the second target sub-image is any one of the plurality of fingerprint sub-images other than the first target sub-image; Splicing the non-locally fused part in the first target sub-image and the target local fused image to form the input image.

7. The method according to claim 6, wherein, The performing local fusion on the overlapping part between the first target sub-image and a second target sub-image among the plurality of fingerprint sub-images to obtain a target local fused image includes: Determine the target fusion weights corresponding to the first target sub-image and the second target sub-image respectively according to the first local image quality of the first target sub-image in the overlapping part and the second local image quality of the second target sub-image in the overlapping part, and if the local image quality is higher, the determined target fusion weight is higher; According to the target fusion weights corresponding to the first target sub-image and the second target sub-image respectively, perform weighted summation on the pixel values of the first target sub-image and the pixel values of the second target sub-image in the overlapping part, so as to locally fuse the overlapping part between the first target sub-image and the second target sub-image to obtain a target local fusion image.

8. The method according to claim 6, wherein The expanding the edge of the input image according to the fusion image to obtain a second expanded image includes: Determine a second image area including the first image area from the fusion image, and determine a third image area other than the first image area within the second image area; Correspondingly fill the third image area outside the edge of the input image to form the second expanded image.

9. The method according to claim 6, wherein If the first target sub-image is the fingerprint sub-image with the best image quality among the multiple fingerprint sub-images, then the second target sub-image is the fingerprint sub-image with the second-best image quality among the multiple fingerprint sub-images.

10. The method according to any one of claims 1-3, wherein, The determining an input image according to at least one fingerprint sub-image and expanding the edge of the input image according to the fusion image to obtain a second expanded image includes: Determine the first target sub-image as the input image, and expand the edge of the first target sub-image according to the fusion image to obtain the second expanded image.

11. The method according to claim 10, wherein The expanding the edge of the first target sub-image according to the fusion image to obtain the second expanded image includes: Determine a second image area including the first image area from the fusion image, and determine a third image area other than the first image area within the second image area; Correspondingly fill the third image area outside the edge of the first target sub-image to form the second expanded image.

12. The method according to any one of claims 1-3, wherein, The fusing the multiple fingerprint sub-images to obtain a fusion image includes: Determine the quality-optimal sub-image with the best image quality among the multiple fingerprint sub-images; For each fingerprint sub-image other than the quality-optimal sub-image among the multiple fingerprint sub-images, determine the similarity of the overlapping part between the fingerprint sub-image and the quality-optimal sub-image according to the reference offset data, and determine the real-time offset at the current moment between the fingerprint sub-image and the quality-optimal sub-image according to the similarity; According to the real-time offset at the current moment between each fingerprint sub-image and the quality-optimal sub-image, align the multiple fingerprint sub-images, and fuse the aligned multiple fingerprint sub-images to obtain the fusion image.

13. A fingerprint image processing device, comprising: A fusion module, configured to obtain a plurality of fingerprint sub-images respectively collected by a plurality of pixel units of an optical fingerprint sensor at the current moment, and perform fusion processing on the plurality of fingerprint sub-images to obtain a fused image; A first extension module, configured to determine a first image region corresponding to a position of a first target sub-image among the plurality of fingerprint sub-images from the fused image, and extend an edge of the first image region according to the fused image to obtain a first extended image; A second extension module, configured to determine an input image according to at least one fingerprint sub-image, and extend an edge of the input image according to the fused image to obtain a second extended image, where the at least one fingerprint sub-image includes the first target sub-image; A determination module, configured to determine one of the first extended image and the second extended image as an output image for fingerprint recognition.

14. An electronic device, comprising: A processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus; The memory is configured to store a computer program; The processor is configured to execute the method according to any one of claims 1-12 by running the computer program stored on the memory.

15. A computer storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the method according to any one of claims 1-12.

16. A computer program product, characterized in that, Including a computer program, and the computer program, when executed by a processor, implements the method according to any one of claims 1-12.

17. A fingerprint recognition device, applied to an electronic device having a display screen, and the fingerprint recognition device includes: An optical fingerprint sensor, configured to: image light signals in a plurality of different directions reflected by a finger above the display screen to collect a plurality of fingerprint sub-images; A processing unit, configured to execute the method according to any one of claims 1-12.

Images