Fingerprint image processing method and apparatus, electronic device, and storage medium

Abstract

Provided in the present application are a fingerprint image processing method and apparatus, an electronic device, and a storage medium. The fingerprint image processing method comprises: acquiring a plurality of fingerprint sub-images respectively collected by a plurality of pixel units of an optical fingerprint sensor at a current moment, and determining an optimum quality sub-image amongst the plurality of fingerprint sub-images; for each fingerprint sub-image other than the optimum quality sub-image amongst the plurality of fingerprint sub-images, on the basis of reference offset data, determining the similarity between an overlapping part of the fingerprint sub-image and the optimum quality sub-image and, on the basis of the similarity, determining a real-time offset of the current moment between the fingerprint sub-image and the optimum quality sub-image; on the basis of the real-time offset of the current moment between each fingerprint sub-image and the optimum quality sub-image, aligning the plurality of fingerprint sub-images, and fusing the plurality of aligned fingerprint sub-images to obtain a fused image for fingerprint recognition.

Description

Fingerprint image processing method, device, electronic device and storage medium Technical Field

[0001] The embodiments of the present application relate to the technical field of optical fingerprint image processing, and in particular to a fingerprint image processing method, device, electronic device, and storage medium. Background Art

[0002] The optical path design of some current under-screen ultra-thin optical fingerprint sensors adopts a multi-angle optical path. By designing multiple pixel units arranged in an array, multiple pixel units obtain fingerprint images at multiple different angles through multi-directional light channels, and then can collect effective fingerprint signals to the maximum extent.

[0003] In the above-mentioned optical path design, each direction of the multi-directional optical channel is fixed. Therefore, there is an approximately stable image offset between the multiple fingerprint images collected along the optical channel in each direction. In order to improve the recognition speed, the related art usually adopts a fixed image offset obtained by a special method to fuse the multiple fingerprint images collected by multiple pixel units.

[0004] However, actual research has found that because fingers are three-dimensional, factors such as the fit caused by the fingerprint state (e.g., dry or wet), slight deformation caused by pressure (e.g., light or heavy press), and the effective pressing area can easily cause the actual offset between fingerprint images to deviate from the fixed offset. This can lead to poor quality after fusion of multiple fingerprint images, thus reducing the success rate of fingerprint recognition. Therefore, a new technical solution is needed to at least partially improve this problem.

[0005] Summary of the Invention

[0006] In view of this, embodiments of the present application provide a fingerprint image processing method, apparatus, electronic device, and storage medium to at least partially solve the above-mentioned problems.

[0007] According to a first aspect of an embodiment of the present application, a fingerprint image processing method is provided, comprising:

[0008] Acquire multiple fingerprint sub-images respectively captured by multiple pixel units of the optical fingerprint sensor at a current moment, and determine the sub-image with the best quality among the multiple fingerprint sub-images;

[0009] For each fingerprint sub-image among the multiple fingerprint sub-images except the sub-image with the best quality, determining, based on the reference offset data, a similarity between the overlapping portion of the fingerprint sub-image and the sub-image with the best quality, and determining, based on the similarity, a real-time offset between the fingerprint sub-image and the sub-image with the best quality at a current moment;

[0010] The multiple fingerprint sub-images are aligned according to the real-time offset between each fingerprint sub-image and the sub-image with the best quality at the current moment, and the aligned multiple fingerprint sub-images are fused to obtain a fused image for fingerprint recognition.

[0011] In some optional embodiments, the reference offset data includes reference offsets between multiple reference fingerprint sub-images and reference quality optimal sub-images, wherein the reference quality optimal sub-image is acquired by a first pixel unit among multiple pixel units, and the multiple reference fingerprint sub-images are acquired by a plurality of second pixel units among the multiple pixel units, respectively, and the plurality of second pixel units are a plurality of pixel units other than the first pixel unit among the multiple pixel units.

[0012] In some optional embodiments, determining the similarity between the overlapping portion of the fingerprint sub-image and the optimal quality sub-image based on the reference offset data, and determining the real-time offset between the fingerprint sub-image and the optimal quality sub-image at the current moment based on the similarity, includes: calculating the current reference offset between the fingerprint sub-image and the optimal quality sub-image based on the reference offsets between the multiple reference fingerprint sub-images and the reference optimal quality sub-images, and determining the real-time offset search range at the current moment based on the current reference offset; determining the similarity between the overlapping portion of the fingerprint sub-image and the optimal quality sub-image corresponding to each offset within the real-time offset search range at the current moment, and using the offset corresponding to the largest similarity as the real-time offset between the fingerprint sub-image and the optimal quality sub-image at the current moment.

[0013] In some optional embodiments, determining the real-time offset search range at the current moment based on the current reference offset includes: determining the real-time offset search range at the current moment based on the current reference offset and a preset offset margin.

[0014] In some optional embodiments, the method further includes: updating the reference offset data according to the real-time offset between each fingerprint sub-image and the sub-image with the best quality at the current moment.

[0015] In some optional embodiments, updating the reference offset data based on the real-time offset between each fingerprint sub-image and the sub-image with the best quality at the current moment includes: determining, based on the real-time offset between each fingerprint sub-image and the sub-image with the best quality at the current moment, reference real-time offsets between the fingerprint sub-images captured by the multiple second pixel units and the fingerprint sub-image captured by the first pixel unit; determining, for each reference real-time offset, a reference offset between the reference fingerprint sub-image captured by the corresponding second pixel unit and the reference sub-image with the best quality, and performing a weighted summation of the reference real-time offset and the reference offset according to an update weight; and updating the reference offset data based on the weighted summation results corresponding to each reference real-time offset.

[0016] In some optional embodiments, the method further includes: adjusting the update weight based on at least one of the image quality of the fingerprint sub-image corresponding to the reference real-time offset and the similarity between the overlapping part of the fingerprint sub-image corresponding to the reference real-time offset and the sub-image with the best quality.

[0017] In some optional embodiments, the fusing of the aligned multiple fingerprint sub-images to obtain a fused image for fingerprint identification includes: for any overlapping area of ​​the aligned multiple fingerprint sub-images, based on the local image quality of each fingerprint sub-image in the overlapping area and the similarity between each fingerprint sub-image in the overlapping area and the overlapping part of the sub-image with the best quality, fusing the local parts of each fingerprint sub-image in the overlapping area to obtain a local fusion result; and splicing the local fusion results corresponding to each overlapping area of ​​the aligned multiple fingerprint sub-images to obtain the fused image for fingerprint identification.

[0018] In some optional embodiments, the fusing of parts of each fingerprint sub-image in the overlapping area according to the local image quality of each fingerprint sub-image in the overlapping area and the similarity between each fingerprint sub-image in the overlapping area and the sub-image with the best quality includes: determining a local image quality score of each fingerprint sub-image in the overlapping area; determining a fusion weight corresponding to each fingerprint sub-image in the overlapping area according to the obtained local image quality score and the similarity between each fingerprint sub-image in the overlapping area and the overlapping part of the sub-image with the best quality; and performing weighted summation of the local pixel values ​​of each fingerprint sub-image according to the obtained fusion weights to fuse the parts of each fingerprint sub-image in the overlapping area.

[0019] In some optional embodiments, if the local image quality score corresponding to the fingerprint sub-image is higher and the similarity is greater, the fusion weight corresponding to the fingerprint sub-image is determined to be higher.

[0020] According to a second aspect of an embodiment of the present application, there is provided a fingerprint image processing device, comprising:

[0021] an acquisition module, configured to acquire a plurality of fingerprint sub-images respectively captured by a plurality of pixel units of the optical fingerprint sensor at a current moment, and determine a sub-image with the best quality among the plurality of fingerprint sub-images;

[0022] a determination module configured to determine, for each fingerprint sub-image among the plurality of fingerprint sub-images except the sub-image with the best quality, a similarity between the overlapping portion of the fingerprint sub-image and the sub-image with the best quality based on the reference offset data, and to determine, based on the similarity, a real-time offset between the fingerprint sub-image and the sub-image with the best quality at a current moment;

[0023] The fusion module is used to align the multiple fingerprint sub-images according to the real-time offset between each fingerprint sub-image and the sub-image with the best quality at the current moment, and fuse the aligned multiple fingerprint sub-images to obtain a fused image for fingerprint recognition.

[0024] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store a computer program; and the processor is used to execute the method provided in the first aspect by running the computer program stored on the memory.

[0025] According to a fourth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method provided in the first aspect is implemented.

[0026] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising a computer program, which implements the method provided in the first aspect when executed by a processor.

[0027] According to a sixth aspect of an embodiment of the present application, a fingerprint recognition device is provided, which is applied to an electronic device with a display screen. The fingerprint recognition device includes: an optical fingerprint sensor, which is used to: image light signals in multiple directions reflected by a finger above the display screen to obtain multiple fingerprint sub-images; and a processing unit, which is used to execute the method provided in the first aspect.

[0028] According to the fingerprint image processing solution provided in the embodiment of the present application, multiple fingerprint sub-images respectively collected by multiple pixel units can be determined, and the sub-image with the best quality among the multiple fingerprint sub-images can be determined. For each fingerprint sub-image except the sub-image with the best quality, the similarity of the overlapping part of the fingerprint sub-image and the sub-image with the best quality can be determined according to the reference offset data. Based on the similarity, the real-time offset between the fingerprint sub-image and the sub-image with the best quality at the current moment can be determined. Then, based on each real-time offset, the multiple fingerprint sub-images can be aligned, and the aligned multiple fingerprint sub-images can be fused to obtain a fused image for fingerprint recognition. Therefore, on the one hand, the present solution no longer uses a fixed image offset to achieve fingerprint image fusion, but can more accurately determine and realize alignment and image fusion of multiple fingerprint sub-images based on the real-time offset of each fingerprint sub-image. Therefore, it can more effectively adapt to the deviations caused by different fingerprint states and pressing states of fingers on the three-dimensional surface, which is beneficial to improving the quality of multiple fingerprint sub-images after fusion, thereby effectively improving the recognition success rate of fingerprint recognition; on the other hand, since the present solution uses the fused image obtained by fusing multiple fingerprint sub-images aligned according to each real-time offset for fingerprint recognition, it can aggregate the effective information of multiple fingerprint sub-images, increase the recognition field of view of fingerprint recognition, and also help improve the recognition success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0030] FIG1 shows a flow chart of an optional fingerprint image processing method of the present application.

[0031] FIG2A is a schematic diagram showing a multi-angle optical path of a pixel unit of an optional optical fingerprint sensor of the present application.

[0032] FIG2B is a schematic diagram showing the real-time offset calculation between the fingerprint sub-image and the sub-image with the best quality.

[0033] FIG3A shows a flowchart of an optional sub-step of “determining the similarity between the overlapping portion of the fingerprint sub-image and the optimal quality sub-image based on the reference offset data, and determining the real-time offset between the fingerprint sub-image and the optimal quality sub-image at the current moment based on the similarity” in step S104.

[0034] FIG3B shows an optional sub-step flow chart of a method for updating reference offset data.

[0035] FIG3C shows a flowchart of an optional sub-step of “fusing the aligned multiple fingerprint sub-images to obtain a fused image for fingerprint identification” in step S106 .

[0036] FIG3D shows a flowchart of an optional sub-step of “merging the parts of each fingerprint sub-image in the overlapping area according to the local image quality of each fingerprint sub-image in the overlapping area and the similarity between each fingerprint sub-image in the overlapping area and the overlapping part of the sub-image with the best quality” in sub-step S1061.

[0037] FIG4 shows a schematic diagram of fusing parts of two images.

[0038] FIG5 shows a schematic diagram of an exemplary fingerprint image processing device of the present application.

[0039] FIG6 shows a schematic diagram of an exemplary electronic device of the present application.

[0040] FIG7 shows a schematic diagram of an exemplary fingerprint recognition device of the present application. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and in detail described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0042] The optical path design of ultra-thin under-screen optical fingerprint sensors requires consideration of two extreme signal acquisition conditions. The first is when the fingerprint is well adhered to the screen, such as when fingerprint recognition is performed at room temperature. In this case, the smaller the signal acquisition optical path's light collection angle, the more likely it is to receive a single fingerprint ridge signal (or fingerprint valley signal), resulting in a higher signal strength and improved recognition. The other is when the fingerprint is poorly adhered to the screen, such as when fingerprint recognition is performed at low temperatures or when a light press is performed. In these cases, it is desirable to increase the signal acquisition optical path's light collection angle. Because a fingerprint is a three-dimensional surface, at high angles of incident light, valley-reflected light is more easily blocked by the ridge sidewalls, increasing the difference between the ridge and valley signals and boosting the signal. Actual fingerprint press signals fall somewhere between these two extremes. To address these considerations, the optical path designs of some current ultra-thin under-screen optical fingerprint sensors typically employ a multi-angle optical path. By designing an array of multiple pixel units, these units capture fingerprint images from different angles through multi-directional light channels, thereby maximizing the number of valid fingerprint signals captured.

[0043] Since multiple fingerprint images can be collected with one press, and different fingerprint images represent signals at different angles, in order to balance recognition performance and speed, there are currently multiple solutions to utilize these fingerprint images. The first is multi-fingerprint image cyclic recognition. A problem with this solution is that multiple fingerprint images are used to compare the registered fingerprint template for fingerprint recognition, which will lead to a decrease in recognition speed. Another solution is to fuse different fingerprint images, obtain one image, and only recognize one image. Taking into account that in the above-mentioned optical path design, the directions of the multi-directional optical channel are fixed, so there is an approximately stable image offset between the multiple fingerprint images collected along the optical channel in each direction. In order to improve the recognition speed, the related art usually adopts a fixed image offset obtained by a special method to fuse the multiple fingerprint images collected by multiple pixel units.

[0044] However, in actual research, it was found that since fingers are three-dimensional surfaces, the actual offset between fingerprint images and the fixed offset can easily deviate due to factors such as the fingerprint state (such as dry or wet fingers), the slight deformation caused by the pressing state (such as light pressing or heavy pressing), and the effective pressing area. This can then lead to deterioration in quality after the fusion of multiple fingerprint images, thereby reducing the success rate of fingerprint recognition.

[0045] In view of this, a fingerprint image processing solution is proposed in this application. The specific implementation of the embodiment of this application is further described below with reference to the accompanying drawings.

[0046] FIG1 shows a flowchart of an optional fingerprint image processing method of the present application. As shown in FIG1 , the fingerprint image processing method includes steps S102, S104 and S106, specifically:

[0047] S102: Acquire multiple fingerprint sub-images respectively collected by multiple pixel units of the optical fingerprint sensor at the current moment, and determine the sub-image with the best quality among the multiple fingerprint sub-images.

[0048] This application does not specifically limit the specific results of the optical fingerprint sensor. For example, the optical fingerprint sensor can be an under-screen optical fingerprint sensor, which can be used to implement the under-screen optical fingerprint recognition function of an electronic device. The optical fingerprint sensor can image light signals in multiple directions reflected by a finger above the display screen of the electronic device to collect multiple fingerprint sub-images. It should be noted that the fingerprint image detection method of the present application can be executed by any processing unit. For example, the method can be executed by a processing unit (for example, including but not limited to one or more chips capable of data processing, including but not limited to a CPU (Central Processing Unit), an MCU (Microcontroller Unit), a GPU (Graphic Processing Unit), an FPGA (Field Programmable Gate Array), etc.) of an electronic device to which the optical fingerprint sensor is applied (for example, including but not limited to a mobile terminal, a computer, a fingerprint lock, etc.), which is conducive to improving the processing effect.

[0049] The optical fingerprint sensor in this application utilizes a multi-angle optical path design, comprising multiple pixel units. Each pixel unit may include one or more optical pixels, and multiple pixel units may include the same number of optical pixels. Figure 2A illustrates a schematic diagram of the multi-angle optical path of a pixel unit of an optional optical fingerprint sensor. As shown in Figure 2A, the schematic diagram illustrates that the optical fingerprint sensor includes multiple pixel units (as shown in the simplified example of Figure 2A, a total of nine pixel units, but it should be understood that the number of pixel units in practice may be greater). Microlenses are disposed on the multiple pixel units. When a finger is recorded through the optical fingerprint sensor, the reflected light can pass through the different optical paths of the microlens units and be received by each pixel unit at various light collection angles. Each pixel unit can generate a fingerprint image based on the received light, thereby enabling each pixel unit to capture the fingerprint image. The light received by each pixel unit at various angles is, in other words, a plurality of light signals from different directions. It should be understood that the multiple fingerprint sub-images described in step S102 are also fingerprint images captured by multiple pixel units. Due to the multi-angle optical path, there is a certain image offset between each of the multiple fingerprint sub-images.

[0050] To facilitate the description of this embodiment, 9 fingerprint sub-images collected by 9 pixel units are taken as an example.

[0051] The optimal quality sub-image in this application is the fingerprint sub-image with the best image quality among multiple fingerprint sub-images. The optimal quality sub-image can be determined according to any suitable method, which is not limited here. For example, any suitable algorithm can be used to calculate one or more of the indicators such as Harris Response (HR), Normalized Gradient (SR), Peak Signal-to-Noise Ratio (PSNR), Normalized Stoichiometric Ratio (NSR, which can characterize the ratio between the peak-to-peak value and the local mean) of each fingerprint sub-image, and the image quality of the multiple fingerprint sub-images is determined based on the calculation results to determine the optimal quality sub-image with the best image quality. Optionally, by calculating one or more of the above indicators, the image quality of each fingerprint sub-image can be scored, and the fingerprint sub-image with the highest image quality score can be determined as the optimal quality sub-image.

[0052] The above metrics can be used to determine image quality because they essentially measure the clarity of banding. Within a small region of a fingerprint subimage, a fingerprint is essentially a periodic light and dark band signal. The clearer the banding, the larger the normalized gradient (SR). Similarly, the difference between the neighborhood maximum and minimum values ​​fluctuates more relative to the maximum or minimum value (corresponding to the peak signal-to-noise ratio (PSNR)). Similarly, the Harris response (HR) eigenvalue exhibits edge characteristics, and the ratio of the peak signal to the local mean (corresponding to the normalized signal strength (NSR)) is also larger (the local mean is linearly related to light intensity and exposure time. Under the same conditions, a good fingerprint will have a larger signal strength, i.e., a higher normalized signal strength (NSR)). Therefore, determining the image quality of a fingerprint subimage using one or more of the above metrics can ensure the reliability of the optimal subimage, facilitating subsequent data processing.

[0053] In addition, in step S102 of the present application, multiple fingerprint sub-images at the current moment are obtained, and the sub-image with the best quality is determined, which is conducive to real-time and accurate fingerprint recognition.

[0054] S104: For each fingerprint sub-image other than the sub-image with the best quality among the multiple fingerprint sub-images, the similarity between the overlapping portion of the fingerprint sub-image and the sub-image with the best quality is determined based on the reference offset data. Based on the similarity, the current real-time offset between the fingerprint sub-image and the sub-image with the best quality is determined. (Please continue to refer to the flowchart in FIG1 to understand this step S104.)

[0055] There will be a certain offset between multiple fingerprint sub-images (also known as image offset). The offset between images is measured in pixels. The offset can include a lateral offset along the horizontal direction of the image (for example, called the x direction, which can also be understood as the horizontal direction) and a longitudinal offset along the vertical direction of the image (for example, called the y direction, which can also be understood as the vertical direction). For ease of explanation, the horizontal rightward direction of the image is regarded as the positive direction of the x direction, the horizontal leftward direction is regarded as the negative direction of the x direction, the vertical downward direction is regarded as the positive direction of the y direction, and the vertical upward direction is regarded as the negative direction of the y direction. In the following, each type of offset can be recorded in the form of (x, y), where x represents the lateral offset in the offset and y represents the vertical offset in the offset. A positive x value represents a lateral offset to the right along the horizontal direction of the image, a negative x value represents a lateral offset to the left along the horizontal direction of the image, a positive y value represents a longitudinal offset downward along the vertical direction of the image, and a negative y value represents a longitudinal offset downward along the vertical direction of the image.

[0056] For example, FIG2B shows a schematic diagram of real-time offset calculation between a fingerprint sub-image and an optimal quality sub-image. FIG2B also illustrates the x-direction (horizontal) and y-direction (vertical) of an image. It should be understood that the fingerprint sub-image and optimal quality sub-image, as well as the overlap between them, shown in FIG2B are merely examples and do not constitute any limitation on the present application.

[0057] Optionally, the reference offset data in the present application includes reference offsets between a plurality of reference fingerprint sub-images and a reference optimal quality sub-image, wherein the reference optimal quality sub-image is acquired by a first pixel unit among the plurality of pixel units, and the plurality of reference fingerprint sub-images are acquired by a plurality of second pixel units among the plurality of pixel units, wherein the plurality of second pixel units are a plurality of pixel units other than the first pixel unit among the plurality of pixel units. In the present application, such reference offset data can be used to facilitate data processing to determine the similarity between the overlapping portions of the fingerprint sub-image and the optimal quality sub-image.

[0058] Before executing the fingerprint image processing method of the present application for the first time, reference offset data can be pre-stored (which can be understood as initialized reference deviation data) and can be retrieved when needed. Optionally, the reference offset data can be initialized by calibration during the mass production phase using striped weights or weights with special patterns (striped weights or weights with special patterns can be used to simulate fingers).

[0059] For example, the reference deviation data determination process in this application can be understood by referring to the following process: Taking the 9 pixel units of the optical fingerprint sensor in Figure 2A as an example, in the mass production stage, 9 images of the striped weight (for example, denoted as image 1 to image 9) can be collected through 9 pixel units (for example, denoted as pixel unit 1 to pixel unit 9), and then the image with the best image quality among images 1 to image 9 is determined. The image with the best image quality is used as the reference sub-image with the best quality (for example, taking image 1 as an example), and the other 8 images can be used as reference fingerprint sub-images (i.e., image 2 to image 9), with pixel unit 1 being the first pixel unit and pixel units 2 to pixel units 9 being the second pixel units respectively; thereafter, the image offsets of images 2 to image 9 relative to image 1 can be directly measured to obtain 8 image offsets, which can then be used as 8 reference offsets. The 8 reference offsets can be stored as reference offset data and can be directly retrieved when needed. It should be understood that this process is only an example for ease of understanding and does not constitute any limitation to this application.

[0060] The present application does not limit the method for calculating the similarity. In some optional embodiments, referring to the flowchart shown in FIG3A , step S104 of "determining the similarity of the overlapping portion of the fingerprint sub-image and the sub-image with the best quality based on the reference offset data, and determining the real-time offset between the fingerprint sub-image and the sub-image with the best quality at the current moment based on the similarity" includes sub-steps S1041 and S1042, specifically:

[0061] S1041: Calculate the current reference offset between the fingerprint sub-image and the sub-image with the best quality based on the reference offsets between the multiple reference fingerprint sub-images and the reference sub-image with the best quality, and determine the real-time offset search range at the current moment based on the current reference offset.

[0062] Since the multiple reference fingerprint sub-images and the reference optimal quality sub-image are also collected by the same multiple pixel units as the multiple fingerprint sub-images, when executing the fingerprint image processing method of this scheme (for example, when fingerprint recognition is performed in practice), if the optimal quality sub-image is not collected by the first pixel unit, the current reference offset between each fingerprint sub-image and the optimal quality sub-image can be converted by using the reference offsets between the multiple reference fingerprint sub-images and the reference optimal quality sub-image.

[0063] For example, let's still take the example of 9 pixel units capturing 9 images respectively, pixel unit 1 being the first pixel unit capturing the reference sub-image with the best quality, and pixel units 2 to 9 being the second pixel units capturing the reference fingerprint sub-image. For example, let's take the example of the reference offset between the reference fingerprint sub-image captured by pixel unit 2 and the reference sub-image with the best quality captured by pixel unit 1 being (x1, y1). If the sub-image with the best quality determined in step S102 is the fingerprint sub-image captured by pixel unit 2, then the current reference offset between the fingerprint sub-image captured by pixel unit 1 and the sub-image with the best quality captured by pixel unit 2 can be converted to (-x1, -y1); similarly, the current reference offset between the fingerprint sub-image captured by other pixel units and the sub-image with the best quality captured by pixel unit 2 can also be converted according to each reference offset. It should be understood that this is merely an example for ease of understanding and is not intended to limit the present application.

[0064] After calculating the current reference offset, the real-time offset search range at the current moment can be determined to accurately determine the real-time offset corresponding to the fingerprint sub-image within the real-time offset search range.

[0065] In some optional embodiments, "determining the real-time offset search range at the current moment according to the current reference offset" in sub-step S1041 may include: determining the real-time offset search range at the current moment according to the current reference offset and a preset offset margin.

[0066] Based on this, the above embodiments of the present application determine the real-time offset search range by using the current reference offset and the preset offset margin to ensure that a suitable real-time offset can be searched within the real-time offset search range.

[0067] For example, the preset offset margin includes a positive offset margin and a negative offset margin along the horizontal direction of the image, and a positive offset margin and a negative offset margin along the vertical direction of the image. Optionally, referring to the example shown in FIG2B , the absolute values ​​of the positive offset margin and the negative offset margin along the horizontal direction of the image are equal, both being wx; the absolute values ​​of the positive offset margin and the negative offset margin along the vertical direction of the image are equal, both being wy. For example, the current reference offset can be recorded as (sx, sy), where sx represents the horizontal offset in the current reference offset, and sy represents the vertical offset in the current reference offset. Combined with FIG2B , the determined real-time offset search range can be expressed as: sx+i, sy+j; wherein, i=[-wx, wx], j=[-wy, wy]. Of course, the absolute values ​​of the positive offset margin and the negative offset margin along the horizontal / vertical direction are equal here, which is a special example. In other optional embodiments, they may also be unequal. For ease of explanation, the following example may be taken as an example of equality.

[0068] The range corresponding to the preset offset margin within the real-time offset search range can be understood as the neighborhood of the range corresponding to the current reference offset within the real-time offset search range. Therefore, this optional implementation of the present application can be understood as a "current reference offset + neighborhood search" approach to ensure that a suitable real-time offset is found within the real-time offset search range.

[0069] S1042: Determine the similarity of the overlapping portion of the fingerprint sub-image and the optimal quality sub-image corresponding to each offset within the real-time offset search range at the current moment, and use the offset corresponding to the largest similarity as the real-time offset between the fingerprint sub-image and the optimal quality sub-image at the current moment.

[0070] Within the real-time offset search range, there are multiple offsets (each offset includes a horizontal offset and a vertical offset). For each offset, the corresponding overlapping portion of the fingerprint sub-image and the optimal quality sub-image may be different. For each offset, the similarity between the corresponding overlapping portion of the fingerprint sub-image and the optimal quality sub-image is calculated, and the offset corresponding to the maximum similarity is used as the real-time offset at the current moment.

[0071] Alternatively, the real-time offset between the fingerprint sub-image and the optimal quality sub-image at the current moment can be calculated by the following formula: {cx,cy}=arg_max{S(pic_A,pic_B,sx+i,sy+j),i=[-wx,wx],j=[-wy,wy]}

[0072] In the above formula, cx represents the lateral offset in the real-time offset at the current moment, cy represents the longitudinal offset in the real-time offset at the current moment; pic_A represents the fingerprint sub-image, and pic_B represents the optimal quality sub-image; S is used to represent the similarity of the overlapping parts of pic_A and pic_B under offset; sx is used to represent the lateral offset in the current reference offset, and sy is used to represent the longitudinal offset in the current reference offset; -wx and wx are used to represent the negative offset margin and positive offset margin along the horizontal direction of the image in the preset offset margin, respectively, and -wy and wy are used to represent the negative offset margin and positive offset margin along the vertical direction of the image in the preset offset margin, respectively. Referring to the above, it can be seen that "sx+i,sy+j, i=[-wx,wx], j=[-wy,wy]" can also represent the real-time offset search range (which can be understood in conjunction with Figure 2B). The above formula can be understood as follows: the real-time offset at the current moment needs to be traversed and searched within the above real-time offset search range to select the offset corresponding to the maximum similarity as the actual offset at the current moment in the final output.

[0073] In the present application, the aforementioned similarity can be used to describe the correlation between the overlapping portions of the fingerprint sub-image and the optimal quality sub-image, and can be implemented using any suitable indicator. Optionally, the similarity can be calculated using at least one indicator selected from the group consisting of Normalized Cross Correlation (NCC), SSIM (structural similarity), and MSE (mean-square error). The greater the NCC and SSIM values, the greater the similarity, while the smaller the MSE value, the greater the similarity.

[0074] Based on this, the present application can accurately and reliably determine the real-time offset between the fingerprint sub-image and the sub-image with the best quality at the current moment through the optional method of the above-mentioned sub-steps S1041 to S1042, thereby facilitating the subsequent steps to achieve alignment and image fusion of multiple fingerprint sub-images through reliable real-time offset, thereby being able to more effectively adapt to the deviations caused by different fingerprint states and pressing states of fingers on the three-dimensional surface, and is more conducive to improving the image quality after the fusion of multiple fingerprint sub-images, thereby effectively improving the recognition success rate of fingerprint recognition.

[0075] In some optional embodiments, the fingerprint image processing method of the present application further includes: updating the reference offset data according to the real-time offset between each fingerprint sub-image and the sub-image with the best quality at the current moment.

[0076] The offset between the fingerprint sub-images detected by each pixel unit will change slightly due to the temperature of the optical fingerprint sensor, and will also change due to the offset of the optical fingerprint sensor module in extreme cases, such as an abnormal drop (for example, a fall) of the electronic device equipped with the optical fingerprint sensor. Therefore, by updating the reference offset data in this application, the updated reference offset data can better adapt to the changes of the optical fingerprint sensor in different environments and different states, thereby making the subsequent real-time offset more accurate and reliable, and facilitating further data processing.

[0077] In some optional embodiments, referring to the flowchart shown in FIG. 3B , the method for updating the reference offset data includes the following sub-steps S202 , S204 , and S206 . Specifically:

[0078] S202: Determine reference real-time offsets between fingerprint sub-images captured by multiple second pixel units and fingerprint sub-images captured by the first pixel unit, respectively, among the multiple fingerprint sub-images based on the current real-time offsets between each fingerprint sub-image and the sub-image with the best quality.

[0079] As described above, the reference offset data may include reference offsets between multiple reference fingerprint sub-images and the reference optimal quality sub-image. In this application, the reference offset data is updated, that is, each reference offset is updated. Therefore, it is necessary to convert it into a real-time offset between the corresponding reference fingerprint sub-image and the reference optimal quality sub-image (that is, the reference real-time offset) for calculation.

[0080] Since the reference optimal quality sub-image is collected by the first pixel unit and the reference fingerprint sub-image is collected by the second pixel unit, each reference real-time offset can be converted into each real-time offset.

[0081] For example, let's take the example of nine pixel units capturing nine images, pixel unit 1 being the first pixel unit capturing the reference sub-image with the best quality, and pixel units 2 to 9 being the second pixel units capturing the reference fingerprint sub-images. If the sub-image with the best quality determined in step S102 is the fingerprint sub-image captured by pixel unit 2, then if the real-time deviation between the fingerprint sub-image captured by pixel unit 1 (i.e., the first pixel unit) and the sub-image with the best quality captured by pixel unit 2 is (x2, y2), then the reference real-time offset between the fingerprint sub-image captured by pixel unit 2 and the fingerprint sub-image captured by pixel unit 1 (i.e., the first pixel unit) can be converted to (-x2, -y2). Similarly, the reference real-time offsets between the fingerprint sub-images captured by the other multiple second pixel units (pixel units 3 to 9) and the fingerprint sub-image captured by the first pixel unit (pixel unit 1) can also be calculated based on the respective real-time offsets. It should be understood that this is merely an example for ease of understanding and is not intended to limit the present application.

[0082] S204: For each reference real-time offset, determine a reference offset between the reference fingerprint sub-image captured by the corresponding second pixel unit and the reference optimal quality sub-image, and perform a weighted sum of the reference real-time offset and the reference offset according to the updated weight.

[0083] For example, let's continue with the example of nine pixel units capturing nine images, with pixel unit 1 being the first pixel unit capturing the reference sub-image of optimal quality, and pixel units 2-9 being the second pixel units capturing the reference fingerprint sub-image. For the reference real-time offset between the fingerprint sub-image captured by any pixel unit from 2-8 and the fingerprint sub-image captured by pixel unit 1, for example, taking pixel unit 2 as an example, the reference offset between the reference fingerprint sub-image captured by pixel unit 2 and the reference sub-image of optimal quality is determined. The reference real-time offset between pixel unit 2 and pixel unit 1, and the reference offset between the reference fingerprint sub-image captured by pixel unit 2 and the reference sub-image of optimal quality are weightedly summed according to the updated weights to obtain the weighted summation result corresponding to pixel unit 2. Similarly, for pixel units 3-9, the weighted summation can be performed in accordance with the calculation method for pixel unit 2, obtaining the corresponding weighted summation results. That is, for the eight reference real-time offsets corresponding to pixel units 2-9, eight weighted summation results can be obtained. It should be understood that this is merely an example for ease of understanding and is not intended to limit the present application.

[0084] Optionally, in sub-step S204, the reference real-time offset and the reference offset may be weighted and summed according to the following two formulas to obtain the corresponding weighted summation result: sx(t)=(1-w)*sx(t-1)+w*Cx sy(t)=(1-w)*sy(t-1)+w*Cy

[0085] Where w and 1-w represent the update weights, with w∈(0,1). Cx represents the lateral offset of the reference real-time offset. sx(t) and sx(t-1) represent the lateral offsets of the reference offsets at the previous and next moments, respectively. In the above embodiment, sx(t) can be understood as the lateral offset of the updated reference offset, and sx(t-1) can be understood as the lateral offset of the reference offset before the update.

[0086] Where Cy represents the longitudinal offset in the reference real-time offset, and sy(t) and sy(t-1) represent the longitudinal offsets in the reference offsets at the preceding and subsequent moments, respectively. In the above embodiment, sy(t) can be understood as the longitudinal offset in the updated reference offset, and sy(t-1) can be understood as the longitudinal offset in the reference offset before the update.

[0087] The horizontal offset and the vertical offset in the updated reference offset are calculated by the above two formulas, that is, an updated reference offset between a reference fingerprint sub-image and a reference optimal quality sub-image is obtained, that is, a weighted summation result is obtained.

[0088] It should be understood that the updated reference offsets obtained by weighted summation with different update weights may be different. Optionally, the update weights w and 1-w may be preset values ​​(which may be understood as initialized update weights) before the fingerprint image processing method of this solution is executed for the first time.

[0089] S206: Update the reference offset data according to the weighted summation results corresponding to the respective reference real-time offsets.

[0090] Each weighted summation result is an updated reference offset between a reference fingerprint sub-image and a reference sub-image with the best quality. According to each updated reference offset, each updated reference offset is used as updated reference offset data, thereby achieving the update of the reference offset data.

[0091] Based on this, the present application includes the optional implementation of the above-mentioned sub-steps S202 to S206, which can effectively and reliably update the reference offset data. The updated reference offset data can better adapt to the changes of the optical fingerprint sensor in different environments and different states, thereby making the subsequently determined real-time offset more accurate and reliable, so as to facilitate further data processing.

[0092] In some optional embodiments, the fingerprint image processing method in the present application further includes: adjusting the update weight based on at least one of the image quality of the fingerprint sub-image corresponding to the reference real-time offset and the similarity between the overlapping part of the fingerprint sub-image corresponding to the reference real-time offset and the sub-image with the best quality.

[0093] In the present application, the update weight is adjusted based on at least one of the image quality of the fingerprint sub-image corresponding to the reference real-time offset and the similarity between the overlapping part of the fingerprint sub-image corresponding to the reference real-time offset and the sub-image with the best quality, so as to better update the reference offset data, so that the reference offset data can better adapt to the changes of the optical fingerprint sensor in different environments and different states, and make the subsequently determined real-time offset more accurate and reliable, so as to facilitate further data processing.

[0094] The image quality of a fingerprint sub-image can be determined by calculating one or more of the following metrics: Harris response (HR), normalized gradient (SR), peak signal-to-noise ratio (PSNR), and normalized signal size (NSR), and determining the image quality of the fingerprint sub-image based on the calculation results. Similarity can be calculated using at least one metric: Normalized Cross Correlation (NCC), structural similarity (SSIM), and mean-square error (MSE). These related details have been previously described and will not be repeated here.

[0095] Optionally, in some optional embodiments, the better the image quality of the fingerprint sub-image corresponding to the reference real-time offset, the higher the adjusted update weight, which can make the calculated update weight after adjustment more reliable.

[0096] Alternatively, in some other optional embodiments, the higher the similarity between the overlapping portion of the fingerprint sub-image corresponding to the reference real-time offset and the sub-image with the best quality, the higher the adjusted update weight. This can also make the calculated update weight after adjustment more reliable.

[0097] Alternatively, in some further embodiments, when the similarity between the overlapping portion of the fingerprint sub-image corresponding to the reference real-time offset and the optimal quality sub-image exceeds a certain similarity threshold, the update weight may be adjusted based on the image quality of the fingerprint sub-image corresponding to the reference real-time offset. Specifically, the better the image quality of the fingerprint sub-image corresponding to the reference real-time offset, the higher the adjusted update weight. This can also improve the reliability of the calculated update weight after adjustment.

[0098] Optionally, in the present application, when adjusting the update weight, the update weight may be adjusted within a preset update weight range. The update weight range may be preset as needed. For example, the update weight range of the update weight w may be, for example, 0.2-0.7, 0.1-0.6, etc. It is understood that the corresponding 1-w may vary with changes in w.

[0099] S106: Align the multiple fingerprint sub-images based on the current real-time offset between each fingerprint sub-image and the sub-image with the best quality, and fuse the aligned multiple fingerprint sub-images to obtain a fused image for fingerprint recognition. (Please continue to understand this step S106 in conjunction with the flowchart in Figure 1)

[0100] Optionally, each fingerprint sub-image other than the sub-image with the best quality can be aligned with its corresponding real-time offset (including horizontal and vertical offsets) at the current moment to align the overlapping portion between the fingerprint sub-image and the sub-image with the best quality. After all fingerprint sub-images are aligned with the sub-image with the best quality, multiple fingerprint sub-images can be aligned. The aligned multiple fingerprint sub-images can then be fused to obtain a fused image for fingerprint recognition, and the fused image can be used for real-time fingerprint recognition.

[0101] In this application, due to the offset between each fingerprint sub-image, the area of ​​the fused image is larger than that of a single fingerprint sub-image, that is, the field of view of the fused image is larger than that of a single fingerprint sub-image. Therefore, by performing fingerprint recognition through fused images, the effective information of multiple fingerprint sub-images can be aggregated, thereby increasing the recognition field of view of fingerprint recognition.

[0102] In some optional embodiments, referring to the flowchart shown in FIG3C , step S106 of “fusing the aligned multiple fingerprint sub-images to obtain a fused image for fingerprint recognition” includes sub-steps S1061 and S1062. Specifically:

[0103] S1061: For any overlapping area of ​​the aligned multiple fingerprint sub-images, based on the local image quality of each fingerprint sub-image in the overlapping area and the similarity between each fingerprint sub-image in the overlapping area and the overlapping part of the sub-image with the best quality, the local areas of each fingerprint sub-image in the overlapping area are fused to obtain a local fusion result.

[0104] In the present application, the aligned multiple fingerprint sub-images may have multiple overlapping areas, and each overlapping area may include local content of the multiple fingerprint sub-images.

[0105] In sub-step S1061 of the present application, the local image quality of each fingerprint sub-image in the overlapping area and the similarity between each fingerprint sub-image in the overlapping area and the overlapping part of the best quality sub-image are used to fuse the parts of each fingerprint sub-image in the overlapping area. This is because: the local image quality of the fingerprint sub-image can describe the clarity of the local fingerprint sub-image. The clearer the local fingerprint sub-image, the better the local image quality (it should be understood that the local image quality score below is also higher); and the similarity between the overlapping part of the fingerprint sub-image and the best quality sub-image can describe the consistency of the texture between the fingerprint sub-image and the best quality sub-image. After multiple fingerprint sub-images are aligned, the more consistent the texture at the same position, the higher the similarity. Therefore, in the present application, when fusing the parts of each fingerprint sub-image in the overlapping area, the above multiple dimensions are comprehensively considered. First, the above-mentioned local image quality is taken into consideration, so that the output local fusion result contains the local clear area of ​​each fingerprint sub-image as much as possible, avoiding the local blurred areas of each fingerprint sub-image from entering the fusion too much, resulting in blurred output local fusion result, and thus avoiding the blurring of the subsequent fusion image; secondly, the above-mentioned similarity is taken into consideration to suppress the areas where the texture of each fingerprint sub-image is inconsistent with the texture of the sub-image with the best quality, so as to avoid the distortion and blurring of the image texture of the output local fusion result, and thus avoid the distortion and blurring of the image texture of the subsequent fusion image.

[0106] In some optional embodiments, referring to the flowchart shown in FIG3D , step S1061 of “merging the parts of the fingerprint sub-images within the overlapping area according to the local image quality of the fingerprint sub-images within the overlapping area and the similarity between the fingerprint sub-images within the overlapping area and the overlapping parts of the sub-image with the best quality” includes sub-steps S1061A, S1061B, and S1061C. Specifically:

[0107] S1061A: Determine the local image quality score of each fingerprint sub-image in the overlapping area.

[0108] Optionally, for any overlapping region, an image quality score can be calculated for each local portion of each fingerprint sub-image using one or more of the following indicators: Harris response (HR), normalized gradient (SR), peak signal-to-noise ratio (PSNR), normalized signal strength (NSR), etc., to obtain a local image quality score for each fingerprint sub-image in the overlapping region. These related contents have been similarly introduced above and will not be repeated here.

[0109] Afterwards, the fingerprint sub-images in the overlapped area can be sorted according to the local image quality score. Here, all fingerprint sub-images in the overlapped area (including the sub-image with the best quality if there is one) need to be calculated. For example, it can be assumed that the local image quality scores of the fingerprint sub-images after sorting are from high to low: Q1, Q2, Q3, ..., Q n Correspondingly, the fingerprint sub-images corresponding to the above order can be assumed to be: P1, P2, P3, ..., P n .

[0110] S1061B: Determine the fusion weight corresponding to each fingerprint sub-image in the overlapped area according to the obtained quality scores of each local image and the similarity between the overlapping portion of each fingerprint sub-image in the overlapped area and the sub-image with the best quality.

[0111] For example, the similarity between each fingerprint sub-image in the overlapped area and the overlapping part of the sub-image with the best quality (as mentioned above, the similarity can be calculated by at least one indicator among the correlation NCC, structural similarity SSIM, mean square error MSE, etc., wherein the larger the correlation NCC and structural similarity SSIM, the greater the similarity, and the smaller the mean square error MSE, the greater the similarity.) can be assumed to be S1, S2, S3, ..., S n It should be noted that the calculation needs to be performed here for all fingerprint sub-images in the overlapping area (including the best quality sub-image if there is one). Optionally, the similarity of the best quality sub-image to itself can be defined as 100.

[0112] Afterwards, the quality scores Q1, Q2, Q3, ..., Q n and each similarity S1, S2, S3, ..., S n Determine the fingerprint sub-images P1, P2, P3, ..., P in the overlapping area n Corresponding fusion weights. For example, corresponding to the above example of 9 fingerprint sub-images, if the local contents of these 9 fingerprint sub-images all exist in a certain overlapping area, then n = 9; if t (t < 9) of these 9 fingerprint sub-images exist in a certain overlapping area, then n = t.

[0113] S1061C: performing weighted summation on the local pixel values ​​of each fingerprint sub-image according to the obtained fusion weights, so as to fuse the local portions of each fingerprint sub-image in the overlapping area.

[0114] The local fusion result is a local image. The local pixel values ​​of each fingerprint sub-image can be weighted and summed by the obtained fusion weights to obtain the pixel values ​​of each pixel point of the local fusion result, thereby achieving the purpose of fusing the local parts of each fingerprint sub-image in the overlapping area and obtaining the local fusion result.

[0115] Based on this, the present application includes the optional implementation of the above sub-steps S1061A to S1061C, which can reliably and effectively fuse the parts of each fingerprint sub-image in the overlapping area, thereby obtaining a reliable local fusion result to facilitate the subsequent generation of a fused image.

[0116] In some optional embodiments, if the local image quality score corresponding to the fingerprint sub-image is higher and the similarity is greater, the fusion weight corresponding to the fingerprint sub-image is determined to be higher.

[0117] In the present application, by determining a higher fusion weight for a fingerprint sub-image with a higher local image quality score and a greater similarity, the image quality of the local fusion result after fusion can be effectively and better guaranteed, and the noise brought by the fingerprint sub-image with poor local image quality during local image fusion can be weakened, which is conducive to avoiding the situation where the image quality of the local fusion result after fusion is worse than the sub-image with the best quality before fusion.

[0118] It can be understood that, in the optional implementation manner of the present application, when locally fusing each fingerprint sub-image within the overlapping area, first, the output local fusion result can be made to include the local clear area of ​​each fingerprint sub-image with a high weight as much as possible, so as to avoid the local blurred area of ​​each fingerprint sub-image entering the fusion with a high weight, resulting in the blurring of the output local fusion result, thereby avoiding the blurring of the subsequently obtained fused image; secondly, the area in which the texture of each fingerprint sub-image is inconsistent with the texture direction of the sub-image with the best quality can be suppressed, so as to avoid the distortion and blurring of the image texture of the output local fusion result, thereby avoiding the distortion and blurring of the image texture of the subsequently obtained fused image.

[0119] For example, the above technical effects can be understood with reference to FIG4 . FIG4 shows a schematic diagram of partially fusing two images. As shown in FIG4 , two images (i.e., image A and image B) are shown with a total of 9 overlapping regions (a1 and b1 overlap, a2 and b2 overlap, ..., a9 and b9 overlap). In some overlapping regions, the local image quality score of image A is higher than that of image B (for example, a1 is better than b1, a4 is better than b4, a5 is better than b5, and a7 is better than b7). In other overlapping regions, the local image quality score of some regions of image B is better than that of image A (for example, b2 is better than a2, b3 is better than a3, b6 is better than a6, b8 is better than a8, and b9 is better than a9). Image C shows an image obtained by directly superimposing and fusing image A and image B in each overlapping region. Each overlapping region of image A and image B is directly fused with a fusion weight of 50%. The local image quality score of each region of the resulting image C is lower than that of the image with the best local image quality in the corresponding overlapping region. Image D shows an image obtained by fusing parts of image A and parts of image B with higher fusion weights according to the higher local image quality scores in each overlapping area (it should be noted that, for the sake of more intuitive understanding, image D shows a special case, which fuses the parts with higher local image quality scores with a fusion weight of 100%, so that the parts of image A or the parts of image B with higher local image quality scores in the overlapping area can completely enter image D). It can be seen that image D has better image quality than image C. In other words, determining a higher fusion weight for fusing images with higher local image quality scores in the overlapping area can effectively and better guarantee the image quality of the local fusion results after fusion, thereby reflecting the above-mentioned technical effects. It should be understood that the description of Figure 4 does not constitute any limitation to the present application.

[0120] Optionally, in this application, the above sub-step S1061C can be implemented by the following formula:

[0121] Among them, BP is used to represent the local fusion result, P i It is used to represent each fingerprint sub-image (P1, P2, P3, ..., P n ), when substituted into the calculation, it can be calculated by the pixel value of the pixel point; Q i Used to represent the local image quality score (Q1, Q2, Q3, ..., Q n ), Q1 is the maximum value of each local image quality score; S i Used to represent the fingerprint sub-image P i The similarity with the overlapping part of the best quality sub-image (S1, S2, S3, ..., S n), S1 is used to represent the similarity corresponding to the fingerprint sub-image P1 with a local image quality score of Q1; n is the total number of fingerprint sub-images in the overlapping area.

[0122] From the above formula, it can be seen that for any fingerprint sub-image P in the overlapping area i , and its fusion weight is: And it can be seen that the fingerprint sub-image P i The corresponding local image quality score Q i The higher the similarity S i The larger the value is, the more accurate the fingerprint sub-image P is. i The higher the corresponding fusion weight, the more effectively the image quality of the local fusion result can be guaranteed.

[0123] When calculating through the above formula, each fingerprint sub-image P i The pixel values ​​of the corresponding pixels in the overlapping area are substituted to implement the weighted sum operation, and the pixel values ​​of each pixel of the local fusion result are obtained, so that the local fusion of each fingerprint sub-image in the overlapping area can be achieved.

[0124] S1062: The local fusion results corresponding to the overlapping areas of the aligned multiple fingerprint sub-images are stitched together to obtain the fused image used for fingerprint recognition. (Please continue to understand this sub-step S1062 in conjunction with the flowchart in Figure 3C)

[0125] After obtaining the local fusion results corresponding to the overlapping areas, they can be stitched together into a fused image, which is the fused image used for fingerprint recognition. Fingerprint recognition can then be performed directly based on the fused image, or the fused image can be further processed before fingerprint recognition.

[0126] Based on this, the present application includes the optional implementation of the above-mentioned sub-steps S1061 to S1062, while taking into account the local image quality of each fingerprint sub-image in the overlapping area and its similarity with the sub-image with the best quality, so as to perform local fusion of the fingerprint sub-images in the overlapping area, so that the fused image spliced ​​by the various local fusion results is more reliable, and the obtained fused image can better aggregate the effective information of multiple fingerprint sub-images, more effectively increase the recognition field of view of fingerprint recognition, and be more conducive to improving the recognition success rate.

[0127] It should also be noted that in other optional embodiments, when "for any overlapping region of the aligned multiple fingerprint sub-images, locally fusing the fingerprint sub-images within the overlapping region to obtain a local fusion result," this can also be achieved based solely on the local image quality of the fingerprint sub-images within the overlapping region, or solely on the similarity between the overlapping portions of the fingerprint sub-images within the overlapping region and the sub-image with the best quality. In other words, the fusion weight can also only consider local image quality or similarity, as long as it meets the requirements.

[0128] It should be understood that the above description of the fingerprint image processing method is merely an exemplary description of the embodiments of the present application and does not constitute any limitation to the embodiments of the present application.

[0129] Based on this, the present application adopts an optional implementation scheme including the above-mentioned steps S102 to S106. On the one hand, a fixed image offset is no longer used to achieve fingerprint image fusion. Instead, the real-time offset of each fingerprint sub-image can be determined more accurately and the alignment and image fusion of multiple fingerprint sub-images can be achieved based on the real-time offset of each fingerprint sub-image. Therefore, it can more effectively adapt to the deviations caused by different fingerprint states and pressing states of fingers on a three-dimensional surface, which is beneficial to improving the quality of multiple fingerprint sub-images after fusion, thereby effectively improving the recognition success rate of fingerprint recognition. On the other hand, since the present solution uses a fused image obtained by fusing multiple fingerprint sub-images aligned according to each real-time offset for fingerprint recognition, it can aggregate the effective information of multiple fingerprint sub-images, increase the recognition field of view of fingerprint recognition, and also help improve the recognition success rate.

[0130] According to a second aspect of the embodiments of the present application, a fingerprint image processing device is provided. Referring to FIG5 , the fingerprint image processing device 500 includes:

[0131] an acquisition module 502 configured to acquire a plurality of fingerprint sub-images currently captured by a plurality of pixel units of the optical fingerprint sensor, and to determine a sub-image with the best quality among the plurality of fingerprint sub-images;

[0132] a determination module 504 configured to determine, for each fingerprint sub-image among the plurality of fingerprint sub-images except the sub-image with the best quality, a similarity between the overlapping portion of the fingerprint sub-image and the sub-image with the best quality based on the reference offset data, and to determine, based on the similarity, a real-time offset between the fingerprint sub-image and the sub-image with the best quality at a current moment;

[0133] The fusion module 506 is configured to align the multiple fingerprint sub-images according to the current real-time offset between each fingerprint sub-image and the sub-image with the best quality, and fuse the aligned multiple fingerprint sub-images to obtain a fused image for fingerprint recognition.

[0134] The solution provided in the embodiments of the present application, on the one hand, no longer uses a fixed image offset to achieve fingerprint image fusion, but can more accurately determine and realize alignment and image fusion of multiple fingerprint sub-images based on the real-time offset of each fingerprint sub-image. Therefore, it can more effectively adapt to the deviations caused by different fingerprint states and pressing states of fingers on a three-dimensional surface, which is conducive to improving the quality of multiple fingerprint sub-images after fusion, thereby effectively improving the recognition success rate of fingerprint recognition; on the other hand, since the present solution uses a fused image obtained by fusing multiple fingerprint sub-images that have been aligned according to each real-time offset for fingerprint recognition, it can aggregate the effective information of multiple fingerprint sub-images, increase the recognition field of view of fingerprint recognition, and also help improve the recognition success rate.

[0135] In some optional embodiments, the reference offset data includes reference offsets between multiple reference fingerprint sub-images and reference quality optimal sub-images, wherein the reference quality optimal sub-image is acquired by a first pixel unit among multiple pixel units, and the multiple reference fingerprint sub-images are acquired by a plurality of second pixel units among the multiple pixel units, respectively, and the plurality of second pixel units are a plurality of pixel units other than the first pixel unit among the multiple pixel units.

[0136] In some optional embodiments, the determination module 504 is specifically configured to: calculate a current reference offset between the fingerprint sub-image and the sub-image with the best quality based on the reference offsets between the multiple reference fingerprint sub-images and the reference sub-image with the best quality, and determine a real-time offset search range at the current moment based on the current reference offset; determine similarities between overlapping portions of the fingerprint sub-image and the sub-image with the best quality, corresponding to respective offsets within the real-time offset search range at the current moment, and use the offset corresponding to the greatest similarity as the real-time offset between the fingerprint sub-image and the sub-image with the best quality at the current moment.

[0137] In some optional embodiments, the determination module 504 is specifically configured to determine a real-time offset search range at a current moment according to the current reference offset and a preset offset margin.

[0138] In some optional embodiments, the fingerprint image processing apparatus 500 is further configured to update the reference offset data according to the real-time offset between each fingerprint sub-image and the sub-image with the best quality at the current moment.

[0139] In some optional embodiments, the fingerprint image processing device 500 is specifically configured to: determine, based on the real-time offsets at current moments between each fingerprint sub-image and the sub-image with the best quality, reference real-time offsets between the fingerprint sub-images captured by the multiple second pixel units and the fingerprint sub-image captured by the first pixel unit; determine, for each reference real-time offset, a reference offset between the reference fingerprint sub-image captured by the corresponding second pixel unit and the reference sub-image with the best quality, and perform a weighted summation of the reference real-time offset and the reference offset according to an updated weight; and update the reference offset data based on the weighted summation results corresponding to each reference real-time offset.

[0140] In some optional embodiments, the fingerprint image processing device 500 is further used to adjust the update weight according to at least one of the image quality of the fingerprint sub-image corresponding to the reference real-time offset and the similarity between the overlapping part of the fingerprint sub-image corresponding to the reference real-time offset and the sub-image with the best quality.

[0141] In some optional embodiments, the fusion module 506 is specifically configured to: for any overlapping area of ​​the aligned multiple fingerprint sub-images, fuse the local areas of the fingerprint sub-images within the overlapping area according to the local image quality of the fingerprint sub-images within the overlapping area and the similarity between the fingerprint sub-images within the overlapping area and the overlapping parts of the sub-image with the best quality to obtain a local fusion result; and splice the local fusion results corresponding to the overlapping areas of the aligned multiple fingerprint sub-images to obtain the fused image used for fingerprint recognition.

[0142] In some optional embodiments, the fusion module 506 is specifically used to: determine the local image quality score of each fingerprint sub-image in the overlapping area; determine the fusion weight corresponding to each fingerprint sub-image in the overlapping area according to the obtained local image quality score and the similarity between the overlapping part of each fingerprint sub-image in the overlapping area and the sub-image with the best quality; and perform weighted summation on the local pixel values ​​of each fingerprint sub-image according to the obtained fusion weights to fuse the local parts of each fingerprint sub-image in the overlapping area.

[0143] In some optional embodiments, if the local image quality score corresponding to the fingerprint sub-image is higher and the similarity is greater, the fusion weight corresponding to the fingerprint sub-image is determined to be higher.

[0144] The fingerprint image processing device 500 provided in the second aspect of the present application is based on the same inventive concept as the fingerprint image processing method provided in the first aspect, corresponds to the corresponding fingerprint image processing methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding fingerprint image processing method embodiments. Therefore, a detailed description thereof will not be given here. Furthermore, the implementation of each module in the fingerprint image processing device 300 of this embodiment can refer to the corresponding descriptions in the aforementioned fingerprint image processing method embodiments, and will not be given here.

[0145] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store a computer program; and the processor is used to execute the fingerprint image processing method provided in the first aspect above by running the computer program stored in the memory.

[0146] 6 , a schematic diagram of the structure of an electronic device according to an embodiment of the present application is shown. The specific embodiments of the present application do not limit the specific implementation of the electronic device. As shown in FIG6 , the electronic device 600 may include: a processor 602, a communications interface 604, a memory 606, and a communication bus 608.

[0147] in:

[0148] The processor 602 , the communication interface 604 , and the memory 606 communicate with each other via a communication bus 608 .

[0149] The communication interface 604 is used to communicate with other electronic devices or servers.

[0150] The processor 602 is configured to execute the computer program 610, and specifically to execute the relevant steps in the above-mentioned fingerprint image processing method embodiment.

[0151] Specifically, the computer program 610 may include program codes including computer operation instructions.

[0152] Processor 602 may be a CPU, a Graphics Processing Unit (GPU), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0153] The memory 606 is used to store the computer program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0154] The computer program 610 may include multiple computer instructions. Specifically, the computer program 610 may enable the processor 602 to execute operations corresponding to the fingerprint image processing method described in any of the aforementioned method embodiments through the multiple computer instructions.

[0155] The specific implementation of each step in the computer program 610 can refer to the corresponding description of the corresponding steps and units in the above-mentioned method embodiment, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working process of the above-mentioned devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, and will not be repeated here.

[0156] According to a fourth aspect of the embodiments of the present application, the embodiments of the present application further provide a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the fingerprint image processing method described in any of the multiple method embodiments provided in the first aspect. The computer storage medium includes, but is not limited to, a compact disc read-only memory (CD-ROM), random access memory (RAM), a floppy disk, a hard disk, or a magneto-optical disk.

[0157] According to a fifth aspect of the embodiments of the present application, the embodiments of the present application further provide a computer program product, including a computer program, which, when executed by a processor, implements any fingerprint image processing method in the multiple method embodiments provided in the first aspect above.

[0158] According to the sixth aspect of the embodiments of the present application, the embodiments of the present application further provide a fingerprint recognition device, which is applied to an electronic device with a display screen. As shown in Figure 7, the fingerprint recognition device 700 includes: an optical fingerprint sensor 702, which is used to: image light signals in multiple directions reflected by the finger above the display screen to collect multiple fingerprint sub-images; and a processing unit 704, which is used to execute the fingerprint image processing method described in any one of the first aspects.

[0159] It should be understood that the optical fingerprint sensor adopts a multi-angle optical path design and can include multiple pixel units. The relevant content of the optical fingerprint sensor has been explained in the method embodiment of the first aspect above and will not be repeated here.

[0160] In addition, as mentioned above, the processing unit can be any processing unit. For example, the processing unit can be a processing unit of an electronic device (for example, including but not limited to a mobile terminal, a computer, a fingerprint lock, etc.) to which the optical fingerprint sensor is applied (for example, it may include one or more chips capable of data processing, including but not limited to a CPU (Central Processing Unit), an MCU (Microcontroller Unit), a GPU (Graphic Processing Unit), an FPGA (Field Programmable Gate Array), etc.), which is conducive to improving the processing effect.

[0161] The fingerprint image processing device 500 / electronic device 600 / computer storage medium / computer program product / fingerprint recognition device 700 embodiments in the embodiments of the present application have been described in detail in the aforementioned fingerprint image processing method embodiments. Therefore, their relevant contents and beneficial effects can be understood with reference to the aforementioned method embodiments and will not be repeated here.

[0162] In addition, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used to train the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0163] It should be noted that, depending on the needs of implementation, the various components / steps described in the embodiments of this application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application. It should be understood that the various technical features in the technical solutions of the embodiments of this application can be combined in any appropriate manner.

[0164] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA)). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., random access memory (RAM), read-only memory (ROM), flash memory, etc.) that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown here.

[0165] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for specific applications, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0166] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.

[0167] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". It should be noted that the concepts of "first", "second", etc. mentioned in the embodiments of the present application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. It should be noted that the modifications of "one" and "multiple" mentioned in the embodiments of the present application are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present application, rather than to limit them. Although the embodiments of the present application have been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A fingerprint image processing method, comprising: Obtaining a plurality of fingerprint sub-images respectively collected by a plurality of pixel units of an optical fingerprint sensor at the current moment, and determining the sub-image with the optimal quality among the plurality of fingerprint sub-images; For each fingerprint sub-image other than the sub-image with the optimal quality among the plurality of fingerprint sub-images, determining the similarity of the overlapping part between the fingerprint sub-image and the sub-image with the optimal quality according to reference offset data, and determining the real-time offset at the current moment between the fingerprint sub-image and the sub-image with the optimal quality according to the similarity; Aligning the plurality of fingerprint sub-images according to the real-time offsets at the current moment between each fingerprint sub-image and the sub-image with the optimal quality, and fusing the aligned plurality of fingerprint sub-images to obtain a fused image for fingerprint recognition.

2. The method according to claim 1, wherein The reference offset data includes reference offsets between a plurality of reference fingerprint sub-images and a reference sub-image with the optimal quality respectively, wherein the reference sub-image with the optimal quality is collected by a first pixel unit among the plurality of pixel units, and the plurality of reference fingerprint sub-images are respectively collected by a plurality of second pixel units among the plurality of pixel units, and the plurality of second pixel units are other than the first pixel unit among the plurality of pixel units.

3. The method according to claim 2, wherein The determining the similarity of the overlapping part between the fingerprint sub-image and the sub-image with the optimal quality according to the reference offset data, and determining the real-time offset at the current moment between the fingerprint sub-image and the sub-image with the optimal quality according to the similarity includes: Calculating a current reference offset between the fingerprint sub-image and the sub-image with the optimal quality according to the reference offsets between the plurality of reference fingerprint sub-images and the reference sub-image with the optimal quality respectively, and determining a real-time offset search range at the current moment according to the current reference offset; Determining the similarities of the overlapping parts between the fingerprint sub-image and the sub-image with the optimal quality respectively corresponding to each offset within the real-time offset search range at the current moment, and using the offset corresponding to the maximum similarity as the real-time offset at the current moment between the fingerprint sub-image and the sub-image with the optimal quality.

4. The method according to claim 2, wherein, The determining the real-time offset search range at the current moment according to the current reference offset includes: Determining the real-time offset search range at the current moment according to the current reference offset and a preset offset margin.

5. The method according to claim 3, wherein The method further includes: Updating the reference offset data according to the real-time offsets at the current moment between each fingerprint sub-image and the sub-image with the optimal quality.

6. The method according to claim 5, wherein, The updating the reference offset data according to the real-time offsets at the current moment between each fingerprint sub-image and the sub-image with the optimal quality includes: Determining reference real-time offsets between the fingerprint sub-images collected by the plurality of second pixel units and the fingerprint sub-image collected by the first pixel unit among the plurality of fingerprint sub-images according to the real-time offsets at the current moment between each fingerprint sub-image and the sub-image with the optimal quality. For each reference real-time offset, determine the reference offset between the reference fingerprint sub-image collected by the corresponding second pixel unit and the reference quality-optimal sub-image, and perform weighted summation on the reference real-time offset and the reference offset according to the updated weight; Update the reference offset data according to the weighted summation results corresponding to each reference real-time offset.

7. The method according to claim 6, wherein, The method further includes: Adjust the updated weight according to at least one of the image quality of the fingerprint sub-image corresponding to the reference real-time offset and the similarity of the overlapping part between the fingerprint sub-image corresponding to the reference real-time offset and the quality-optimal sub-image.

8. The method according to any one of claims 1-7, wherein The step of fusing the aligned multiple fingerprint sub-images to obtain a fused image for fingerprint recognition includes: For any overlapping region of the aligned multiple fingerprint sub-images, fuse the local parts of the fingerprint sub-images in the overlapping region according to the local image quality of each fingerprint sub-image in the overlapping region and the similarity of the overlapping part between each fingerprint sub-image in the overlapping region and the quality-optimal sub-image, to obtain a local fusion result; Stitch the local fusion results corresponding to the overlapping regions of the aligned multiple fingerprint sub-images to obtain the fused image for fingerprint recognition.

9. The method according to claim 8, wherein, The step of fusing the local parts of the fingerprint sub-images in the overlapping region according to the local image quality of each fingerprint sub-image in the overlapping region and the similarity between each fingerprint sub-image in the overlapping region and the quality-optimal sub-image includes: Determine the local image quality scores of the local parts of each fingerprint sub-image in the overlapping region; Determine the fusion weights corresponding to the fingerprint sub-images in the overlapping region according to the obtained local image quality scores and the similarity of the overlapping parts between the fingerprint sub-images in the overlapping region and the quality-optimal sub-image; Perform weighted summation on the pixel values of the local parts of each fingerprint sub-image according to the obtained fusion weights, to fuse the local parts of the fingerprint sub-images in the overlapping region.

10. The method according to claim 9, wherein, If the local image quality score corresponding to the fingerprint sub-image is higher and the similarity is greater, the determined fusion weight corresponding to the fingerprint sub-image is higher.

11. A fingerprint image processing apparatus, comprising: An acquisition module, configured to acquire multiple fingerprint sub-images respectively collected by multiple pixel units of an optical fingerprint sensor at the current moment, and determine the quality-optimal sub-image among the multiple fingerprint sub-images; A determination module, configured to, for each fingerprint sub-image other than the quality-optimal sub-image among the multiple fingerprint sub-images, determine the similarity of the overlapping part between the fingerprint sub-image and the quality-optimal sub-image according to reference offset data, and determine the real-time offset at the current moment between the fingerprint sub-image and the quality-optimal sub-image according to the similarity; A fusion module, configured to align the multiple fingerprint sub-images according to the real-time offset at the current moment between each fingerprint sub-image and the quality-optimal sub-image, and fuse the aligned multiple fingerprint sub-images to obtain a fused image for fingerprint recognition.

12. An electronic device, comprising: A processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus; The memory is used for storing a computer program; The processor is configured to execute the method according to any one of claims 1-10 by running the computer program stored on the memory.

13. A computer storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method according to any one of claims 1-10 is implemented.

14. A computer program product, including a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1-10 is implemented.

15. A fingerprint recognition device, applied to an electronic device with a display screen, the fingerprint recognition device includes: An optical fingerprint sensor, configured to: image optical signals in a plurality of different directions reflected by a finger above the display screen to collect a plurality of fingerprint sub-images; A processing unit, configured to execute the method according to any one of 1-10.

Images