Foreign matter detection method, program product, storage medium and electronic equipment
The gradient value detection method is used to identify foreign objects in the image of the under-screen fingerprint module, which solves the problem of missed foreign object detection in the existing technology and improves the accuracy of fingerprint recognition.
Patent Information
- Application Number
- CN202411943013.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-26
- Publication Date
- 2025-09-12
AI Technical Summary
In the existing technology, the foreign object detection algorithm in the under-screen fingerprint recognition technology has a high missed detection rate for semi-transparent foreign objects such as bubbles, which affects the accuracy of fingerprint recognition.
The gradient value detection method is used to identify foreign objects by calculating the gradient value of the image collected by the under-screen fingerprint module. The presence of foreign objects is determined by using pixels with larger gradient values. The detection accuracy is improved by combining area threshold and connected domain analysis.
It effectively reduces the missed detection rate of foreign objects and improves the accuracy of fingerprint recognition, especially significantly improving the detection effect of foreign objects such as bubbles.
Smart Images
Figure CN120635951A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of fingerprint recognition technology, and in particular to a foreign body detection method, program product, storage medium and electronic equipment. Background Art
[0002] In recent years, under-screen fingerprint recognition technology has been widely used on mobile devices. This technology mainly relies on the fingerprint module set under the device screen to collect fingerprint images. If there are foreign objects on the screen where the fingerprint module is located, typically bubbles, impurities, and stains, it will affect the quality of the collected fingerprint image, resulting in a decrease in the accuracy of fingerprint recognition. In addition, during the testing phase of the under-screen fingerprint module, image collection will also be performed (although the user's fingerprint image is collected at this time), and a series of indicators will be calculated based on the collected image. If there is foreign matter on the screen at this time, these indicators will be calculated inaccurately, which may eventually cause the fingerprint module to have a low recognition accuracy after it is put into use.
[0003] Although there are some algorithms in the existing technology that can detect bad pixels and bad lines in fingerprint modules, bad pixels and bad lines are problems of the fingerprint module itself and are not the same as external foreign objects, resulting in a high missed detection rate of these algorithms. Summary of the Invention
[0004] The purpose of the embodiments of the present application is to provide a foreign matter detection method, program product, storage medium and electronic device to improve the above technical problems.
[0005] To achieve the above objectives, this application provides the following technical solutions:
[0006] In a first aspect, an embodiment of the present application provides a foreign body detection method, comprising: obtaining an image to be detected captured by a fingerprint module arranged below a screen; determining the gradient values at at least some pixels in the image to be detected to obtain a gradient image; and determining a foreign body detection result based on pixels in the gradient image whose gradient values are greater than a first threshold.
[0007] In the image to be detected, there is a certain difference in pixel code values between the foreign object and the surrounding background, so the pixel code values in the area where the foreign object is close to the background change rapidly. The gradient value just reflects the speed of change of the pixel code value in the image. Therefore, based on the pixels with larger gradient values (greater than the first threshold) in the gradient image, the situation of the foreign object in the image to be detected can be well determined, and then corresponding processing measures can be taken in time, ultimately improving the accuracy of fingerprint recognition.
[0008] It should also be pointed out that for some foreign objects, such as bubbles, the pixel code values in the image to be detected are not significantly different from the background. Ordinary code value-based card control algorithms (such as bad pixel and bad line detection algorithms) are less effective in detecting such foreign objects. However, the method of using gradient value for card control can better adapt to this situation and achieve better detection results.
[0009] In an implementation of the first aspect, determining the gradient values at at least some pixels in the image to be detected to obtain the gradient image includes: calculating the gradient value at each pixel of the image to be detected to obtain the gradient image; or determining an effective detection area in the image to be detected, and at least calculating the gradient value of each pixel of the image to be detected within the effective detection area to obtain the gradient image.
[0010] In the above implementation, if the overall quality of the image to be detected is high, the gradient value can be calculated at each pixel to achieve more comprehensive foreign body detection; if the quality of some parts of the image to be detected is not high (for example, there are negative factors such as dark corners that seriously affect detection), the effective detection area can be first delineated from the image to be detected, and the gradient value is calculated only at the pixels within the effective detection area (as for the gradient value at the pixels outside the effective detection area, there are different processing methods, which are explained in detail later), so as to avoid the impact of poor image quality on foreign body detection results.
[0011] In an implementation of the first aspect, at least calculating the gradient value of each pixel of the image to be detected within the effective detection area includes: for each pixel i of the image to be detected within the effective detection area, determining its neighborhood within the effective detection area; for each pixel i' other than pixel i in the neighborhood, calculating a candidate gradient value based on the difference between the code value of pixel i' and the code value of pixel i and the distance between pixel i' and pixel i; and taking the maximum value of all the obtained candidate gradient values as the gradient value at pixel i.
[0012] In the above implementation, the gradient value at pixel i may depend on any pixel i' within the neighborhood of pixel i. This means that gradients in all directions are fully considered, rather than just the x- and y-directions, as in some existing gradient calculation methods. Therefore, the calculated result is more valuable. Furthermore, the gradient values calculated by this method have good continuity (meaning that the gradient values corresponding to spatially similar pixels change gradually rather than abruptly), which is more conducive to implementing subsequent foreign object detection steps (especially some detection algorithms based on pixel connectivity). In contrast, the gradient values calculated by some existing gradient calculation methods are more scattered and have poor continuity, which is not conducive to implementing subsequent foreign object detection steps.
[0013] In an implementation of the first aspect, determining the effective detection area in the image to be detected includes: determining the area composed of pixels in the image to be detected whose code values are greater than a second threshold as the effective detection area; or determining the area composed of pixels in the image to be detected whose code values are greater than the second threshold as the first area, and performing a shrinkage operation on the first area to obtain the effective detection area.
[0014] In the above implementation, if the brightness of the foreign object is greater than the brightness of the background in the image to be detected, the area in the image to be detected with a larger code value (greater than the second threshold) can be determined as a potential area for the presence of the foreign object, namely the first area. As for the area with a smaller code value, it is basically impossible to detect the foreign object and can be excluded. The first area can be directly determined as the effective detection area, or the first area can be further reduced to serve as the effective detection area to further improve the reliability of detection.
[0015] In an implementation of the first aspect, after obtaining the image to be detected captured by the fingerprint module set below the screen, and before determining the gradient values of at least part of the pixels in the image to be detected, the method further includes: performing edge-preserving smoothing filtering on the image to be detected.
[0016] In this implementation, the image to be detected is smoothed before gradient calculation to reduce high-frequency noise, improve the accuracy of gradient calculation, and thus enhance the accuracy of foreign object detection. At the same time, this smoothing filter preserves edges, avoiding smoothing out valuable edges between foreign objects and background.
[0017] In an implementation of the first aspect, determining the foreign object detection result based on pixels in the gradient image whose gradient values are greater than a first threshold includes: determining a connected domain formed by pixels in the gradient image whose gradient values are greater than the first threshold; if the connected domain with the largest area has an area greater than a third threshold, determining that a foreign object exists in the image to be detected.
[0018] In the above implementation, the connected domain composed of pixels with larger gradient values (greater than the first threshold) in the gradient image may be the area where the foreign matter is located, but it is not ruled out that it is just caused by some noise or small dust. Therefore, further judgment is made based on the area of the area. If the area is large enough (greater than the third threshold), it can be confirmed as a foreign matter, and the detection result has a high accuracy.
[0019] In an implementation of the first aspect, the foreign matter is an air bubble, and the image to be detected is an image captured by the fingerprint module when a black object is used to press the fingerprint collection area of the screen.
[0020] Because bubbles are generally translucent, meaning the background can be vaguely seen through them, the code values between the bubbles and the background are typically not significantly different. Therefore, detection methods based on gradient values rather than code values can more accurately detect bubbles. Furthermore, for images captured when a black object is pressed, the bubbles reflect more strongly than the black background, making the code value difference between the bubbles and the background more pronounced, thus further facilitating accurate bubble detection.
[0021] In second aspect, an embodiment of the present application provides a foreign object detection device, comprising: an image acquisition module for acquiring an image to be detected collected by a fingerprint module arranged below the screen; a gradient calculation module for determining the gradient value of at least part of the pixels in the image to be detected to obtain a gradient image; and a foreign object detection module for determining a foreign object detection result based on pixels in the gradient image whose gradient values are greater than a first threshold.
[0022] In a third aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, which, when read and executed by a processor, execute the method provided by the first aspect or any one of the implementations of the first aspect.
[0023] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are read and run by a processor, the method provided by the first aspect or any one of the implementations of the first aspect is executed.
[0024] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory stores computer program instructions, and when the computer program instructions are read and run by the processor, the method provided by the first aspect or any one of the implementations of the first aspect is executed. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.
[0026] Figure 1 This is a diagram of the working principle of the under-screen fingerprint module;
[0027] Figure 2 A flow chart of a foreign body detection method provided in an embodiment of the present application;
[0028] Figure 3The image to be detected in the foreign body detection method provided in the embodiment of the present application;
[0029] Figure 4a This is the image to be detected when a flesh-colored rubber tip is used;
[0030] Figure 4b This is the image to be detected when a black rubber head is used;
[0031] Figure 5a is the image to be detected after mean filtering;
[0032] Figure 5b is the image to be detected after median filtering;
[0033] Figure 6 for Figure 3 The gradient image corresponding to the image to be detected in ;
[0034] Figure 7 for Figure 6 The binary image corresponding to the gradient image in ;
[0035] Figure 8 A functional module diagram of a foreign body detection device provided in an embodiment of the present application;
[0036] Figure 9 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0037] Biometric recognition technology has been widely applied to various terminal devices or electronic devices. Biometric recognition technology includes but is not limited to fingerprint recognition, palm print recognition, vein recognition, iris recognition, face recognition, liveness recognition, anti-counterfeiting recognition and other technologies. Among them, fingerprint recognition generally includes optical fingerprint recognition, capacitive fingerprint recognition and ultrasonic fingerprint recognition. With the rise of full-screen technology, the fingerprint recognition module can be set in a partial area or the entire area below the display screen, thereby forming under-screen (under-display) optical fingerprint recognition; alternatively, part or all of the optical fingerprint recognition module can be integrated into the display screen of the electronic device, thereby forming in-screen (in-display) optical fingerprint recognition. The display screen can be an organic light emitting diode (OLED) display screen or a liquid crystal display (LCD) display screen. The fingerprint recognition method generally includes the steps of fingerprint image acquisition, preprocessing, feature extraction, feature matching and the like. Some or all of the above steps can be implemented by traditional computer vision (CV) algorithms, or by deep learning algorithms based on artificial intelligence (AI). Fingerprint recognition technology can be applied to portable or mobile terminals such as smartphones, tablets, gaming devices, as well as other electronic devices such as smart door locks, cars, bank ATMs, etc., for fingerprint unlocking, fingerprint payment, fingerprint attendance, identity authentication, etc.
[0038] Figure 1 This is the working principle diagram of the under-screen fingerprint module. Figure 1 , electronic devices (e.g., mobile phones, tablet computers) include a screen, and the fingerprint module is set below the screen (e.g., on the motherboard of the electronic device), so it is sometimes referred to as an under-screen fingerprint module. The area on the screen corresponding to the fingerprint module ( Figure 1 When the target object is pressed, the light signal (the medium gray circle) generates a light signal. After being reflected by the target object, the light signal is received by the photosensitive element on the fingerprint module through the screen (the approximate propagation path of the light signal is shown by the arrow), and photoelectric conversion is performed to produce an image of the target object. The fingerprint module may perform some preliminary processing on the image before outputting it. The electronic device can further process the image based on the image collected by the fingerprint module.
[0039] After the electronic device is produced, the target object can be the user's finger, and the image of the target object is the user's finger image, or fingerprint image. The electronic device can compare the currently collected fingerprint image with the fingerprint image in the database to determine whether the current user's identity is legal. This process is the fingerprint recognition process.
[0040] At this time, if there are foreign objects on the screen, such as bubbles between the screen and the film, impurities or stains on the screen surface, the foreign objects will also appear in the fingerprint image, causing the fingerprint image quality to deteriorate, thereby affecting the accuracy of fingerprint recognition, and even causing fingerprint recognition failure.
[0041] In addition, during the production process of electronic devices, under-screen fingerprint modules may also be used to capture images for testing purposes. For example, during the assembly process of mobile phone manufacturers, there is generally a mold testing step:
[0042] After the mobile phone is installed with the under-screen fingerprint module, it is placed on the production line machine. The machine presses the same or different types of rubber heads on the screen area corresponding to the under-screen fingerprint module in turn. The fingerprint module collects the rubber head image after the corresponding rubber head is pressed. The mobile phone performs corresponding test items based on the rubber head image. For example, under the first type of rubber head, it calculates the exposure time, high-frequency component, bad pixel detection, occlusion detection, etc.; under the second type of rubber head, it calculates the signal amount and light leakage ratio; under the third type of rubber head, it calculates the signal-to-noise ratio and magnification, etc. The calculation results of these indicators and preset standards are used to evaluate whether there are any problems with the basic functions of the module, such as imaging.
[0043] During the tooling test, foreign matter on the screen can affect the accuracy of the test results, potentially leading to poor fingerprint recognition accuracy after the phone is put into use. For example, due to worker error, not removing the screen protector during testing, bubbles between the protector and the screen can lead to mismeasurements in several items, such as inaccurate exposure time calibration and bubbles in the generated reference image.
[0044] There's no effective method for detecting foreign objects on screens. While some algorithms use images captured by fingerprint modules to detect bad pixels and lines, the inventors found that these algorithms generally use a specific threshold to control pixel values, distinguishing between normal pixels and those corresponding to bad pixels and lines. However, for translucent foreign objects like bubbles, the pixel values corresponding to these objects don't differ significantly from those of surrounding pixels. This control method is less effective, resulting in significant missed detections.
[0045] Based on this, the embodiments of the present application provide a foreign object detection method, program product, storage medium, and electronic device, which detect foreign objects based on the gradient value in the image collected by the under-screen fingerprint module rather than the pixel code value, which is conducive to improving the above-mentioned problems, reducing the rate of missed detection of foreign objects, and thus helping to improve the accuracy of fingerprint recognition. It should be understood that in addition to the technical solutions proposed in the embodiments of the present application, the above discovery and analysis process of the problems existing in the prior art also constitutes the contribution made by the inventor to this application, and is not content that already exists in the prior art.
[0046] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings. It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.
[0047] The terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.
[0048] The terms "first," "second," etc. are only used to distinguish one entity or operation from another entity or operation, and are not to be understood as indicating or implying relative importance, nor are they to be understood as requiring or implying any actual relationship or order between these entities or operations.
[0049] Figure 2 This is a flow chart of the foreign body detection method provided in the embodiment of the present application. The method can be but is not limited to Figure 9 The electronic device 300 shown in FIG. 1 is executed. For the specific structure of the electronic device 300, please refer to the following description of the electronic device 300. Figure 9 Reference Figure 2 , the method comprising:
[0050] Step S110: Acquire the image to be detected collected by the fingerprint module disposed below the screen.
[0051] Figure 2 The method described above can be applied to the application or testing phase of an under-screen fingerprint module. During the application phase, the image to be detected can be a fingerprint image captured by the fingerprint module after a finger presses the screen. If a foreign object is detected, various measures can be taken: for example, outputting a prompt indicating the presence of a foreign object on the screen or requiring a new fingerprint; another example is removing the portion of the fingerprint image containing the foreign object and performing fingerprint recognition based on the remaining portion; another example is using the entire fingerprint image for fingerprint recognition, but reducing the weight of the portion containing the foreign object, and so on.
[0052] During the testing phase, the image to be detected can be the object image captured by the fingerprint module after the test object (for example, the flesh-colored, black, striped rubber head and other objects mentioned above) presses the screen. If a foreign object is detected, various measures can be taken: for example, output a prompt message indicating the presence of a foreign object on the screen, asking the worker to remove the foreign object and retest; for example, continue the test but reduce the credibility of the test results; for example, continue the test but use a specific compensation algorithm to reduce the impact of foreign objects on the test results, and so on.
[0053] If the device executing step S110 is a device with an under-screen fingerprint module installed, then the image to be detected obtained in step S110 can be the image collected by the fingerprint module; if the device executing step S110 is not a device with an under-screen fingerprint module installed, then the image to be detected may be transferred to the device through network transmission, copying, etc. after the fingerprint module is collected. In this case, the image to be detected can be directly read from the local storage medium of the device. For example, if a mobile phone has an under-screen fingerprint module installed, the mobile phone (its processor) can obtain the image to be detected and perform foreign object detection; if the mobile phone has an under-screen fingerprint module installed, the mobile phone can also send the image to be detected to a remote server to perform foreign object detection on the server.
[0054] The image to be detected can be the original image directly output by the under-screen fingerprint module (which can be called a raw image), or it can be an image obtained after further processing (for example, encoding, denoising, brightness adjustment) of the original image.
[0055] Figure 3 This is the image to be detected in the foreign body detection method provided in the embodiment of the present application. The foreign body to be detected here is a bubble. The green part in the figure is the background (referring to other objects other than bubbles, such as a rubber head), and the yellow part may be a bubble. Note that the image to be detected in practice may be a grayscale image. In order to better display the image content, the grayscale image is colorized. Therefore, Figure 3 These are not completely real images to be detected. Some of the following figures are similar and will not be explained one by one.
[0056] Step S120: determining the gradient values of at least some pixels in the image to be detected to obtain a gradient image.
[0057] The gradient value at a certain pixel in the image to be detected represents the speed of change of the pixel code value at that pixel position in the image to be detected. Ignoring the direction of the gradient, the absolute value of the calculated original gradient value can be taken, so that the obtained gradient values are all non-negative numbers, which facilitates unified size comparison.
[0058] For pixels that need to calculate gradients, a gradient operator can be applied to calculate the gradient value, such as the Sobel operator, Scharr operator, Roberts operator, etc. A gradient calculation method newly proposed by the inventor will be given later.
[0059] At least some of the pixels in step S120 may be all the pixels in the image to be detected. In this way, the size of the calculated gradient image is the same as that of the image to be detected (a corresponding gradient value is calculated for each pixel). Alternatively, at least some of the pixels in step S120 may be a selected portion of the pixels in the image to be detected. The corresponding gradient values are calculated only for these pixels to form the gradient image. In this case, the size of the gradient image is smaller than that of the image to be detected. Of course, the size of the gradient image can also be kept the same as that of the image to be detected. However, the gradient values of some pixels in the gradient image are not calculated, for example, they may be undefined values or default values. The gradient values of these pixels do not affect the foreign object detection result in step S130.
[0060] Step S130: determining a foreign body detection result based on pixels in the gradient image whose gradient values are greater than a first threshold.
[0061] First, it should be noted that the code value of a pixel in a gradient image is called a gradient value, but the gradient value is actually for the image to be detected, not for the gradient image itself.
[0062] In the image to be detected, there is a certain difference in pixel code values between the foreign object and the background. Therefore, in the area where the foreign object is close to the background, the pixel code value changes relatively quickly, or the gradient value is relatively large. Therefore, once the gradient value of a pixel in the gradient image is relatively large (greater than the first threshold), it indicates that the pixel corresponding to the gradient value in the image to be detected is likely to belong to the foreign object. Of course, if only individual gradient values in the gradient image are greater than the first threshold, it cannot be determined that a foreign object has been detected, because foreign objects usually have a certain area. Therefore, the number of pixels in the gradient image with gradient values greater than the first threshold and the preset rules can also be combined to determine whether a foreign object has been detected.
[0063] In addition, the foreign body detection result here may include not only whether a foreign body is detected, but also more information related to the foreign body, such as the location of the foreign body, the number of foreign bodies, the type of foreign body, and one or more items of information.
[0064] Furthermore, the foreign object detection result can also reflect the condition of the object to be detected, where the object to be detected refers to an object that may contain foreign matter. For example, the object to be detected may be the screen of an electronic device, and the foreign matter is a bubble or impurity on the screen. The foreign object detection result can indicate whether the screen is film-coated or clean, and does not necessarily directly describe the bubble or impurity itself. For another example, the object to be detected may be the lens of a fingerprint module, and the foreign matter is a stain on the lens. The detection result can indicate whether the lens is clean, and does not necessarily directly describe the stain itself.
[0065] The first threshold value mentioned above may be a preset value, and different values may be used for detecting different types of foreign matter. The second threshold value and the third threshold value described below are similar and will not be described again.
[0066] Brief summary Figure 2 The method detects foreign objects by calculating the gradient values of at least some pixels in the image to be detected. Since the gradient value reflects the speed of change of the pixel code value in the image to be detected, the pixels with larger gradient values in the gradient image correspond to the areas where the foreign objects are close to the background in the image to be detected. Therefore, based on these pixels with larger gradient values in the gradient image, the situation of foreign objects in the image to be detected can be judged more accurately. If a foreign object is detected, corresponding processing measures can be taken, thereby ultimately improving the accuracy of fingerprint recognition.
[0067] It should also be pointed out that for some foreign objects, such as bubbles, the pixel code values in the image to be detected are not significantly different from the background. Ordinary code value-based card control algorithms (such as bad pixel and bad line detection algorithms) are less effective in detecting such foreign objects. However, the method of using gradient value for card control can better adapt to this situation and achieve better detection results.
[0068] In some implementations, if the foreign object is an air bubble, the image to be detected in step S110 can be an image captured by the under-screen fingerprint module when a black object is pressed against the fingerprint collection area of the screen. The black object here can be the black rubber head mentioned above, but other black objects of other materials are not excluded.
[0069] For the image to be detected collected when a black object is pressed, the bubble reflection will be stronger than the black background, making the code value difference between the bubble and the background relatively more significant, which is more conducive to accurate bubble detection. Of course, this does not mean that the image to be detected collected by other means cannot be used for bubble detection. For example, if the under-screen fingerprint module has been put into use, it will basically only collect fingerprint images, and bubble detection will only be performed based on the fingerprint image. However, during the testing phase of the under-screen fingerprint module, in principle, different rubber head images can be used to detect bubbles. At this time, the black rubber head image can be given priority to detect bubbles.
[0070] Figure 4a This is the image to be detected collected by the under-screen fingerprint module when using a flesh-colored rubber tip. Figure 4b This is the image to be detected collected by the under-screen fingerprint module when using a black rubber head. It is not difficult to see that Figure 4a The yellow part in the middle (probably corresponding to the bubbles) has a more blurred edge, while Figure 4b The yellow part has sharper edges, which is more conducive to distinguishing it from the background, which can improve the accuracy of the detection results.
[0071] In some implementations, between step S110 and step S120 , the image to be detected may be subjected to edge-preserving smoothing filtering, and the filtered image may be used to execute step S120 .
[0072] For example, median filtering is a type of edge-preserving smoothing filter. Specifically, for each pixel in the image to be detected, the code values of all pixels in its neighborhood (e.g., an N×N window centered on the pixel, where N is an odd number greater than 3, such as 3 or 5) are sorted, and the median of the sequence is taken as the filtered code value for the pixel. In addition to median filtering, other methods such as bilateral filtering and guided graph filtering also belong to edge-preserving smoothing filters.
[0073] Smoothing filtering can reduce high-frequency noise in the image to be detected, improving the accuracy of subsequent gradient calculations and, consequently, enhancing the accuracy of foreign object detection. Furthermore, preserving edges in the image during filtering prevents the smoothing of valuable edges between foreign objects and background, which also helps improve the accuracy of foreign object detection. Conversely, some smoothing filters that simply weight pixel values, such as mean filtering and Gaussian filtering, are less effective at preserving edges, hindering the subsequent differentiation of foreign objects from background.
[0074] Figure 5a is the image to be detected after mean filtering, Figure 5b is the image to be detected after median filtering. It is not difficult to see that Figure 5a The yellow part in the middle (probably corresponding to the bubbles) has a more blurred edge, while Figure 5bThe yellow part has sharper edges, which is more conducive to distinguishing it from the background, which can improve the accuracy of the detection results.
[0075] In some implementations, step S120 includes at least the following two implementations:
[0076] Method 1
[0077] Calculate the gradient value at each pixel of the image to be detected to obtain a gradient image. In addition to using some existing operators mentioned above, the gradient value can also be calculated by the following method:
[0078] Step 1: For each pixel in the image to be detected, determine its neighborhood in the image to be detected.
[0079] For ease of explanation, pixel i is used to represent any pixel in the image to be detected. In the image to be detected, the neighborhood of pixel i can be a small area consisting of pixel i itself and several surrounding pixels adjacent to it.
[0080] For example, the neighborhood of pixel i may be an N×N window centered at pixel i, where N may be an odd number not less than 3, such as 3, 5, and so on.
[0081] Step 2: For each pixel in the neighborhood except pixel i, let's call it pixel i'. Calculate the candidate gradient value based on the difference between the code value of pixel i' and the code value of pixel i, as well as the distance between pixel i' and pixel i.
[0082] For example, the candidate gradient value g' corresponding to pixel i' can be calculated using the formula g' = abs(v - v') / distance(p, p'), where v represents the code value of pixel i, v' represents the code value of pixel i', abs represents the absolute value, p represents the coordinates of pixel i, p' represents the coordinates of pixel i', and distance represents the calculated distance. If N = 5 in step 1, a total of 24 candidate gradient values are calculated.
[0083] Step 3: Take the maximum value of all candidate gradient values as the gradient value at pixel i.
[0084] For example, if the maximum value among the 24 candidate gradient values is 200, then 200 is used as the gradient value at pixel i. Since the corresponding gradient value is calculated for each pixel in the image to be detected, the gradient image obtained by method 1 has the same size as the image to be detected.
[0085] In the above gradient value calculation method, the gradient value at pixel i may depend on any pixel i' in the neighborhood of pixel i, and the positional relationship between these pixels and i is different. That is, the calculation fully considers the gradients in all directions to obtain the final gradient value, rather than considering only the gradients in the x and y directions like some existing gradient calculation methods. Therefore, the calculation result is more valuable. In addition, the gradient value calculated by this method has good continuity (meaning that the gradient values corresponding to spatially close pixels are gradual rather than abrupt), which is more conducive to implementing the foreign object detection step in step S130 (especially some detection algorithms based on pixel connectivity). In contrast, the gradient values calculated by some existing gradient operators are more scattered and have poor continuity, which is not conducive to implementing the subsequent foreign object detection step.
[0086] Because Method 1 calculates the gradient value at every pixel in the image to be detected, step S130 actually performs foreign body detection across the entire image to be detected, resulting in a more comprehensive detection. If the overall quality of the image to be detected is high (image quality here can be understood as quality specific to the specific goal of foreign body detection, such as whether there is interference in the image, and not necessarily image quality in the conventional sense, such as image clarity), Method 1 can be selected to calculate the gradient image. Of course, this is not mandatory; Method 2 is also acceptable.
[0087] Method 2
[0088] First, an effective detection area in the image to be detected is determined, and then at least a gradient value of each pixel of the image to be detected located in the effective detection area is calculated to obtain a gradient image.
[0089] There may be some interference in the actual image to be detected, such as Figure 3 The four dark corners or occlusions in the image may affect the foreign object detection results, such as mistakenly detecting the interfering object as a foreign object. Therefore, before performing foreign object detection, the effective detection area can be first delineated from the image to be detected, that is, the area where foreign objects may exist, and these interferences can be excluded from this area. Subsequently, foreign object detection is performed only in the effective detection area (or equivalently only in the effective detection area) to improve the accuracy of the detection results.
[0090] For example, if the brightness of the foreign object is greater than the brightness of the background in the image to be detected, it can be determined that the pixel code value corresponding to the foreign object is relatively large, so the area in the image to be detected where the pixels with smaller code values are located is not the area where the foreign object is located. Therefore, the code value of the pixels in the detection image can be compared with the second threshold, and the area consisting of pixels in the image to be detected whose code value is greater than the second threshold is determined as the effective detection area. As for whether the brightness of the foreign object is greater than the brightness of the background in the image to be detected, it can be determined based on experience. For example, if the foreign object to be detected is a bubble, which is relatively bright, its brightness can be determined to be greater than the brightness of the background.
[0091] Optionally, the region consisting of pixels in the image to be detected whose code values are greater than the second threshold is referred to as the first region. A shrinkage operation may also be performed on the first region, with the shrunk region serving as the effective detection region. A shrinkage operation is an operation that reduces the area of the first region, such as a morphological erosion operation. Since the first region is derived solely through simple pixel code value control, proper shrinkage may better reflect the actual location of the foreign object, thereby improving the reliability of foreign object detection.
[0092] There is no limit to how the position of the area to be detected is recorded. For example, it can be recorded using a separate mask image. The mask image can be a binary image, for example, where the pixel code value can only be 0 and 1. A value of 0 represents that the pixel corresponding to the pixel in the image to be detected does not belong to the area to be detected, and a value of 1 represents that the pixel corresponding to the pixel in the image to be detected belongs to the area to be detected. The shrinkage operation mentioned above can also be performed based on this mask image. Alternatively, the position of the area to be detected can be recorded by the pixel position at the boundary of the area to be detected, and so on.
[0093] The above example is for the case where the brightness of the foreign object is greater than the brightness of the background in the image to be detected. For other cases, similar analysis can be applied and no further examples will be given.
[0094] After obtaining the effective detection area in method 2, the gradient image can be obtained in at least two ways:
[0095] Method a
[0096] Calculate the gradient value of each pixel in the effective detection area of the image to be detected to obtain a gradient image.
[0097] The method for calculating the gradient value can refer to the relevant content in Method 1. The difference is that when determining the neighborhood in step 1, only those pixels within the valid detection area can be included. For example, if only 20 pixels in the 5×5 neighborhood of pixel i are within the valid detection area, only the area consisting of these 20 pixels can be used as the neighborhood of pixel i.
[0098] The gradient image calculated by method a can be considered to have the same shape as the effective detection area. Since the size of the effective detection area is smaller than the size of the image to be detected (if the two are equal, it is equivalent to method 1), the size of the gradient image is also smaller than the size of the image to be detected.
[0099] Of course, for storage convenience, the gradient image and the image to be detected can also be kept the same size. However, in this case, only the code values of some pixels in the gradient image are meaningful (gradient values), and the code values of the remaining pixels have not been calculated and are meaningless.
[0100] For method a, when step S130 is subsequently executed, the position of the effective detection area can be used to ensure that detection is only performed within the effective detection area. For example, only the gradient values of pixels in the gradient image located within the effective detection area are compared with the first threshold, so that meaningless code values are not compared with the first threshold.
[0101] Method b
[0102] The gradient value of each pixel of the image to be detected located in the effective detection area is calculated, and the gradient value of each pixel of the image to be detected located outside the effective detection area is determined as a default value to obtain a gradient image.
[0103] The difference between step b and step a is that step b adds the operation of specifying a default gradient value for pixels in the image to be detected that are outside the effective detection area. As long as this default gradient value is not greater than the first threshold, it will not affect the foreign object detection result in step S130. For example, because the calculated gradient value is the absolute value of the original gradient value, the first threshold is a positive number, so the default gradient value here can be 0 or a negative number.
[0104] The gradient image obtained by method b is the same size as the image to be detected, and the pixel code values in the gradient image (the calculated gradient value or the default gradient value) are all valid and meaningful. Therefore, step S130 can be directly applied to the entire gradient image without considering the location of the area to be detected, making the operation more convenient. However, because the default gradient value does not exceed the first threshold, pixels in the gradient image with the default gradient value will not be detected as foreign objects. This does not affect the foreign object detection results, and is equivalent to performing foreign object detection only within the effective detection area.
[0105] It should be pointed out that if directly specifying the default gradient value is also considered a calculation behavior, then method B can also be considered to calculate the corresponding gradient value for each pixel in the image to be detected, but its calculation method is different for inside and outside the effective detection area, which is obviously different from method A.
[0106] Figure 6 for Figure 3 The gradient image corresponding to the image to be detected is shown in , where the larger the gradient value, the more it tends to yellow, and the smaller the gradient value, the more it tends to blue. Figure 6 The gradient image in can be considered as calculated according to method b. The gradient value of the dark blue area around the image (including the lower left corner) is the specified default gradient value, and the gradient value in the middle is the gradient value actually calculated by a specific algorithm. Figure 6 and Figure 3 ,In the part where the bubble is close to the background, the gradient value is indeed the largest, which is consistent with the bubble detection principle.
[0107] If the quality of some parts of the image to be detected is not high (for example, there are negative factors such as dark corners that seriously affect the detection), you can choose method 2 to obtain the gradient image. Of course, this is not mandatory, and method 1 is also acceptable.
[0108] In some implementations, step S130 may determine whether there is a foreign object in the image to be detected by:
[0109] Step 1: Determine the connected domain consisting of pixels in the gradient image whose gradient values are greater than a first threshold.
[0110] The gradient image here may have different forms. In this step, the location information of the effective detection area may or may not be used. Please refer to the above description of step S120 for details. For example, the gradient image can be binarized based on a first threshold to obtain a binary image. Then, a specific algorithm (such as depth-first traversal or breadth-first traversal) is used in the binary image to determine the connected components therein.
[0111] Figure 7 for Figure 6 The binary image corresponding to the gradient image in , which contains two main connected domains, shown in yellow.
[0112] Step 2: For all connected domains obtained in step 1, the one with the largest area is selected. If the area of the connected domain is greater than the third threshold, it is determined that there is a foreign object in the image to be detected.
[0113] The connected domain formed by pixels with large gradient values (greater than the first threshold) in the gradient image may be the area where the foreign matter is located, but it is not ruled out that it is just some noise or small dust (small dust can also be considered as foreign matter in a physical sense, but it has little effect on fingerprint recognition or test results and can be ignored). Therefore, a further judgment is made based on the area of the area. If the area is large enough (greater than the third threshold), it can be confirmed as a foreign matter, and the detection result is highly accurate.
[0114] For example, Figure 7Of the two connected domains, the larger one has an area of 256 (which may refer to the number of pixels contained in the connected domain), the third threshold is 100, and 256>100, then it can be determined that there are bubbles in the image to be detected.
[0115] Furthermore, based on the connected domain obtained in step 1, in addition to detecting the presence of foreign objects in the image, other information about the foreign objects can also be detected. For example, the approximate location of the foreign object can be determined based on the location of the connected domain; for example, the number of foreign objects can be determined based on the number of connected domains greater than a third threshold; for example, the type of foreign object can be roughly determined based on the shape of the connected domain, and so on.
[0116] Figure 8 FIG2 shows a functional module diagram of a foreign body detection device 200 provided in an embodiment of the present application, which can be used to implement the foreign body detection method provided in an embodiment of the present application. Figure 8 , the foreign matter detection device 200 includes:
[0117] The image acquisition module 210 is used to acquire the image to be detected collected by the fingerprint module provided below the screen;
[0118] A gradient calculation module 220 is used to determine the gradient values of at least some pixels in the image to be detected to obtain a gradient image;
[0119] The foreign body detection module 230 is configured to determine a foreign body detection result based on pixels in the gradient image whose gradient values are greater than a first threshold.
[0120] In one implementation of the foreign object detection device 200, the gradient calculation module 220 determines the gradient values at at least some pixels in the image to be detected to obtain a gradient image, including: calculating the gradient value at each pixel of the image to be detected to obtain the gradient image; or determining an effective detection area in the image to be detected, and at least calculating the gradient value at each pixel of the image to be detected within the effective detection area to obtain the gradient image.
[0121] In one implementation of the foreign object detection device 200, the gradient calculation module 220 at least calculates the gradient value of each pixel of the image to be detected within the effective detection area, including: for each pixel i of the image to be detected within the effective detection area, determining its neighborhood within the effective detection area; for each pixel i' in the neighborhood except pixel i, calculating a candidate gradient value based on the difference between the code value of pixel i' and the code value of pixel i and the distance between pixel i' and pixel i; and taking the maximum value of all the candidate gradient values obtained as the gradient value at pixel i.
[0122] In one implementation of the foreign object detection device 200, the gradient calculation module 220 determines the effective detection area in the image to be detected, including: determining the area composed of pixels in the image to be detected whose code values are greater than a second threshold as the effective detection area; or determining the area composed of pixels in the image to be detected whose code values are greater than the second threshold as the first area, and performing a shrinkage operation on the first area to obtain the effective detection area.
[0123] In one implementation of the foreign object detection device 200, the device also includes: a filtering module for performing edge-preserving smoothing filtering on the image to be detected after the image acquisition module 210 acquires the image to be detected captured by the fingerprint module set below the screen, and before the gradient calculation module 220 determines the gradient value of at least part of the pixels in the image to be detected.
[0124] In one implementation of the foreign object detection device 200, the foreign object detection module 230 determines a foreign object detection result based on pixels in the gradient image whose gradient values are greater than a first threshold, including: determining a connected domain formed by pixels in the gradient image whose gradient values are greater than the first threshold; if the connected domain with the largest area has an area greater than a third threshold, then it is determined that a foreign object exists in the image to be detected.
[0125] In one implementation of the foreign object detection device 200 , the foreign object is an air bubble, and the image to be detected is an image captured by the fingerprint module when a black object is pressed on the fingerprint collection area of the screen.
[0126] The foreign matter detection device 200 provided in the embodiment of the present application, its implementation principle and the technical effects produced have been introduced in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding content in the method embodiment.
[0127] Figure 9 This is a structural diagram of an electronic device 300 provided in an embodiment of the present application. Figure 9 The electronic device 300 includes a processor 310 and a memory 320 , which are interconnected and communicate with each other via a communication bus 330 and / or other forms of connection mechanisms (not shown).
[0128] The memory 320 includes one or more (only one is shown in the figure), which may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The processor 310 and other possible components can access the memory 320 and read and / or write data therein.
[0129] The processor 310 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 310 can be a general-purpose or special-purpose processor, such as a central processing unit (CPU), a micro control unit (MCU), a network processor (NP), a graphics processing unit (GPU), a neural network processing unit (NPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0130] One or more computer program instructions may be stored in the memory 320 , and the processor 310 may read and execute these computer program instructions to implement the foreign matter detection method and other desired functions provided in the embodiments of the present application.
[0131] I understand. Figure 9 The structure shown is for illustration only. The electronic device 300 may also include Figure 9 More or fewer components than shown, or with Figure 9For example, the electronic device 300 may further include a communication module for communicating with other electronic devices. For another example, the electronic device 300 may further include a screen and a fingerprint module, and the fingerprint module is arranged below the screen (see Figure 1 ), used to collect images to be detected, etc.
[0132] also, Figure 9 Each component shown in the figure can be implemented using hardware, software, or a combination thereof. The electronic device 300 may be a physical device, such as a PC, laptop, tablet, mobile phone, server, embedded device, etc., or a virtual device, such as a virtual machine, virtualized container, etc. Furthermore, the electronic device 300 is not limited to a single device and may also be a combination of multiple devices or a cluster consisting of a large number of devices.
[0133] The present application also provides a computer-readable storage medium, which stores computer program instructions. When these computer program instructions are read and executed by a computer processor, they execute some or all of the steps of the sports training auxiliary method provided in the present application. For example, the computer-readable storage medium can be implemented as Figure 9 The memory 320 in the electronic device 300, or a mobile hard disk, optical disk, USB disk, etc. independent of the electronic device 300.
[0134] The present application also provides a computer program product, which includes computer program instructions. When these computer program instructions are read and executed by a computer processor, they execute some or all of the steps of the sports training auxiliary method provided by the present application. For example, the computer program product can be stored in Figure 9 The computer program product is stored in the memory 320 in the electronic device 300 and read and executed by the processor 310. Of course, the computer program product can also be stored in a storage medium independent of the electronic device 300, such as a mobile hard disk, an optical disk, or a USB flash drive.
[0135] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.
Claims
1. A foreign body detection method, characterized in that: include: Obtain the image to be detected collected by the fingerprint module set below the screen; Determining gradient values at at least some pixels in the image to be detected to obtain a gradient image; A foreign body detection result is determined based on pixels in the gradient image whose gradient values are greater than a first threshold.
2. The foreign body detection method according to claim 1, characterized in that: Determining the gradient values of at least some pixels in the image to be detected to obtain a gradient image includes: Calculating the gradient value at each pixel of the image to be detected to obtain the gradient image; or, An effective detection area in the image to be detected is determined, and at least a gradient value of each pixel of the image to be detected located in the effective detection area is calculated to obtain the gradient image.
3. The foreign matter detection method according to claim 2, characterized in that: The step of at least calculating the gradient value of each pixel of the image to be detected within the effective detection area includes: For each pixel i of the image to be detected that is located in the effective detection area, determine its neighborhood that is located in the effective detection area; For each pixel i' in the neighborhood except pixel i, calculate a candidate gradient value based on the difference between the code value of pixel i' and the code value of pixel i and the distance between pixel i' and pixel i; The maximum value of all candidate gradient values obtained is taken as the gradient value at pixel i.
4. The foreign matter detection method according to claim 2, characterized in that: Determining the effective detection area in the image to be detected includes: Determine an area consisting of pixels in the image to be detected whose code values are greater than a second threshold as the effective detection area; or, An area consisting of pixels in the image to be detected whose code values are greater than a second threshold is determined as a first area, and a contraction operation is performed on the first area to obtain the effective detection area.
5. The foreign matter detection method according to claim 1, wherein: After acquiring the image to be detected that is set below the screen and before determining the gradient values of at least some pixels in the image to be detected, the method further includes: Performing edge-preserving smoothing filtering on the image to be detected.
6. The foreign matter detection method according to claim 1, characterized in that: The determining of the foreign object detection result according to pixels in the gradient image whose gradient values are greater than a first threshold value includes: Determining a connected domain consisting of pixels in the gradient image whose gradient values are greater than a first threshold; If the area of the connected domain with the largest area is greater than a third threshold, it is determined that there is a foreign object in the image to be detected.
7. The foreign matter detection method according to any one of claims 1 to 6, characterized in that: The foreign matter is an air bubble, and the image to be detected is an image captured by the fingerprint module when a black object is used to press the fingerprint collection area of the screen.
8. A computer program product, characterized in that The method comprises computer program instructions, and when the computer program instructions are read and executed by a processor, the method according to any one of claims 1 to 7 is executed.
9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the method according to any one of claims 1 to 7 is executed.
10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer program instructions, and when the computer program instructions are read and executed by the processor, the method according to any one of claims 1 to 7 is executed.