Method for identifying an object in an image and mobile device for executing the method

Abstract

A method for identifying a user using an image of a user target having biometric features of the user, such as a fingerprint or a set of fingertip prints, the method comprising: acquiring an image of the target via an optical sensor of a mobile device; providing the image to a neural network; processing the image by the neural network to identify the target in the image and the location of the target in the image; extracting biometric features from the identified target; storing the biometric features in a storage device and / or providing at least the biometric features as input to an identification mechanism, the identification mechanism comprising processing the input to determine whether the biometric features identify the user.

Description

Technical Field

[0001] The present invention relates to a method for identifying a user by using an object in an image having a biometric feature representing the identity of the user, and a mobile device suitable for executing the corresponding method. Background Art

[0002] Image recognition in general is very versatile and can be used in a variety of applications. For example, identifying specific people and faces or objects in an image is used almost ubiquitously by social networks and other media. In addition, in newer smartphones, identity recognition technology is also used to identify the user through, for example, a fingerprint sensor.

[0003] Previous techniques, whether they used "brute force" or newer networks that were specifically trained to recognize objects, required enormous amounts of computer resources to identify objects within images.

[0004] However, a "You Only Look Once" technique has recently been proposed that can achieve significantly faster and more reliable object recognition within an image. The basic principles of this technique are explained in the papers "You Only Look Once: Unified, Real-Time Object Detection" by Redmon et al. and "OLO9000: Better, Faster, Stronger" by Redmon et al.

[0005] The basic concept of the "You Only Look Once" technology (referred to as the YOLO technology in this article) is to divide the obtained image into multiple grids and use a trained neural network in order to identify the target in one or more grid cells using a neural network including multiple reduction layers and convolution layers that each process the obtained image.

[0006] Although the neural networks used achieve adequate results and can detect multiple objects in real time in moving images (videos), they have proven to be not fully applicable to other very specific objects such as fingertips, which results in a longer time required to identify the object.

[0007] In addition, due to the relative complexity of neural networks, a large amount of computer resources are required to achieve real-time target identification, which makes YOLO technology unsuitable for application in existing mobile devices such as smartphones and tablets.

[0008] Purpose of the Invention

[0009] In view of the above, an object of the present invention is to provide a method and system that can achieve rapid user identification while providing high detection accuracy, and at the same time, can simplify the user's interaction with the mobile device for identification. In addition, it is beneficial to reduce the computer resources required for identification, so that identification can be implemented in the current generation of mobile devices. Summary of the invention

[0010] This object is achieved by a method implemented on a mobile computing device and a mobile computing device, and preferred embodiments of the present invention are given.

[0011] According to the method of the present invention for identifying a user using an image of a user target having a biometric feature of the user, such as a fingerprint or a set of fingertip fingerprints, the method comprises:

[0012] Acquiring an image of the target by an optical sensor of the mobile device;

[0013] providing the image to a neural network;

[0014] The neural network processes the image to identify an object in the image and a position of the object in the image;

[0015] extracting biometric features from the identified target;

[0016] The biometric feature is stored in a storage device and / or at least the biometric feature is provided as input to an identification mechanism, which includes processing the input to determine whether the biometric feature identifies the user.

[0017] Here, the storage device may be any device associated with the mobile device itself, or may be a remote storage device provided outside the mobile device. For example, the storage device may be a memory associated with a server of a company to which the biometric features are to be forwarded via a data transmission mechanism such as the mobile Internet or other transmission mechanisms.

[0018] Providing the biometric feature to the identification mechanism may similarly include forwarding the biometric feature internally within the mobile device, for example to a specific application, or forwarding the biometric feature to a remote identification mechanism such as a login server of a bank, social network, etc. via an appropriate transmission mechanism.

[0019] The step of obtaining the image is preferably performed by a free-shot image or an application that allows free-shot images of hands or fingers. This means that such an application does not provide any prompts indicating how the user should position his or her hands and fingers to capture the image for recognition.

[0020] The method enables easy and reliable identification of users while freeing them from any constraints imposed by how they behave or interact with the mobile device.

[0021] In one embodiment, the target is at least one fingertip and the biometric feature is a fingerprint of the fingertip, and wherein processing of the input by the recognition mechanism includes extracting biometric features, such as the location and type of feature points, from the fingerprint, and comparing the extracted biometric features with biometric features stored in a storage device, wherein if the difference between the extracted biometric features and the stored biometric features is below a threshold, the recognition mechanism confirms that the user is identified by the fingerprint, and if the difference between the biometric features and the stored biometric features is above a threshold, the recognition mechanism confirms that the user is not identified by the fingerprint.

[0022] A biometric feature may be any feature that enables biometric identification of a user, or may at least utilize a biometric feature and potentially other features to assist in identifying a user.

[0023] The threshold may be a value indicating whether the biometric feature obtained or acquired using the image matches the stored biometric feature and how much it matches. For example, the threshold may be a real number x, where 0≤x≤1. Here, a larger x means that the obtained biometric feature and the stored biometric feature are allowed to be more different from each other, but at the same time the user can still be identified. If x is smaller, the obtained biometric feature must match the stored biometric feature to a higher degree in order to obtain an identification result.

[0024] By setting the threshold value as needed, the security of identification can be improved.

[0025] In a more specific implementation of this embodiment, the image includes more than one fingertip, and the method further includes: identifying the position of each fingertip in the image, and using the fingerprint of each fingertip to identify the user by the recognition mechanism.

[0026] By using, for example, all fingers for identification, the method of identifying a user is less prone to failure, since forging more than one fingerprint requires a lot of resources and is less likely.

[0027] In one implementation of this embodiment, the recognition mechanism determines that the user is identified through the fingerprints of the fingertips by confirming that the combined recognition accuracy of the fingerprints of all fingertips in the image is higher than a given threshold, or the recognition mechanism determines that the user is identified through the fingerprints of the fingertips by confirming, for each fingertip, whether the difference between the biometric features of the fingertip fingerprint and the biometric features of the stored fingertip fingerprint is lower than a threshold, and determines that the user is identified through the fingertip fingerprint only when all the determined differences are lower than the corresponding threshold.

[0028] The combined recognition accuracy has to be understood as the combination of the recognition accuracies of the individual biometric features acquired in isolation. This means, for example, that the biometric features of each fingerprint need to be evaluated separately from the other fingerprints. In the above embodiment, the biometric feature is considered to be consistent with the stored biometric feature if the difference is below a given threshold. The relative value of the degree of compliance between the obtained biometric feature and the stored biometric feature can represent the recognition accuracy. For example, if there is a 99.9% match between the obtained biometric feature and the stored fingertip biometric feature, the recognition accuracy can have a value of 0.999. The sum of all recognition accuracies can then be obtained, and if this sum is above a threshold, which can, for example, depend on the threshold value of whether a single obtained biometric feature is considered to be consistent with the stored biometric feature, the user is considered to be identified by the obtained biometric feature.

[0029] In one embodiment, the image is obtained by a camera as an optical sensor of the mobile device. This makes the method of the present invention applicable to contemporary mobile devices such as smartphones, since almost every currently available smartphone has at least one camera.

[0030] In one embodiment, processing an image as an input by a neural network includes: processing the input by a first layer of the neural network to create a first intermediate output, and processing the output of the previous layer by each subsequent layer, wherein the neural network includes a plurality of layers, each layer being a depthwise separable convolution, including, in the order in which the input is processed within the layer, a depthwise convolutional layer, a first batch normalizer, a first rectified linear unit, a pointwise convolutional layer, a second batch normalizer, and a second rectified linear unit;

[0031] Among them, by using multiple layers to process the input, the neural network obtains the recognition result of the target and the position of the target in the image as output.

[0032] The expected deep convolutional layers use the product or inner product of the feature map (matrix) corresponding to the original image and the kernel in the form of a matrix of size, for example, 3×3, to calculate another matrix. For recognition efficiency, it is more efficient to use these layers. This is specifically because the maximum pooling layer will cause information loss, which requires more iterations. In view of this, the deep convolutional layers proposed in the above embodiments are more effective than the commonly used convolutional layers in terms of their parameter sensitivity.

[0033] Depthwise convolutional layers and pointwise convolutional layers can also be referred to as depthwise convolutional sublayers and pointwise convolutional sublayers. In fact, they are "layers within layers" in neural networks and thus constitute sublayers.

[0034] By applying this specific implementation method of deep convolutional layers and point-by-point convolutional layers, batch normalizers and rectified linear units as given in the above embodiments, the computer resources required for the neural network to perform real-time recognition of targets carrying biometric features such as fingertips in images are significantly reduced compared to currently known neural technologies. The reason is that the well-known YOLO technology relies on a maximum pooling layer as one of the layers in the layer group within the neural network used.

[0035] In one embodiment, creating the output includes separating the image into a grid comprising Q×R grid cells during processing, wherein at least one bounding box is created within each grid cell, the bounding box having a predetermined position and predetermined geometric features within the grid cell, wherein creating the output further includes modifying the position and geometric features of the bounding box to obtain a resulting bounding box, wherein the resulting bounding box is a bounding box having a resulting position and resulting geometric features that most closely matches the location of the target.

[0036] Separating the obtained image into grid cells with predetermined bounding boxes can correctly display the target identified by using the bounding boxes in the final result and correctly provide feedback about the target to mark the location of the target as well as the target itself.

[0037] In a more specific implementation of this embodiment, the position of the bounding box is calculated relative to the center of the grid cell in two dimensions, and the geometric features of the bounding box include the height and width of the bounding box, wherein, further, the probability of the target being within the bounding box is associated with each bounding box.

[0038] Associating the bounding boxes with the corresponding probabilities provides a matrix or vector representing the bounding boxes, and when it has to be combined with other objects represented in matrix or vector form, it can be processed by the graphics processing unit with accurate efficiency. Thus, the required computer resources are even further reduced.

[0039] More specifically, the output may be a tensor T of dimension Q×R×B×A, where A is the number of different bounding boxes in each grid cell, and B is a vector associated with each bounding box, which has 5 dimensions and is expressed as follows:

[0040]

[0041] The resulting tensor can be processed by the GPU in an efficient manner. In addition, presenting the recognition results in the form of such a tensor makes it possible to easily infer the result that is most likely to recognize a particular object.

[0042] In addition, the output may include displaying an image and the resulting bounding boxes in each grid cell, where the resulting bounding box is the bounding box with the highest probability among all the bounding boxes in that grid cell.

[0043] By only providing the grid cells with the highest probability, the position and the identification of the object are provided to the user by including the bounding boxes of the separately identified objects, which provides easily recognizable feedback. In addition, the resulting bounding box only represents a vector within the result tensor that was provided as output in the previous embodiments, and thus can be easily extracted by the user or other programs by only taking the respective coordinates of the resulting bounding box and used for further processing.

[0044] While this way of identifying the position of the fingertip within the image may be preferred as it has proven to consume fewer resources than other methods, other methods may also be considered. For example, initially proposals may be made for regions where a fingertip may be present. These proposals can then be further processed to find out if there is indeed an object similar to a fingertip among the proposals for that region.

[0045] In another embodiment, processing the image by a neural network includes creating at least one matrix I representing the color values of the individual pixels in the image from the image and providing this matrix as an input to the neural network, where the image includes N×M pixels and matrix I is a matrix including N×M values, where the entries of matrix I are given by I ij where i and j are integers and i = 1...N and j = 1...M.

[0046] This separation of the image into a matrix of the individual color values enables the color values to be processed separately, thereby advantageously increasing the recognition efficiency while reducing the computer resources required.

[0047] More specifically, each depth convolutional layer applies a predefined kernel K to matrix I, where kernel K is a matrix of size S×T, where S,T < N; S,T < M and includes entries S ab where applying the kernel to the matrix includes calculating the inner product of matrix K with each reduction matrix R of size (N×M) S,T where matrix R is the same size as kernel K, and matrix Z is of size ((N + 2P w )×(M + 2P h )) and the entries of matrix Zcd (c,d ∈ N+) are given by the following formula

[0048]

[0049] and provides matrix P as output, where the size of matrix P is where W wand W h Define the span width, each item P of the matrix P ij is the value of the inner product of the ijth reduced matrix R and the kernel K, where the matrix P is provided as the output by the deep convolutional layer to the first batch normalizer.

[0050] The kernel allows to properly weigh the information obtained from neighboring pixels in the feature map without losing any information, thereby increasing the efficiency with which successive layers in the neural network can support object recognition. To this end, the kernel includes terms corresponding to specific weighting values ​​or parameters obtained before receiving the image (i.e., during training the neural network).

[0051] One discovery of the present invention is that if this training is performed before the mobile device is actually equipped with an application or other program capable of performing the various methods according to the above-described embodiments, the computer resources required on the mobile device can be advantageously reduced.

[0052] Although the invention has been found to be most advantageous to implement separable convolutions using depthwise convolutional layers and pointwise convolutional layers, since this combination shows improved performance in terms of recognition and required computer resources, it is still conceivable to replace the depthwise convolutional layers with convolutional layers that are particularly suitable for recognizing fingers or fingertips. Therefore, even though the description of the invention focuses on the use of depthwise convolutional layers, the invention may also be implemented using convolutional layers.

[0053] In another embodiment, the batch normalizer computes To calculate the mean V from the matrix P, and create a matrix with the term P' ij =P ij - The batch normalization reduction matrix P' of V.

[0054] By applying this normalization, unintentional effects such as overexposure can be filtered out through various layers throughout the image processing process, making it possible to improve the efficiency of identifying objects in the image.

[0055] Furthermore, the kernel sizes S and T can be equal for all convolutional layers, or different for at least one convolutional layer.

[0056] By selecting the same kernel for each convolutional layer (i.e., each depthwise convolutional layer), the size of the resulting program installed on the corresponding mobile device can be reduced. On the other hand, if different kernels are used for at least one of the convolutional layers, known problems with recognition failures can be avoided if the kernels are appropriately rewritten. For example, using a larger kernel (equivalent to larger S and T) at the beginning of the recognition process can provide the possibility to obtain and focus on more important parts of the image, thereby improving recognition efficiency.

[0057] In a specific embodiment, S,T=3 and is the same for all depthwise convolutional layers, and at least one of S a′b′ ≠S a≠a′，b≠b′ .

[0058] One finding of the present invention is that the corresponding kernel represents an optimal balance between the size of the kernel, the recognition efficiency and the computer resources required to implement the corresponding method, thereby improving the overall efficiency in terms of recognition accuracy and required computer resources.

[0059] In a further embodiment, the batch normalizer provides the normalized reduction matrix P' to the rectified linear unit, and the rectified linear unit performs a linear transformation on each term P'. ij Apply a correction function that computes the term The new matrix And, if the rectified linear unit is the first rectified linear unit, then the matrix is fed as output to a point-wise convolutional layer, or if the rectified linear unit is the second rectified linear unit, the matrix It is given as output to the next layer of the neural network.

[0060] This correction function makes it possible to filter out parts of the image that could have a negative impact on the recognition accuracy after each layer of the neural network. As a result, the number of false recognitions can be reduced and the number of iterations necessary to achieve adequate recognition accuracy can be reduced accordingly, thus saving computer resources.

[0061] It can also be specified that by taking the matrices P, P' or Each entry in is multiplied by a weight value α, and the point-by-point convolutional layer applies the weight value α to the matrix I, P, P' or

[0062] Even if the same weighting value α is applied to each point in the feature map, this embodiment can effectively attenuate parts of the image (corresponding to entries in the matrix that do not significantly affect recognition). This attenuation is achieved by reducing the absolute contribution of these parts in the matrix and removing them in the next cycle together with the rectified linear unit.

[0063] In a preferred embodiment, each step of the method explained above is performed on a mobile device.

[0064] This may include at least the steps of the method described above involving image processing and user identification. The storage of the image or biometric features or biometric identification features may still be performed by any storage device inside or outside the mobile device. Furthermore, it is still conceivable that the identification step of identifying the user is performed on a device different from the mobile device (e.g. a company's server).

[0065] By only performing the individual steps on the mobile device, it is no longer necessary to keep open a channel for transmitting data, for example, to a server on which the actual identification process is run. Thus, target identification can also be used in areas where a mobile network or a local area network cannot be accessed.

[0066] The mobile device according to the present invention comprises an optical sensor, a processor and a storage unit storing executable instructions, wherein when the executable instructions are executed by the processor of the mobile device, the processor executes the method of any one of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A general block diagram showing a method for identifying a user according to the present invention;

[0068] Figure 2 A more specific flow chart representing identification using biometric features extracted from an image;

[0069] Figure 3 A schematic diagram showing a general process flow for identifying an object within an image according to one embodiment is shown;

[0070] Figure 4 Schematically illustrates the structure of a layer in a neural network and data processing in the layer according to one embodiment;

[0071] Figure 5 is a schematic diagram of processing an image of a hand using a bounding box;

[0072] Figure 6 Represents the process of training a neural network;

[0073] Figure 7 A mobile device for obtaining an image and identifying an object within the image is shown according to one embodiment. DETAILED DESCRIPTION

[0074] Figure 1 The overall block diagram of the method for identifying a user using an image of an object of the user according to the present invention is shown. The method starts at step 11, in which an image of this object is obtained. This object of the user will have a biometric feature, with which the user can be identified. In particular, the object can be a fingertip or hand or multiple fingertips of the user, and the biometric feature obtained from the image can be at least one fingertip fingerprint or even a set of fingerprints of, for example, two, three or four fingertips.

[0075] The image may be obtained using an optical sensor such as a camera. Most preferably, the optical sensor is an optical sensor of a mobile device such as a common smartphone on the market. The camera may be a camera capable of obtaining a high-definition image of one million pixels or more.

[0076] The obtained image is then provided to the neural network for processing in step 12, which step will be described in detail below. Providing the image to the neural network may include forwarding or transmitting the image to a corresponding application implementing the neural network within the mobile device or providing the image to a remote location. This remote location may be a server or other computing entity. However, preferably, the image is provided to the neural network resident in the mobile device.

[0077] Then, in step 13, the image is processed by a neural network, which will be referred to below. Figures 3 to 6 Explained in more detail. In any case, the processing of the image by the neural network will result in the identification of the location of the target with the biometric feature and the target itself in the image. This means that, for example, in the case where the target is a fingertip, the neural network will identify the fingertip in the image (i.e. determine that the fingertip is present in the image) and identify its location in the image. Identifying the location of the fingertip within the image may, for example, include: identifying all pixels belonging to the fingertip, or at least identifying a sub-portion within the image that is different from the entire image, for example, a portion corresponding to one tenth of the total area of ​​the image.

[0078] In a next step 14, biometric features are extracted from the identified object. This extraction may consist, for example, in extracting only those parts of the identified fingertip that actually constitute the fingerprint.

[0079] The biometric feature can then be further processed. This is shown in steps 15 and 16.

[0080] In step 15, only the biometric features are stored. Storing the biometric features may include storing the biometric features preferably on a non-volatile storage device. This storage device may be a device device such as a solid-state memory in the mobile device itself, or a storage device in a remote storage location. The remote storage location may be a company's server or any other remote storage location. In this case, the biometric features are forwarded in the form of data packets (e.g., images or PDFs or numerical values, etc.) via a data transmission mechanism such as a LAN connection or a WLAN connection or via the mobile Internet.

[0081] In addition to storing the biometric features in any manner, or instead of storing the biometric features as in step 15, the biometric features may be forwarded to an identification mechanism as input as in step 16. This identification mechanism may be an application resident in the mobile device that captured the image of the user target with the biometric features, or it may be a remote identification mechanism, such as a log in a server or other entity, which uses the biometric features to identify the user and perform other steps, such as logging into a social network, a bank account, etc.

[0082] Figure 2 A more detailed explanation of how a biometric is used to identify a user is presented where the biometric is a fingerprint, where the target would be at least one fingertip.

[0083] Figure 2 The method in the embodiment begins at step 21, in which a biometric feature is extracted from a fingertip. Thus, these steps are at least Figure 1 It is performed after the step of extracting biometric features from the identified target in step 14 explained in the above.

[0084] Extracting biometric features from a fingertip may, for example, include extracting the location of the fingerprint and the type of feature points. It may also include extracting only very specific types of feature points (such as the intersection of two or more lines in a fingerprint).

[0085] In order to use this information to identify the user, it is of course necessary to provide a reference in the form of a corresponding biometric feature. For this reason, it may be the case that the above reference Figure 1 The identification mechanism explained is associated with or includes a storage device in which biometric features for a specific user are stored. For example, for each user, there may be a file in which one or more biometric features are stored, for example in the form of an image, a numerical value, or other data structure.

[0086] In a next step 22, the biometric feature obtained from the fingerprint is compared with the corresponding stored biometric feature. This may include, in the case where the stored biometric feature is represented by the positions of a plurality of feature points, comparing the corresponding positions in the extracted biometric feature. Of course, other means for comparing the obtained biometric feature with the stored biometric feature are known and may be used, such as image recognition techniques, frequency transformations, etc. According to the invention, the comparison of the obtained biometric feature with the stored biometric feature is performed in such a way that the degree of correspondence between the obtained biometric feature and the stored biometric feature can be calculated. In other words, this comparison will result in the difference between the stored biometric feature and the obtained biometric feature being calculated. This difference may be a single real number or a tensor or a vector or any other mathematical structure. It may also be a difference image obtained by subtracting the obtained biometric image from the stored biometric image pixel by pixel.

[0087] A threshold may be provided that may be used to determine whether the acquired biometric characteristic matches the stored biometric characteristic, thereby enabling user identification.

[0088] Accordingly, in step 23, it is determined whether the difference between the obtained biometric feature and the stored biometric feature is below or above this threshold. If below the threshold, it is determined in step 25 that the user has been identified by the biometric feature. If the difference is above the threshold, it is determined in step 24 that the user has not been identified by the identification feature.

[0089] This will then result in the identification mechanism confirming whether the user is identified by the acquired fingerprint or whether the user is not identified by the acquired fingerprint.

[0090] Figure 1 and Figure 2 A method is described for identifying a user using biometric features obtained from an original captured image when only one fingertip is used to identify the user and the fingertip is present in the image.

[0091] However, it is also conceivable that the recognition mechanism can evaluate not only a single fingertip, but more than one fingertip, for example two fingertips or even all fingertips available in the image, in order to recognize the user. The recognition mechanism matches the biometric features obtained from the fingerprint of a single fingertip or multiple fingertips with the stored biometric features in a manner similar to that of the reference. Figure 2 consistent with the method described.

[0092] However, if more than one fingerprint is evaluated, it may be possible to identify the user only if the combined recognition accuracy of the biometrics is above a given threshold, or only if for each of the acquired fingertips, e.g. Figure 2The user is only identified if the comparison of the acquired biometric features explained in steps 22 and 23 with the stored biometric features leads to the result of step 25 .

[0093] The last case is simple, since a reference is performed for each fingerprint in the image. Figure 2 The method explained above is described in detail and for each obtained fingerprint, the user can be identified only if the difference between the obtained biometric feature and the stored biometric feature for each feature is below a given threshold. In any other case, the user may not be identified.

[0094] However, in the case where a user is identified, the comparison of the acquired biometrics with the stored biometrics for each fingertip does not necessarily have to be as exact as for each fingertip if the combined identification accuracy of the fingerprints of all fingertips in the image is above a given threshold. Figure 2 The difference below the threshold is obtained as in step 23 of FIG.

[0095] For example, considering that the recognition accuracy of a biometric is a number ranging from 0 (no recognition) to 1 (perfect match between the obtained biometric and the stored biometric), if the combined recognition accuracy is determined by the sum of the isolated recognition accuracies obtained only for each biometric individually, the combined recognition accuracy can have a value less than 4 (corresponding to perfect recognition accuracy for four fingerprints).

[0096] For example, the corresponding threshold value for the combined recognition accuracy may be 3.5. In this case, for example, if the recognition accuracy of each fingerprint is about 0.9, it is sufficient to recognize the user, because the sum of these recognition accuracies (i.e., the combined recognition accuracy) is 3.6, which is higher than the corresponding threshold value. As another example, considering that three fingerprints are to be recognized with a recognition accuracy of 0.95, it is sufficient if the fourth finger is recognized with only an accuracy of 0.75.

[0097] Note that recognition accuracy can be viewed as the relative degree of similarity or agreement between the acquired biometric and the stored biometric. Thus, if the acquired biometric is 90% consistent with the stored biometric, the recognition accuracy (i.e., how accurately the user can be identified using that biometric) would be 0.9.

[0098] Obviously, other values ​​can also be used for the recognition accuracy, or even other values ​​for the threshold value. In addition, there are other means by which the combined recognition accuracy can be determined. For example, the combined recognition accuracy can be calculated by determining the average value of the recognition accuracies or by determining the product of the recognition accuracies.

[0099] In the following figures, the processing of the original obtained image to finally extract the biometric features in steps 12 to 14 will be described in more detail, and an explanation will be further given on how to train the neural network to be able to recognize fingertips with high accuracy.

[0100] Figure 3 Depicted is a diagram according to an embodiment Figure 1 The method now explained aims to achieve the goal of identifying a biometric feature of the user in the image obtained. In the sense of the present invention, the image is preferably in the first step 101 (corresponding to Figure 1 The image is obtained in step 11) by an optical sensor of a mobile computing device. The mobile computing device may be a smartphone or a tablet computer or other corresponding device. The optical sensor will therefore typically be a camera, but may also be an infrared camera or other optical sensor. The camera may be a camera with a resolution of 1 megapixel (MP), or may be an HD camera, or may even have a lower resolution. Preferably, the resolution of the obtained image is at least 224×224, with each pixel having three color values.

[0101] The image obtained may include an object which according to the invention is to be identified not only for itself (e.g. a fingertip) but also for its position in the image. The object is intended to carry or have a biometric feature of the user, so that the user can be correctly identified. This means that the object must be an object such as a fingertip or a set of fingertips with a fingerprint. It is known that fingerprints can be used to uniquely identify a user, i.e., except for some very special cases, a fingertip is unique for each person, so that two people can be distinguished based on the fingerprints obtained.

[0102] Although for Figure 1 The present invention and the subsequent figures all refer to "images", but the present invention enables real-time object recognition, so the required processing time is in the range of a few milliseconds, and thus also enables correct recognition of objects in continuous images, such as videos obtained by optical sensors or real-time streams. Therefore, the term "image" should be understood to refer not only to a single image, but also to images obtained continuously in a short period of time, such as a video stream.

[0103] In fact, as is common with smartphones, when the camera is activated, the user of the smartphone can obtain the actual view of the camera without even taking a picture. Therefore, this "preliminary view" is also composed of a number of images taken by the camera, usually at a lower resolution. Even for these images, the described method of the invention can be used.

[0104] In the second step 102 of the method, according to Figure 1In step 12, the acquired image (or images acquired successively one after the other) is provided to a neural network, wherein the neural network preferably but not necessarily resides on the mobile device.

[0105] The neural network may be implemented in an application (app) or in any other program running on the mobile device. In a preferred embodiment of the invention, the further processing by the neural network and any other steps performed in the method of the invention are performed without having to refer to any computing entity external to the mobile device, thus also providing the possibility to perform the method in an "offline" mode of the mobile device.

[0106] Step 102 may be implemented by forwarding the image directly to the neural network without any further processing of the image or without any further preprocessing of the image. However, this step may also include preprocessing of the image, wherein for example the resolution of the originally obtained image is changed, in particular reduced. One finding of the present invention is that, in particular in the case of identifying fingertips within an image, it is sufficient to have a relatively low resolution of 224 x 224 x 3 ("3" corresponds to the three color values ​​of the image, namely blue, red and green). In the case where the image obtained has a much larger resolution than the 224×224 image resolution required for identifying the fingertips, step 102 or a step provided between steps 102 and 103 may include reducing the resolution of the image. This preprocessing may also include other steps, such as changing the brightness conditions, changing the gamma value within the image or providing any other preprocessing deemed suitable.

[0107] After the image is provided as input to the neural network in step 102, the input is processed by the neural network in step 103 in such a way as to create possible outputs for identifying objects and / or locations of objects within the image. In the case where the object is a fingertip, this means that at least one fingertip present in the identified image (e.g., in the form of a label) and its location (e.g., the coordinates of the pixels that constitute the fingertip) are also provided in some way in the output. As will be explained later, this can be achieved by providing a bounding box that surrounds and includes the identified fingertip at a position corresponding to the fingertip, the bounding box being superimposed on the fingertip. The coordinates of this bounding box relative to the image can then be used as the location of the fingertip.

[0108] The processing of the input (i.e., essentially the received image) in step 103 may be facilitated in a number of ways through the use of a neural network. In any case, it is intended that the neural network is a trained neural network specifically trained to recognize a desired target with a biometric feature. More preferably, the neural network is trained to recognize fingertips within the input image regardless of the position and arrangement of the fingertips relative to the optical sensor, as long as the optical sensor can capture an image of at least one fingertip. As will be explained later, the processing may involve processing the input through multiple layers of a neural network.

[0109] According to the present invention, this includes at least: the input is processed by a first layer of the neural network to create a first intermediate output, which is then processed by a layer that follows the first layer in the processing direction of the neural network to create a second intermediate output. The second intermediate output is then forwarded to the next layer in the neural network, where it is processed to create a third intermediate output, and so on, until all layers in the neural network have processed their respective received intermediate outputs. The last layer in the neural network will provide a "final" output, which may later be the output in step 104, as explained later.

[0110] In addition, according to the present invention, each layer of the neural network is composed of two convolutional layers, so that each layer of the neural network represents a depth-separable convolution filter, also known as a depth-separable convolution. In the processing order of the input through the depth-separable convolution, the depth-separable convolution (i.e., the layer of the neural network) includes a depth-convolution layer, a first batch normalizer, and a first rectified linear unit. In the processing order after the first rectified linear unit, a point-by-point convolution layer, a second batch normalizer, and a second rectified linear unit are provided, wherein the second rectified linear unit or a processing module receiving the output from the rectified linear unit will forward the intermediate output to the next layer of the neural network.

[0111] After the image is processed through all the layers of the neural network, an output is created that will ultimately identify the location of the object and the object itself.

[0112] This is done in step 104, where the output of the neural network is output. According to a preferred embodiment of the present invention, the output may be a "modified image" where the image may be enhanced with a bounding box surrounding the identified fingertip to provide feedback to the user regarding the identified target and its location.

[0113] However, the output need not be displayed on the display of the mobile device or any other display associated with the mobile device. In fact, the output can also be provided in the form of a matrix or tensor, as described below, which can correctly identify the position of the fingertip in the image (particularly the coordinates of the pixels in the image that constitute the fingertip), and the matrix or tensor can be forwarded to a further processing module to use this information, specifically the coordinates of the fingertip, to further process the identified fingertip. Figure 1 This output can then be used to extract biometric features from the identified target, as in step 14 of FIG.

[0114] This further processing may preferably include evaluating the identified fingertips to identify the user's fingerprint. For example, considering a high-resolution image acquired from a fingertip, the method of the present invention may include: in a first step, identifying the position of the fingertip in the image using a method including steps 101-104 as described above, and then forwarding the final output to another image processing component, which uses the output of the identified fingertips and their positions to evaluate the high-resolution image to identify the fingerprint. This can be used to identify the user, thereby increasing the reference Figure 2 Security of further processing of the explanation. For example, if a user uses the method of the invention to identify himself via his mobile device to make a bank transfer, the method of the invention can improve the security of the bank transfer by enabling correct and unique identification of each user, since the user's fingerprint will uniquely identify this person.

[0115] The invention is not limited to performing bank transfers using a corresponding method for identifying fingertips, but can also be used to identify a user in order to enable, for example, access to mobile device functions or any other activity requiring identification and authentication of a user.

[0116] Figure 4 200 of a neural network according to an embodiment of the present invention. The layer 200 may be the first layer that receives the original input after step 102 explained above, or any intermediate layer arranged between two other layers 240 and 250, or the layer 200 may even be the final layer of the neural network that will be processed according to the reference. Figure 1 Step 104 of the explanation provides the final layer of output.

[0117] In any case, the layer 200 will receive an input 230 that corresponds, at least to some extent, to the original acquired image. This input is preferably provided in the form of at least one matrix of size N×M, where N and M are integers greater than 0. For example, the matrix may represent pixels of at least one color value in the image (for example red). Thus, the entries in the matrix may have values ​​corresponding to the values ​​of the corresponding color of that particular pixel (red in the example case). As will be clear from the following, this input may not be the same as the acquired image, but may be a matrix P obtained from a matrix representing the original image by some processing performed by the layers in the neural network or even by some preprocessing (for example, resolution reduction as described above).

[0118] However, for ease of discussion, it will be assumed that the input 230 corresponds to an N×M matrix representing the original obtained image, and each entry in this N×M matrix corresponds to the value of the color (e.g., red) of a pixel in the corresponding image. It is straightforward to apply this idea to any other transformation matrices derived from the original N×M matrix and obtained by processing the matrix in various layers of the neural network.

[0119] Now follow Figure 4 In the process illustrated in , input 230 is received by the depthwise convolutional layer 211 for processing. Subsequently, a relatively simple example of how the depthwise convolutional layer processes the input matrix 230 will be given. This will involve using a kernel K to calculate the inner product with the matrix. The kernel operates on the matrix with a so-called "stride". Although the examples below use a horizontal and vertical stride width value of 1, any other value greater than 1 can be used as long as the stride width is an integer greater than 0. The size of the kernel K is S×T, where S and T are integers and are less than N and M.

[0120] In addition, it is assumed that only the original input matrix I (i.e., input matrix 230) of size N×M is used to calculate the inner product with the kernel. However, it is also possible to consider using an extended matrix Z for calculating the inner product with the kernel. The extended matrix Z is obtained by "pasting" rows and columns above the first row and below the last row, and to the left of the first column and to the right of the last column of the original matrix I.

[0121] This is called "padding". Padding usually consists of adding an amount P in the row direction. w rows, and increase the number of columns by P h Column. Number P w can be equal to S-1, quantity P h can be equal to T-1, so that any inner product computed between Z and the kernel contains at least one entry of the original matrix I. Therefore, the resulting matrix Z will have ((N+2P w )×(M+2P h)) size. In view of this, matrix Z will have the following terms:

[0122]

[0123] In this case, it can be seen that the new matrix obtained by calculating all the inner products and arranging them appropriately by rows and columns generally has size, where W w and W h define the span widths in the row direction and column direction respectively. Obviously, for a given kernel K of size S×T, only allowing these paddings and these span widths will result in an integer size for the new matrix. In addition, the span widths W w and W h are preferably less than S and T respectively, because otherwise the kernel will move on matrix I in a way that some rows or columns of the original matrix will be missed when calculating the new matrix.

[0124] For the sake of discussion, it will be assumed below that no padding is provided for the original matrix I, and the span widths of both the horizontal and vertical spans are 1. In addition, assume that the kernel is a matrix of size S×S, that is, assume the special case where S = T. Applying the explanations given below to arbitrary paddings and span widths and any kernel size can be easily achieved according to the ideas below.

[0125] In the depth convolution layer 211, the received input matrix 230 is used to form the inner product with the kernel K, where the size of the kernel K is S x S and S < N,M. The inner product is calculated for each reduction matrix of the original N×M matrix, where the size of the reduction matrix is S x S and contains the coherent terms in the original N×M matrix. For example, considering S = 3, the first reduction matrix R of the N×M original matrix includes the terms i = 1, 2, 3; j = 1, 2, 3, so that the reduced matrix (N×M) S contains nine terms and the calculated inner product with the kernel K is a single number. The next reduction matrix in the row direction of the original N×M matrix is the matrix with i incremented by 1, so the next matrix in this direction is composed of the terms i = 2, 3, 4; j = 1, 2, 3 in the original N×M matrix. Then this matrix can be used to calculate the next inner product with the kernel. Note that the given example of the S×S matrix with S = 3 is just an example, and other kernels can also be used.

[0126] To calculate the size in the row / column direction as (N×M) SFor the next reduced matrix R, the subscript j of the entries in the original N×M matrix is ​​increased by 1. This is done until the last reduced matrix in the row direction, where in the case of S=3, the last reduced matrix in the line direction is i=N-S+1,N-S+2,N-S+3. This process is done in a corresponding manner for the columns, where j=M-S+1,M-S+2,M-S+3. By calculating these inner products, a new matrix is ​​calculated, the matrix P, whose size is (N-S+1)×(M-S+1). Its entries P ij corresponds to the respective inner products computed with the original N×M matrix and the corresponding reduced matrix of kernel K. It should be noted that, in practice, a matrix of this size will be forwarded to the point-wise convolutional layer of layer 200.

[0127] The kernel K constitutes the terms obtained through the learning process in which the neural network is trained to correctly identify the expected target. The size and terms of the kernel K used in the layer 200 of the neural network do not have to be the same as the kernels used in other layers of the corresponding neural network. In addition, the terms in the kernel do not need to be the same as each other, but at least they are numbers greater than or equal to 0. These terms can be regarded as representing "weighted values" obtained through learning of the neural network.

[0128] The result of the processing of the matrix 230 by the deep convolutional layer is the matrix 231. As described above, if the kernel is moved on the original N×M matrix by a stride with a distance Δi=1 in the row direction and a distance Δj=1 in the column direction, then the size of the matrix 231 is (N-S+1)×(M-S+1). However, in the case where these strides have a larger distance, such as Δi=2 or Δi=3 (and may correspond to columns accordingly), the size of the result 231 will be changed accordingly as described above.

[0129] In further processing, the result 231 is forwarded to the first batch normalizer 212 according to Figure 4 , the first batch normalizer 212 follows the depthwise convolutional layer 211. The batch normalizer attempts to normalize the received result matrix 231. This is done by calculating the sum of each entry in the (N-S+1)×(M-S+1) matrix and then dividing it by the number of entries in the (N-S+1)×(M-S+1) matrix. The (N-S+1)×(M-S+1) matrix (hereinafter denoted as P, with corresponding entries P ij ) is given by the following formula where n and m represent the number of rows and columns in an N×M matrix or the number of rows and columns in a matrix P. The term P ij is an entry of the matrix P, where a given entry P ij is the element in the i-th row and j-th column of the matrix.

[0130] The batch normalizer then normalizes the original matrix by ij Subtract the mean value V from the matrix to calculate the reduced matrix P', so that P' ij =P ij - V. Thus, the values ​​in the reduced matrix P' are normalized so that anomalies (extremely large or extremely low values) in one direction or another can be filtered out.

[0131] The result 232 produced by the first batch normalizer 212 is still of size (in Figure 4 In the example given in , it is a (N-S+1)×(M-S+1) matrix because no further size reduction has been achieved so far.

[0132] The result 232 is then provided to the first rectified linear unit 213 located after the first batch normalizer 212 .

[0133] The rectified linear unit further calculates the new matrix term To modify the items in matrix 232,

[0134]

[0135] This results in values ​​that would be less than 0 after passing through the batch normalizer being set to 0, and thus having no further impact on further processing in the deep convolutional layers, as explained below. This means, for example, that color values ​​below the mean calculated in the batch normalizer will not be considered further, and only values ​​that are at least the same as the mean V will affect the result of the next calculation.

[0136] Therefore, the result 233 output by the first rectified linear unit 213 is still a matrix of shape / size (N-S+1)×(M-S+1), and the matrix is ​​forwarded to the point-by-point convolution layer 221.

[0137] This point-by-point convolution layer 221 produces a result 234. This result 234 is produced by the point-by-point convolution layer 221 by taking each item in the (N-S+1)×(M-S+1) matrix 233 and multiplying the item by a weight value α. α is preferably a number greater than 0 in any case, and the number is the same for each item in the (N-S+1)×(M-S+1) matrix. Therefore, the result 234 obtained from the point-by-point convolution layer 221 is a matrix of the same size (N-S+1)×(M-S+1), but each item is multiplied by a weight value α.

[0138] The result 234 is then provided to the second batch normalizer 222, where it is normalized in the manner explained for the first batch normalizer 212, and a normalization matrix P' of the same size as the result 235 is calculated, and this matrix / result 235 is forwarded to the second rectified linear unit 223, where a rectification function is applied to obtain the result / matrix 236, which is then forwarded to the next layer in the neural network, or if there are no further layers in the neural network, the result 236 is provided as an output.

[0139] One of the findings of the present invention is that, in order to identify fingertips, thirteen Figure 4 The same layers as layer 200 explained in are the most suitable because they achieve a relatively high recognition accuracy of the fingertips and their positions, while requiring only a small amount of computer resources to implement the corresponding methods, which makes them more suitable for mobile devices.

[0140] Figure 5 A further embodiment is shown which extends Figure 4 to allow the use of multiple bounding boxes to identify the fingertips (specifically the pixels in the original image that make up the fingertips) and to separate the original image into a grid. Note that the steps described below can be performed after the original image has been processed in each layer of the neural network, or only after the image has been processed in the last layer of the neural network, so that it is immediately followed by Figure 3 Step 104 will output before outputting.

[0141] Figure 5 The embodiment described in assumes that the neural network has completed learning, and it is fully capable of recognizing fingertips or other targets with high accuracy in accordance with the present invention based on the outputs received from the layers of the neural network.

[0142] according to Figure 5 In the embodiment of FIG. 1 , it is assumed that the output received from the layers of the neural network can still be represented in some way in the form of an image 300 of a hand 350 including fingertips. Therefore, only reference will be made to “image”, although it is obvious that in addition to images, other terms such as Figure 2 One of the output matrices shown.

[0143] In the first step, the received image 300 is segmented into a plurality of grid cells 310, 311, and 313. The number of grid cells in each direction is not limited, but in a preferred embodiment, the image 300 is segmented into 13 grid cells in the horizontal direction and 13 grid cells in the vertical direction, thus creating a 13×13 grid instead of the general Q×R grid.

[0144] In the next step, the center point 330 of each grid cell is identified and used to establish the origin of a coordinate system for each grid cell that is independent of any other grid cell. Around this center 330, at least one bounding box 331 and 332, which is generally rectangular in shape, is arranged, as can be seen in grid cell 313, with an initial height h0 and width or extension b0. These values ​​can be different from each other for multiple bounding boxes in each grid cell. For example, the initial values ​​h0 and b0 can be taken for the smallest bounding box of each grid cell, and these values ​​can be increased by 1.5 or 2 times or any other multiple to calculate the size of other bounding boxes in the corresponding grid.

[0145] Note that the position of a bounding box, such as bounding box 331 in the coordinate system of each grid cell, will be represented by the position of the center point of bounding box 331 relative to the center point 330 in each grid cell (i.e., the origin of the respective coordinate system in each grid cell). Therefore, the position of the corresponding bounding box in grid cell 311 can be represented by two coordinates x and y. The width and height of the bounding box are considered to represent the geometric characteristics of the bounding box, which can be represented by two values ​​greater than 0.

[0146] Since these bounding boxes will be used later to identify the positions of the fingertips, it is also appropriate to associate a fifth value with each of these bounding boxes, which is the probability that the bounding box includes the respective fingertip that is to be identified.

[0147] Therefore, each bounding box can be represented by 5-dimensional vector representation in the form of .

[0148] This means that the grid cells and their corresponding bounding boxes can be represented in the form of a tensor T of dimensions Q×R×B×A, where A is the number of bounding boxes per grid cell. In the most preferred case for identifying fingertips, Q=R=13, B=5 (the dimension of vector b), and A can be set to an integer between 3 and 10, most preferably 5.

[0149] As described above, it is assumed that the neural network has fully learned how to recognize a particular target, preferably a fingertip. This involves the neural network being able to recognize a particular pattern of pixels that most likely represents a fingertip. This may refer to a particular pattern of color values ​​or other features (e.g., the brightness of a spot). However, it is clear that image 300 may arbitrarily show a fingertip, which may not correspond in size or arrangement to the fingertip used to train the neural network.

[0150] However, with the help of bounding boxes and grids, the neural network can identify a specific bounding box that is most likely to contain a fingertip. To identify this specific bounding box, the neural network (or a related component that processes image 300) compares the pixel values ​​within each bounding box for each grid cell to the pixel patterns that the neural network has previously learned correspond to fingertips. In this first stage, it is unlikely that a perfect match will be found, but there will be bounding boxes that are more likely than other bounding boxes to contain at least a portion of a fingertip.

[0151] exist Figure 5 In the illustrated case, for example, bounding box 341 centered at point M in grid cell 313 includes a portion of a fingertip of hand 350. In contrast, neither grid cell 310 nor 311 includes a bounding box that includes a portion of a fingertip. As the method continues to evaluate pixel values ​​within bounding box 341 and possibly bounding box 340, the process may determine that bounding box 341 includes more patterns corresponding to a fingertip than bounding box 340.

[0152] Given this, the method may conclude that bounding boxes 331 and 332 (and possibly other bounding boxes in other grid cells) do not contain a fingertip, and their probability values ​​in their corresponding B vectors may be set to zero.

[0153] Since both bounding boxes 340 and 341 centered at point M include at least a portion of the fingertip, it can be considered that they may actually include the fingertip, and the probability value will be greater than 0 in the first step.

[0154] Although the smaller grid cell 340 is almost completely filled with a pattern that may correspond to a fingertip, only the left border of the larger bounding box 341 may be considered by the process to include a pattern that corresponds to a fingertip.

[0155] With this, the method can proceed to calculate a loss function that determines the difference between the pattern identified within each bounding box 341 and 340 and the pattern obtained by learning that actually corresponds to the fingertip.

[0156] In the next step, the method will try to minimize this difference by modifying the size and position of the corresponding bounding box. To this end, it is conceivable to use the larger bounding box 341 as a starting point and modify its position and shape, or to use the smaller bounding box 340 as a starting point and modify its position and size to minimize the difference with the learned pattern.

[0157] This minimization process may first include modifying the position of the bounding box (hereinafter, it is assumed that the bounding box 341 is used for further calculations), which is done by surrounding the center point M of each grid cell, such as Figure 3As shown, the bounding box is moved a small amount in orthogonal directions, first along the x-axis and then along the y-axis (or vice versa). The movement is performed along the positive and negative x-axis and y-axis directions, and at each position, a comparison is made to determine the difference function between the pattern obtained by learning and the actual pattern recognized in the image. This makes it possible to calculate a two-dimensional function that represents the difference d(x,y) as a function of the coordinates.

[0158] Based on this, the gradient ▽ can be calculated xy d, the gradient ▽ xy d determines in which direction in the coordinate system to move the bounding box to increase and preferably maximize the match with the learned pattern (equivalent to minimizing the value of the function d(x,y)). This is ▽ xy Case where d=0.

[0159] This may result in the bounding box being moved in direction r to a new center point M' where the function d(x,y) has a minimum. In the next step, the size of the corresponding bounding box at position M' may be increased or decreased to determine whether increasing or decreasing the size in one or both directions (i.e., height and / or width) changes the value of another difference function compared to the original pattern, which may be denoted as e(h,b) and varies with height h and width b. This function is minimized so that for a bounding box having position M' and having height h f and width b f The difference between a specific bounding box and the learned pattern is minimized.

[0160] This bounding box will then be used as the final bounding box with the maximum probability p of identifying those parts of the image 300 that contain the corresponding fingertip. The output vector of this bounding box will then have the form

[0161]

[0162] As a result of this processing, a tensor T with dimensions Q×R×B×A is output, where, for each bounding box in each grid cell, the x and y positions relative to the center of the grid cell and the width and height of the corresponding bounding box are given, along with the probability of identifying or including a fingertip.

[0163] To prevent the bounding boxes of adjacent grid cells from being moved in the same direction so that they overlap each other, and to prevent the bounding boxes of different grid cells from being moved into other grid cells, the method may be provided that the movement of the bounding box center can only be performed within its original grid cell.

[0164] Therefore, the result will be a tensor including multiple vectors B, where one or more of these vectors have a high probability of identifying a fingertip, while other vectors have a lower probability. By setting all of their corresponding values ​​to 0, those vectors with a lower probability can be completely ignored, thereby reducing the amount of processing work required when processing the tensor.

[0165] The vector B with the highest probability will then be used to enable further processing of the image, in particular those parts of the image where a fingertip is identified, for example in order to identify the user of a mobile device by processing the fingertip to identify a fingerprint.

[0166] While the above method is able to correctly identify bounding boxes that will be used for further processing of the identified biometric (e.g., fingerprint), further explanation will be given regarding bounding boxes that must be discarded.

[0167] As described above, the vector b of the bounding box includes a probability p, which indicates the likelihood that the corresponding bounding box includes or represents a fingertip. This can be used to sort all bounding boxes (or their vectors, respectively) in descending order, starting with those vectors b with the highest probability values ​​p.

[0168] In doing so, the list can be traversed in descending order starting with the bounding box with the highest value p. This traversal can include selecting a specific bounding box with value p from the list and calculating the intersection with all the remaining bounding boxes for this specific bounding box. This means that the area of ​​the selected specific bounding box is compared with the areas of the remaining bounding boxes, and any area they have in common (i.e. where the bounding boxes intersect) contributes to the calculated intersection.

[0169] The amount of intersection can be calculated as a ratio relative to the area of ​​the selected bounding box. Thus, for each calculated intersection, a dimensionless value ranging from 0 (no intersection) to 1 (the remaining bounding box considered completely intersects or covers the area of ​​the selected bounding box) is obtained.

[0170] In the next step, a preset threshold can be used to ignore or discard bounding boxes or remove them. In the above example, the threshold can be the calculated intersection of 0.75. For each pair of calculated selected bounding boxes and remaining bounding boxes, if the intersection exceeds this threshold, the bounding box with the lower p-value can be ignored or removed from the previously mentioned list.

[0171] This will eventually result in only one bounding box remaining, which will represent the tip of the finger. Of course, depending on the number of fingers visible in the image, this may result in up to four bounding boxes remaining.

[0172] exist Figure 6In this paper, we will now explain how to correctly train the neural network so that the neural network learns the weighted values ​​of the kernel K and the reference Figure 4 The weighted value α of the explanation and the pattern that truly identifies the fingertip.

[0173] Figure 6 The method begins with providing training data 401 and preset bounding boxes 408. The training data may consist of multiple images, such as a fingertip or multiple fingers drawn in one image along with other objects. These images may be multiplied by using rotated, highlighted, darkened, enlarged or otherwise modified copies from the same image (introduced as training data). The bounding boxes provided in accordance with item 408 are the bounding boxes corresponding to their respective images in the training data, wherein these bounding boxes are the bounding boxes correctly associated with the objects to be recognized, i.e., as referenced Figure 5 The explanations have the correct size and correct location along with the corresponding probability values. These bounding boxes are provided for each and every image in the training data.

[0174] In the next step, a specific input image 402 is provided to the neural network in a training environment, in which, in addition to the neural network, an optimizer 407 and a loss function calculator 406 are also provided.

[0175] In the first round, the input image is processed using a deep convolutional layer, a first batch normalizer, and a first rectified linear unit 403, collectively referred to as DCBR, and then the input image is passed to a point-by-point convolutional layer, a second batch normalizer, and a second rectified linear unit, collectively referred to as PCBR, according to Figure 4 This means that Figure 6 The steps or portions 403 and 404 shown in FIG. 4 are preferably as described with reference to FIG. Figure 4 As described above, the corresponding weight values ​​of the point-wise convolution layer (PC) and the kernel K of the depth-wise convolution layer (DC) are used in each part 403 and 404 for 13 times. The first and second batch normalizers and the rectified linear unit of items 403 and 404 are as described above. Figure 5 Explanation of the way it works.

[0176] As a result, Figure 5 Consistently, the output will be the first tensor T of size Q×R×B×A, which has the first item T qrbaThis result is then provided to the loss function, which compares it with the preset bounding box to identify the difference between the result 405 and the correct bounding box obtained from 408. This difference obtained by the loss function 406 is then provided to the optimizer 407, which in turn modifies the weight values, i.e., α, and the terms in the kernel K of each point-by-point convolution layer and each depth-wise convolution layer. This means that the weight values ​​α of the point-by-point convolution layer and the terms in the kernel K of the depth-wise convolution layer are maintained once for all layers in the network or individually for each layer.

[0177] Using these new values, the cycle is repeated for very similar images and the resulting image with the term T' qrba The tensor T is provided to the loss function and compared with the correct bounding box, and the result is then provided to the optimizer 407 to modify the weighted value again.

[0178] As long as the resulting tensor T (n) This process is continued if the difference between the identified bounding box, in particular the predefined bounding box of item 408, exceeds a given threshold (which substantially corresponds to the expected identification accuracy).

[0179] After that, the next input image 402 is obtained from the training data 401 and the corresponding bounding box is provided to the loss function. Then, the process just explained is repeated again for this new image and the best weighting values ​​for the point-wise convolutional layer and the depth-wise convolutional layer are obtained. This process is repeated until a specific combination of weighting values ​​results in a suitable recognition accuracy for all input images. The combination of weighting values ​​obtained is then output as the final weighting value 410.

[0180] These final weighted values ​​are then introduced into an application on the mobile device that executes the method of the invention.

[0181] Thus, in the concept of the present invention, the neural network provided to the mobile device is already fully adapted to recognize a specific target carrying a biometric feature, preferably a fingertip, and can therefore be used without the need for further learning, reducing the computer resources required on the mobile device.

[0182] In general, by using point-wise convolutional layers, depth-wise convolutional layers, batch normalizers, and rectified linear units (as previously described in Figure 3 and Figure 4 explained), and by dividing the original image into grid cells and following Figure 3By identifying the corresponding bounding box according to the description, an application of less than 1 megabyte can be provided, so that the application can be used independently on a mobile device even without any opportunity to access other data sources through the Internet, etc. This makes it suitable for environments where access to wireless networks, etc. is not possible. In addition, the processor power required to run this application is minimized while still being able to produce appropriate identification results of the fingertip, which can be used for subsequent user identification through the fingerprint associated with the fingertip, as explained above.

[0183] The previous explanations have focused on images of hands or fingers showing the side of the fingers with fingerprints. However, the user may also accidentally or intentionally present one or more fingers to the optical sensor from the other side (i.e., the back of the hand). Since the fingerprints are not visible, it is impossible to extract the fingerprints from such finger images.

[0184] In order to distinguish images of fingertips carrying fingerprints from images of fingertips that do not show fingerprints but instead show nails or knuckles, the following processing procedure can be used, which can extend the method described previously to improve recognition accuracy.

[0185] In the example above, the bounding box is characterized by the vector And only finger images showing the side of the fingertip with fingerprints are used for training.

[0186] When images are allowed to be captured from both sides of the fingertip (i.e., the side with the fingerprint and the side with the nail or knuckle), it is advantageous to consider two types of objects identified in the image, namely, those objects that constitute the fingertip showing the fingerprint and those objects that constitute the fingertip showing the nail or knuckle.

[0187] In this case, the aforementioned vector can be expanded by a dimension c to obtain where c represents the so-called classes. The first class may represent a positive identification (a fingertip with a fingerprint can be identified) and the second class may represent a negative identification (a fingertip with a nail or a knuckle). The classes may be represented by numerical values, for example 1 for a positive identification and 0 for a negative identification. Obviously, unlike the rest of the values ​​in the vector b, the classes are discrete values ​​and can only take a finite number of different values ​​corresponding to the number of classes.

[0188] The training described previously may then be performed in the following manner: the neural network is provided with positive and negative recognitions (rather than just training data representing images of fingertips carrying fingerprints and bounding boxes 408) so as to be able to distinguish whether an image belongs to the first class or the second class. In this case, it is conceivable that multiple images of fingers are shown with everything but the fingerprint. All of these "targets" may be classified as the second class (i.e., negative recognitions), so that the neural network is trained to distinguish images of fingertips carrying fingerprints from images of "any other" fingertips. Of course, the bounding boxes provided for training will also contain the correct class c, so as to allow the network to be properly trained.

[0189] In order to identify all fingertips with fingerprints in the image, the above processing will ignore all bounding boxes representing fingertip locations and are considered to belong to the second category (i.e., negative identification), thereby preventing further processing of fingertip images or image portions that do not display biometric features.

[0190] In order to provide an environment in which the method of the present invention can be performed, Figure 7 A mobile device in the form of a smartphone according to one embodiment of the present invention is depicted.

[0191] The mobile device 500 is embodied as a currently known smart phone. It includes an optical sensor 520, which is preferably on the back side of a camera on the side opposite to the side on which the display 530 is arranged on the mobile device 500. The camera can be a camera with a resolution of 1MP, 2MP or even higher, such as an HD camera. The camera can be equipped with a flash, but it is not necessary. It can also be suitable for capturing real-time images at a reduced resolution, and once the camera is activated, the display 530 can show the picture that the camera actually "sees". The picture can be, for example, a hand 510.

[0192] In one embodiment of the present invention, once the method of the present invention is performed on the captured image, bounding boxes 511-514 identifying the fingertips of the hand are added to the image of the hand displayed on the display 530. As further explained above, the identified bounding boxes do not need to be displayed, but can also be further processed within the mobile device, such as to process the portion of the image corresponding to the fingertips, thereby identifying the user by identifying the fingerprint associated with the fingertips.

Claims

1. A method for identifying a user using an image of a user target having a biometric feature of the user, the biometric feature being a fingerprint or a set of fingertip fingerprints, the method comprising: Acquiring an image of the target by an optical sensor of the mobile device; providing the image to a neural network; Processing the image through the neural network to identify the target in the image and the position of the target in the image; extracting biometric features from the identified target; storing the biometric feature in a storage device and / or providing at least the biometric feature as input to an identification mechanism, the identification mechanism including processing the input to determine whether the biometric feature identifies the user, the steps of the method being performed on a mobile device; The image as input is processed by a neural network, including: the input is processed by a first layer of the neural network to create a first intermediate output, and each subsequent layer processes the output of the previous layer, wherein the neural network includes multiple layers, each layer is a depthwise separable convolution, and in the order of processing the input within the layer, includes a depthwise convolution layer, a first batch normalizer, a first rectified linear unit, a pointwise convolution layer, a second batch normalizer, and a second rectified linear unit; wherein, by using multiple layers to process the input, the neural network obtains a recognition result of the target and a position of the target in the image as an output; identifying the position of the target includes separating the image into a grid including Q×R grid cells, wherein at least one bounding box is created within each grid cell, the bounding box having a predetermined position and predetermined geometric features within the grid cell, wherein creating the output also includes modifying the position and geometric features of the bounding box to obtain a result bounding box, wherein the result bounding box is a bounding box with a result position and result geometric features that most closely match the location of the target.

2. The method according to claim 1, characterized in that The target is at least one fingertip and the biometric feature is a fingertip fingerprint, wherein processing of the input by the recognition mechanism includes extracting biometric features from the fingerprint, wherein the biometric features are locations and types of feature points, and comparing the extracted biometric features with biometric features stored in a storage device, wherein if the difference between the extracted biometric features and the stored biometric features is below a threshold, the recognition mechanism confirms that the user has been identified by the fingerprint, and if the difference between the biometric features and the stored biometric features is above a threshold, the recognition mechanism confirms that the user has not been identified by the fingerprint.

3. The method according to claim 2, characterized in that The image includes more than one fingertip, and the method further includes: identifying the position of each fingertip in the image, and using the fingerprint of each fingertip to identify the user by the recognition mechanism.

4. The method according to claim 3, characterized in that The recognition mechanism determines that the user is identified through the fingerprints of the fingertips by confirming that the combined recognition accuracy of the fingerprints of all fingertips in the image is higher than a given threshold. Alternatively, the recognition mechanism determines that the user is identified through the fingerprints of the fingertips by confirming, for each fingertip, whether the difference between the biometric features of the fingertip fingerprint and the biometric features of the stored fingertip fingerprint is lower than a threshold, and only determines that the user is identified through the fingertip fingerprint if all the determined differences are lower than the corresponding threshold.

5. The method according to any one of claims 1 to 4, characterized in that The image is obtained by a camera which is an optical sensor of the mobile device.

6. The method according to claim 1, characterized in that The position of the bounding box is calculated relative to the center of the grid cell in two dimensions, and the geometric features of the bounding box include the height and width of the bounding box, wherein the probability of the object being within the bounding box is associated with each bounding box.

7. The method according to claim 6, characterized in that Also included is a bounding box displaying the image and identifying the location of the fingertip.

8. The method according to claim 1, characterized in that Processing the image by the neural network includes creating at least one matrix I representing the color values ​​of each pixel in the image from the image, and providing the matrix as input to the neural network, wherein the image includes N×M pixels and the matrix I is a matrix including N×M values, wherein the entries of the matrix I are composed of I ij Given, where i and j are integers and i=1...N and j=1...M.

9. The method according to claim 8, characterized in that Each deep convolutional layer applies a predefined kernel K to a matrix I, where the kernel K is a matrix of size S×T, where S,T < N; S,T < M, including the term S ab , where applying the kernel to the matrix includes computing the inner product of the matrix K with each reduction matrix R of size (N×M) S,T of the matrix Z, where the matrix R is the same size as the kernel K, and the matrix Z is of size ((N + 2P w ×(M + 2P h ), and the matrix Z cd , c,d ∈ N + , whose terms are given by the following formula and provides a matrix P as output, where the size of the matrix P is Where W w and W h Define the span width, each item P of the matrix P ij is the value of the inner product of the ijth reduced matrix R and the kernel K, where the matrix P is provided as the output by the deep convolutional layer to the first batch normalizer.

10. The method according to claim 9, characterized in that The kernel sizes S and T are equal for all depthwise convolutional layers, or for at least one depthwise convolutional layer and / or at least one item in kernel K. a′b′ ≠S a≠a′，b≠b′ are different.

11. The method according to any one of claims 8 to 10, characterized in that The batch normalizer feeds the normalized reduced matrix P' to the rectified linear unit, and the rectified linear unit performs the normalization on each item P' ij Apply a correction function that computes the term The new matrix And, if the rectified linear unit is a first rectified linear unit, then the matrix is fed as output to the point-wise convolutional layer, or if the rectified linear unit is the second rectified linear unit, the matrix Provided as output to the next layer of the neural network; and / or where the matrix P, P' or Each entry in is multiplied by a weight value α, and the point-by-point convolutional layer applies the weight value α to the matrix I, P, P' or 12. A mobile device, characterized in that: The invention comprises an optical sensor, a processor and a storage unit storing executable instructions, wherein when the executable instructions are executed by a processor of a mobile device, the processor is caused to perform the method according to any one of claims 1 to 11.

Images